JacobSMoller opened a new pull request, #11140:
URL: https://github.com/apache/paimon/pull/11140

   ### Purpose
   Attempted solution for https://github.com/apache/paimon/issues/6439 which 
was closed by  #6664 but that was reverted again in 
[here](https://github.com/apache/paimon/commit/820c6e0e54f56e75f758a908c7b96885058de25d)
 but the issue stayed closed even though the functionality never made it in.
   
   ### Tests
   tests added in PartitionTimeExtractorTest.java
   
   
   ### Addtional info
   We have run it like this in production for a while.
   
   we do something like
   
   ```
   SELECT
     id, trace_id, timestamp_received, ip, url, user_agent, `method`, body, 
manual_request_id,
     SECONDS_TO_DATE_PARTITION(timestamp_received.seconds) AS partition_date,  
-- DATE
     SECONDS_TO_HOUR_PARTITION(timestamp_received.seconds) AS partition_hour   
-- INT
   FROM data_stream ds
   WHERE ...
   ```
   SECONDS_TO_DATE_PARTITION and   SECONDS_TO_HOUR_PARTITION is used to convert 
protobuf WKT timestamp to something human readable. 
   
   And then we do 
   ```
   "partition.expiration-time": "181 d"
   "partition.timestamp-pattern": "$partition_date::epoch_day 
$partition_hour:00:00"
   "partitionKeys":
     - "partition_date"
     - "partition_hour"
   ```
   
   
   e.g.  in plain flink sql
   ```
   -- Flink SQL
   CREATE TABLE events (
     id STRING,
     ts TIMESTAMP(3),
     partition_date DATE,
     partition_hour INT
   ) PARTITIONED BY (partition_date, partition_hour) WITH (
     'partition.expiration-time' = '181 d',
     'partition.timestamp-pattern' = '$partition_date::epoch_day 
$partition_hour:00:00'
   );
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to