JacobSMoller opened a new pull request, #11140: URL: https://github.com/apache/paimon/pull/11140
### Purpose Attempted solution for https://github.com/apache/paimon/issues/6439 which was closed by #6664 but that was reverted again in [here](https://github.com/apache/paimon/commit/820c6e0e54f56e75f758a908c7b96885058de25d) but the issue stayed closed even though the functionality never made it in. ### Tests tests added in PartitionTimeExtractorTest.java ### Addtional info We have run it like this in production for a while. we do something like ``` SELECT id, trace_id, timestamp_received, ip, url, user_agent, `method`, body, manual_request_id, SECONDS_TO_DATE_PARTITION(timestamp_received.seconds) AS partition_date, -- DATE SECONDS_TO_HOUR_PARTITION(timestamp_received.seconds) AS partition_hour -- INT FROM data_stream ds WHERE ... ``` SECONDS_TO_DATE_PARTITION and SECONDS_TO_HOUR_PARTITION is used to convert protobuf WKT timestamp to something human readable. And then we do ``` "partition.expiration-time": "181 d" "partition.timestamp-pattern": "$partition_date::epoch_day $partition_hour:00:00" "partitionKeys": - "partition_date" - "partition_hour" ``` e.g. in plain flink sql ``` -- Flink SQL CREATE TABLE events ( id STRING, ts TIMESTAMP(3), partition_date DATE, partition_hour INT ) PARTITIONED BY (partition_date, partition_hour) WITH ( 'partition.expiration-time' = '181 d', 'partition.timestamp-pattern' = '$partition_date::epoch_day $partition_hour:00:00' ); ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
