taoran92 opened a new issue, #10546: URL: https://github.com/apache/paimon/issues/10546
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Motivation This request comes from customer feedback in production: a partial-update table needs to retain the latest value for one group of fields and the earliest value for another. Currently, sequence groups prefer larger ordering values. Paimon already supports `sequence.field.sort-order` for record-level ordering through #4385. We propose similar support for individual sequence groups. ### Solution Add the following option: ```text fields.<ordering-fields>.sequence-group.sort-order = ascending | descending ``` - `ascending` (default): prefer larger ordering values, preserving existing behavior. - `descending`: prefer smaller ordering values. Each sequence group can configure its direction independently. For groups with multiple ordering fields, comparison remains lexicographic in the declared field order, with the configured direction applied to each non-null field comparison. For example, retain the latest user status and the channel associated with the earliest event: ```sql CREATE TABLE user_summary ( user_id BIGINT PRIMARY KEY NOT ENFORCED, status STRING, status_version BIGINT, channel STRING, first_event_time BIGINT ) WITH ( 'merge-engine' = 'partial-update', 'fields.status_version.sequence-group' = 'status', 'fields.first_event_time.sequence-group' = 'channel', 'fields.first_event_time.sequence-group.sort-order' = 'descending' ); ``` Input: | user_id | status | status_version | channel | first_event_time | |---|---|---|---|---| | 1 | active | 200 | organic | 100 | | 1 | registered | 100 | referral | 50 | Expected output: | user_id | status | status_version | channel | first_event_time | |---|---|---|---|---| | 1 | active | 200 | referral | 50 | The status group retains version `200`, while the event group retains time `50` and its corresponding channel. Existing null handling and equal-value behavior should remain unchanged: - An incoming record skips a group when all of that group's ordering fields are null. - During each ordering-field comparison, a non-null value takes precedence over null in either direction. - Equal ordering values still replace non-aggregated protected fields, including with null. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
