taoran92 opened a new issue, #1225: URL: https://github.com/apache/paimon-rust/issues/1225
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-rust/issues) and found nothing similar. ### Motivation A partial-update table may need to retain the latest user status while retaining the channel associated with the earliest event. Each sequence group needs to choose its ordering direction independently. Java Paimon supports this through `fields.<ordering-fields>.sequence-group.sort-order`, introduced by [apache/paimon#10546](https://github.com/apache/paimon/issues/10546) and implemented in [apache/paimon#10547](https://github.com/apache/paimon/pull/10547). Paimon Rust currently always prefers larger sequence-group ordering values and silently ignores `descending`, so the same table can produce different results depending on the client. For example, consider this table configuration: ```sql CREATE TABLE user_summary ( user_id BIGINT, status STRING, status_version BIGINT, channel STRING, first_event_time BIGINT, PRIMARY KEY (user_id) ) WITH ( 'bucket' = '1', 'merge-engine' = 'partial-update', 'fields.status_version.sequence-group' = 'status', 'fields.first_event_time.sequence-group' = 'channel', 'fields.first_event_time.sequence-group.sort-order' = 'descending' ); ``` Input records, in arrival order: ```text user_id status status_version channel first_event_time 1 active 200 organic 100 1 registered 100 referral 50 ``` Expected result, matching Java: ```text user_id status status_version channel first_event_time 1 active 200 referral 50 ``` Current Rust result: ```text user_id status status_version channel first_event_time 1 active 200 organic 100 ``` The status group should retain version `200`, while the event group should retain time `50` and its corresponding channel. The current behavior was reproduced at [370428b](https://github.com/apache/paimon-rust/commit/370428b053da5692fad3cf9db32f95e62d72ab98) through the Rust Python binding, both within one write flush and across separate commits. Projecting only `channel` also returns `organic` instead of `referral`. ### Solution Support the same option and semantics as Java: ```text fields.<ordering-fields>.sequence-group.sort-order = ascending | descending ``` - `ascending` is the default and retains larger ordering values, preserving existing behavior. `descending` retains smaller values. Accept values case-insensitively. - Configure each group independently. For composite ordering fields, compare lexicographically in the declared field order and apply the direction to every non-null field comparison. - Preserve existing null and tie behavior: an all-null ordering tuple skips the group; a non-null value takes precedence over a top-level null in either direction; equal tuples replace non-aggregated protected fields, including with null. A non-null `ROW(NULL)` still participates in comparison, with its internal comparison reversed for `descending`. - Apply the direction consistently during writes and reads, including aggregation and sequence-group deletion. Preserve it through projected reads and column renames. - Validate the option during table creation and when opening tables for reads or writes. Require a corresponding sequence group on a primary-key table using `merge-engine=partial-update`, and reject invalid direction values. Add regression coverage for the example above, reversed input order, same-flush and cross-commit merges, composite and ROW ordering fields, nulls and ties, projection, rename, aggregation, deletion, and invalid configuration. ### Anything else? This request ports the Java capability to the Rust engine. Python callers using the Rust Native backend benefit from the same implementation; implementing sequence groups in the independent Python backend is a separate feature. Document that changing the direction cannot recover values discarded by earlier merges. All readers, writers, and compaction jobs must support the option before using `descending`, because older clients may ignore it. ### Willingness to contribute - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
