shyjsarah opened a new pull request, #584:
URL: https://github.com/apache/paimon-rust/pull/584
### Purpose
Linked issue: close #583
Paimon Rust currently rejects existing `merge-engine=partial-update` tables
that use `fields.<sequence-fields>.sequence-group`. This change adds read-side
support so Rust and DataFusion can read Java/Flink-created sequence-group
tables without falling back to incorrect basic latest-non-null semantics.
### Brief change log
- Parse and validate single or composite sequence fields, protected fields,
multiple independent groups, missing fields, primary-key conflicts, and
repeated protected fields.
- Widen projected reads with the sequence fields required to merge projected
or predicate-referenced protected fields, then remove those internal
dependencies from the final projection.
- Merge each sequence group independently using Java-compatible ascending
lexicographic comparison with nulls first.
- Skip a group when its entire sequence tuple is null.
- Let a newer or equal sequence tuple update the group.
- Update sequence fields atomically and allow accepted protected fields to
be set to null.
- Keep Rust CREATE/ALTER and writer paths fail-closed for sequence-group
options.
- Continue rejecting sequence-group aggregation and remove-record/delete
semantics, which are outside this PR.
### Tests
- `cargo test -p paimon --lib` — 1797 passed, 1 ignored.
- `cargo clippy -p paimon --lib --tests -- -D warnings`.
- `cargo test -p paimon-datafusion --test pk_tables` — 52 passed.
- Added coverage for independent groups, composite tuples, all-null tuple
skipping, partial-null ordering, null protected-field replacement, projection
dependency widening, malformed options, and end-to-end projected reads.
### API and Format
No public API or storage-format changes. This only expands read
compatibility for existing partial-update tables. Rust table creation and
writes remain unsupported for sequence groups.
### Documentation
Updated the SQL documentation with the supported read-side behavior and
explicit unsupported scope.
### Review notes
The comparison behavior follows Java `UserDefinedSeqComparator.create(...,
true)`: ascending field order and nulls first. Equal sequence tuples use the
later row in the existing merge order. The implementation intentionally does
not add ownership restrictions beyond primary-key and repeated protected-field
validation used by the Java path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
