JingsongLi opened a new pull request, #1017: URL: https://github.com/apache/paimon-rust/pull/1017
## What changes This prepares the Rust core paths needed by PyPaimon native cross-partition and ordinary postpone writes. There are no new Python binding APIs or dependencies. ### Cross-partition primary keys - Build and probe the global index with the full primary key, including primary-key columns that are also partition columns. The previous trimmed key could conflate tenants. - Preserve Java's DELETE-before-input order inside each partition/bucket, including A → B → A migrations in a single Arrow batch. - Preserve generated `rowkind.field` kinds without adding a duplicate `_VALUE_KIND` column. - Match Java's ordered bucket reuse and smallest-unused bucket allocation, using one ordered count map per partition. - Stream bootstrap batches and reject duplicate global keys rather than silently selecting one location. - Reject `bucket-key`, copied `sequence.field` options and unsupported index TTL at writer construction. ### Postpone files - Write decodable first/last primary-key bounds and actual retract counts, matching `KeyValueDataFileWriter`. - Honor `target-file-row-num` at batch boundaries. - Restore file creation order after asynchronous closes. Java's postpone replay sorts by millisecond creation time, so equal timestamps must retain arrival order. ## Java references `RowPartitionAllPrimaryKeyExtractor`, `GlobalIndexAssigner`, `DeleteExistingProcessor`, `UseOldExistingProcessor`, `BucketAssigner`, `KeyValueDataFileWriter`, `KeyValueFileWriterFactory`, and `PostponeUtils` in apache/paimon master. ## Tests New regressions cover full composite keys, repeated migrations at multiple batch sizes, all four merge engines across writer restarts, generated deletes, dynamic overwrite, invalid bootstrap/configuration, bucket reuse, physical postpone row order, key bounds, delete counts, sequence ranges and row-count rolling. Existing postpone metadata assertions are updated to the Java contract. - `cargo fmt --all -- --check` - `cargo clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings` - `cargo test --locked -p paimon --all-targets --features fulltext,vortex` (local run in progress) - PyPaimon tests against the locally built binding: 253 focused native tests and 221 Python-path tests passed; broader native writer interoperability tests also passed. Cross-partition index TTL and distributed ownership coordination remain unsupported. No compaction is added. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
