JingsongLi opened a new pull request, #1056: URL: https://github.com/apache/paimon-rust/pull/1056
## Purpose Enable Native for existing PyPaimon grouped writes. PyPaimon exposes `with_dynamic_bucket_index` and `write_arrow_batch_to_bucket`, and Daft workers call both. The Rust writer previously lacked these APIs, forcing the worker back to Python; calling them after a native write could fail instead. ## Changes - Implement both APIs in Rust core, with thin Python bindings and matching stubs. Support HASH_FIXED and HASH_DYNAMIC groups, including selected-column append inputs. - Restore only the supplied dynamic bucket's complete HASH file. Pin index restoration to the coordinator's base snapshot, including an explicit empty base and overwrite. Ordinary assignment can reuse mappings already notified by a direct group. - Reuse the ordinary writer's schema normalization, RowKind filtering, file writers, changelog, and ownership paths. Keep supplied Java BinaryRow hashes aligned with surviving rows; accept signed and unsigned 32-bit hashes at the Python boundary. - Notify every surviving key as Java does. A `new_mappings` hint cannot suppress a key when filtering removed its first notification. - Keep key-value sequence counters across successful prepares, including preparation before publication. Initialize a newly opened writer from current data files, independently of the index base snapshot. - Keep prepared files outside writer cleanup ownership. Reject invalid bucket IDs, metadata lengths, mixed partitions, and unsupported modes before staging a partial group. The design follows Java `TableWriteImpl.write(row, bucket)`, `DynamicBucketIndexMaintainer`, and the retained `MergeTreeWriter` checkpoint lifecycle. No new dependency is added. ## Verification - Core table module: 1814 passed, 3 ignored. - Precomputed bucket, dynamic bucket, and partial-write integration tests: 26 passed, including 15 new tests. - Python bindings: 317 passed across the full run and the rerun of four fixture-dependent catalog tests. The local smoke warehouse was provisioned with PyPaimon. - Paired PyPaimon Native write/Daft/schema/sequence tests: 246 passed, 1 skipped, 11 subtests passed. - PyPaimon ordinary write/Daft regressions: 111 passed, 10 skipped, 17 subtests passed. - `cargo +1.98.0 clippy --locked -p paimon -p pypaimon_rust --all-targets -- -D warnings` and pinned rustfmt checks pass. - Two focused independent review passes; sequence restoration, overwrite validation, and filtered first-mapping issues found in review were fixed and covered by regression tests. Paired Python PR will continue to build Rust from `main` and depends on this PR merging first. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
