JingsongLi opened a new pull request, #1016: URL: https://github.com/apache/paimon-rust/pull/1016
### Purpose Complete the Rust core dynamic-bucket writer contracts needed to enable PyPaimon Native writes for primary-key tables with `bucket=-1` and partition keys contained in the primary key. Reproduced on the previous implementation: - `dynamic-bucket.max-buckets=1`, target rows 1, and three new keys produced buckets 0, 1, 2. - A partition with existing data and missing HASH indexes was treated as empty, allowing an existing primary key to be assigned to another bucket. - HASH output ignored external index/data destinations, and a later HASH write failure could leave previously prepared indexes behind. ### Brief change log - Match Java `PartitionIndex`: validate max-buckets, enforce the signed-short bucket ID range, reuse existing buckets after an explicit limit, and allocate unused IDs after restoration. Align the default target row count with Java's 2,000,000. - Restore HASH and active data buckets from the same snapshot; reject missing, duplicate, conflicting, truncated or invalid indexes before producing new files. - Reject `bucket-key` for dynamic writes, matching Java SchemaValidation and trimmed-primary-key hashing. - Reuse the existing bucket path factory for external HASH files, preserving round-robin state across checkpoints. Honor `global-index.external-path` for table-level indexes. - Share legacy Python bucket-index location resolution with deletion vectors. - Clean all indexes from a failed prepare, including earlier buckets/partitions; poison writers after stateful routing failures. - Keep the Java HASH commit format: a new full index replaces the previous index for its partition/bucket; no synthetic DELETE messages or Python binding logic. ### Tests - 103 focused Rust tests passed: dynamic assigner (14), index paths (14), table writer (69), dynamic write integration (5), dynamic/cross-partition scan integration (1). - Includes fault injection on the second HASH write, asserting that both HASH and Parquet files are removed and prepare cannot be retried. - `cargo clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings` - `cargo fmt --all --check` - Rebuilt the Python binding and ran 390 PyPaimon tests with native plan/read/write/commit enabled (plus 33 subtests), including both committers and Python/Rust writer interchange. ### API and Format Adds the existing Java `dynamic-bucket.max-buckets` core option accessor. No Python binding API or storage format changes. HASH files remain big-endian int32 sequences. Independent writers must retain Java's single-owner-per-bucket contract. ### Documentation The companion PyPaimon PR documents Native dynamic-bucket support, configuration constraints and HASH ownership. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
