JingsongLi opened a new pull request, #845: URL: https://github.com/apache/paimon-rust/pull/845
### Purpose Continue Java parity for dynamic primary-key and first-row tables after #841. Two gaps were hidden by snapshot-only or empty first-row reads: - Cross-partition updates wrote UPDATE_BEFORE with the new partition's physical values into the old partition directory. Java `DeleteExistingProcessor` writes DELETE with the old partition values. Incremental Rust reads therefore exposed the wrong row kind and partition. - Explicit first-row all-files scans packed overlapping L0 runs separately and pruned non-key values before merging. Reading versions `(1, 10)` and `(1, 99)` could return both rows, or incorrectly return `(1, 99)` for `value = 99`. ### Brief change log - Restore old partition columns before writing migration DELETEs, sharing typed partition-to-Arrow conversion with format-table reads. - Merge first-row versions by earliest sequence, with Java's retract validation and `ignore-delete` handling. - Preserve overlapping first-row versions and apply value predicates after merging when L0 is included. Ordinary first-row batch scans continue to skip L0; incremental scans preserve physical events. - Match Java's DV precedence for L0 visibility, without changing the existing materialized DV index pruning path. ### Tests - `cargo test -p paimon --lib`: 2,806 passed, 2 ignored. - `first_row_scan_test`, `dynamic_bucket_scan_test`, `incremental_batch_scan_test`, `scan_planning_parity_test`, `audit_log_table_test`: 76 passed. - `cargo clippy -p paimon --lib --test first_row_scan_test --test dynamic_bucket_scan_test -- -D warnings` and `cargo fmt --all -- --check` passed. - Built the Python wheel from this branch and ran the Python native-planning, split serialization and global-index suites: 196 passed, 276 native plans exercised. - Regression assertions cover bucket growth, restored cross-partition indexes, deduplicate/first-row migration behavior, partition-filtered incremental events, compacted first-row reads, overlapping files, residual filters, sequence ties, retracts, null partitions, decimals and nanosecond timestamps. ### API and Format No public API or format version changes. New cross-partition retracts carry Java-compatible DELETE kinds and old-partition values. Existing files are not rewritten. ### Documentation Implementation comments describe the corrected planning and read contracts. This does not claim support for reading first-row PK-clustering files sorted by non-key columns. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
