JingsongLi opened a new pull request, #706: URL: https://github.com/apache/paimon-rust/pull/706
## What changed - support `deletion-vectors.merge-on-read=true` for deduplicate primary-key batch snapshot reads - include level-0 files, preserve overlapping key ranges in one split, and apply deletion vectors before key merging - keep non-key predicates out of pre-merge file pruning and enforce them after merging - load deletion vectors lazily per physical file and cap DV merge fan-in at 256 sorted runs - match Java behavior by treating merge-on-read as an ignored option when deletion vectors are disabled - document the option and add core plus DataFusion regression coverage ## Why Deletion-vector tables normally hide uncompacted level-0 files until compaction. Java Paimon allows batch readers to opt into merge-on-read so recent level-0 data is visible immediately. The Rust reader parsed the option but did not implement the corresponding scan and read path. ## Impact Deduplicate primary-key tables with deletion vectors can now expose uncompacted data in batch snapshot queries while preserving primary-key, deletion-vector, and residual-filter correctness. The default remains unchanged. Streaming and changelog behavior are not affected, and partial-update or aggregation with DV merge-on-read remain unsupported. ## Validation - `cargo test -p paimon --lib` — 2244 passed, 1 ignored - `cargo test -p paimon-datafusion --test pk_tables` — 54 passed - `cargo clippy -p paimon --all-targets -- -D warnings` - `cargo clippy -p paimon-datafusion --test pk_tables -- -D warnings` - `cargo fmt --all -- --check` - `git diff --check` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
