JingsongLi opened a new pull request, #9839: URL: https://github.com/apache/paimon/pull/9839
### Purpose Align Python batch scans with Java `CoreOptions.batchScanSkipLevel0` and enable the corresponding native planning modes. Previously, Python read uncompacted first-row files in batch scans and always excluded level 0 with deletion vectors, even when merge-on-read was enabled. - Exclude level 0 for first-row batch scans and honor `deletion-vectors.merge-on-read` for deletion-vector tables. Keep level 0 visible to write/overwrite, incremental, and streaming scans, and apply non-primary-key file statistics only when level 0 is excluded. - Enable native planning for dynamic buckets, cross-partition primary keys, ordinary first-row batch scans, and deletion-vector merge-on-read. Forward the effective merge-on-read option, including removal through `copy()`. - Read Rust-written manifests with top-level Avro union schemas through the compatible reader while retaining partition/bucket filtering. Add real bucket-growth and partition-migration tests, historical snapshot/shard/filter/limit checks, and batch/write/streaming regressions. Native planning remains optional. Chunk shuffle, scored or primary-key global-index results, and first-row scans including level 0 retain their existing fallback. The README also records the existing incremental deletion-vector difference from Java, which this PR does not resolve. Related Rust fixes: https://github.com/apache/paimon-rust/pull/839. ### Tests - Relevant Python planner, schema evolution, writer, scanner, and manifest tests: **261 passed, 1 deselected**, using the Rust wheel built from apache/paimon-rust#839. Native-path regressions fail explicitly if planning falls back to Python. - Additional ordinary Python test group: **763 passed, 1 deselected**. This group overlaps the targeted tests, so the counts are not cumulative. - Changed Python files pass Flake8 with `paimon-python/dev/cfg.ini`; `git diff --check` passes. Optional codec subprocess validation did not complete locally, and an additional Torch distributed test timed out. Java/Python fixture-based e2e tests require Java-created tables that were not available in this local run. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
