JingsongLi opened a new pull request, #574: URL: https://github.com/apache/paimon-rust/pull/574
## What changed - apply Paimon predicates exactly by default in file readers, including compound Parquet predicates with a residual fallback - add a pruning-only predicate channel so metadata pruning cannot accidentally become row-level filtering - add a per-file Parquet decoder-filter factory for engine-specific expressions - adapt live DataFusion runtime filters to Parquet `RowFilter` while retaining an exact scan-level fallback - keep primary-key merge, schema-evolution, `_ROW_ID`, and position-based reads on correctness-preserving paths - remove the `paimon.read.row_filter` session option and the TPC-DS `--paimon-row-filter` switch - align the TPC-DS DataFusion and Paimon reader batch size through `--batch-size` ## Why Paimon scans previously used pushed predicates mainly for metadata pruning unless row-level filtering was explicitly enabled. This left selective queries, especially queries driven by dynamic runtime filters, decoding and materializing substantially more rows than DataFusion's native Parquet scan. The scan now always enforces accepted predicates exactly and can reuse DataFusion's Parquet decoder filter implementation for expressions that Paimon's native predicate model cannot represent. Pruning-only predicates remain separate so merge and positional semantics are not changed. ## Impact DataFusion can remove parent filters for supported runtime predicates because the Paimon scan now evaluates them exactly. Static Paimon predicates also filter by default. Primary-key merge and data-evolution readers still apply residual predicates only after the required merge/evolution step. ## Validation - `cargo fmt --all -- --check` - `cargo test -p paimon --lib` — 1706 passed, 1 ignored - `cargo test -p paimon-datafusion physical_plan::scan::tests --lib -- --test-threads=1` — 22 passed - `cargo test -p paimon-datafusion --test blob_tests test_blob_descriptor_filter_before_resolve_by_default -- --exact` - `cargo test -p paimon-tpcds-bench` — 7 unit and 5 smoke tests passed - `cargo check -p paimon --features vortex` - `cargo clippy -p paimon -p paimon-datafusion -p paimon-tpcds-bench --all-targets -- -D warnings` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
