JunRuiLee opened a new pull request, #448: URL: https://github.com/apache/paimon-rust/pull/448
### Purpose This makes predicate evaluation exact across all file formats in the Rust reader, so a consumer that returns scan batches verbatim gets exactly the rows matching the pushed-down predicate. The motivating case is pypaimon delegating its read kernel to the Rust core: that path runs the whole scan on Rust and returns the resulting Arrow batches directly, without re-applying the predicate above the scan. Today that is only safe for Parquet — the other formats apply the predicate partially or not at all, so a filtered read silently returns rows that do not match: - **ORC** pruned by stripe/row-group statistics only — non-matching rows inside a selected stripe survived. - **Avro** and **Row** ignored the predicate entirely (the reader's `predicates` parameter was unused). A silent over-read (wrong data, no error) is the worst failure mode for that delegation. After this change, every format applies the predicate exactly, closing this class of silent-wrong-read for predicate pushdown regardless of format. (Exact `LIMIT` remains the caller's responsibility, matching `ReadBuilder::with_limit`'s documented hint semantics and Paimon's Java behavior.) ### Brief change log - Consolidate the two near-duplicate Arrow-batch predicate evaluators (one private to the Parquet reader, one in the Vortex reader) into a shared `arrow::residual` module. The Vortex copy previously fell open on `startsWith`/`endsWith`/`contains`/`like`/`between`; the shared evaluator implements them, so those operators are now evaluated exactly wherever the residual filter runs. - Apply the shared exact residual filter after ORC stripe pruning (pruning kept as an optimization). - Apply the residual filter in the Avro and Row readers, which previously ignored predicates. - Parquet is unchanged in behavior (already exact; now shares the evaluator). ### Tests - `cargo test -p paimon --lib`: all pass. Per-format exactness tests (ORC/Avro/Row) assert that reading `[10,20,30,40,50]` with `age > 25` returns exactly `[30,40,50]`, plus string-operator cases; an end-to-end test drives a `.row` file through `DataFileReader`. - `cargo test -p paimon --lib --features vortex`: all pass (the consolidation touched the Vortex reader). - Shared-evaluator unit tests cover the comparison/set/string/range leaves, And/Or/Not composition, and the NULL→false convention. ### API and Format No public API or storage-format change. ### Documentation Behavior documented inline; no separate docs update required. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
