SteNicholas opened a new pull request, #1469: URL: https://github.com/apache/paimon-rust/pull/1469
### Purpose Closes #1445. A selective filter on a Data Evolution table can make the reader decode an entire group of column files. Older files can hold overwritten values, and the current providers can have different Parquet page boundaries. This change identifies candidate row IDs from the files that supply the filtered columns and reads those positions from every contributing file. ### Brief change log - Resolve column ownership by stable field ID, historical schema and file version. Use a file's row-group statistics and page indexes for a conjunct only when it supplies every referenced field. Evaluate predicates spanning providers on the reconstructed rows. - Intersect candidate row IDs with requested ranges and file-index hits, exclude deleted rows, and apply the same selection to every source. Attach row IDs per batch and evaluate the full predicate before `LIMIT`. - Reuse the Parquet footers and indexes loaded during planning. Fall back to row-group statistics when page positions or byte ranges are invalid, and restore the original selection when file and manifest row counts disagree. - Enable pruning for ordinary Parquet files that share a complete row-ID range on row-tracked append tables with `bucket=-1`. Dedicated BLOB and vector groups, nested-field evolution and ROW sidecar reads use their existing read paths. ### Tests - **4,996 core unit tests passed** with 8 ignored by default. **45 integration tests passed** across bitmap merges, ROW sidecars, vector compaction, system columns, partition restoration and Parquet page indexes. Coverage includes renamed and recreated columns, nulls, mismatched page boundaries and corrupt offsets. Counting-I/O tests verify one footer fetch per provider and reuse of a separately fetched page index. - Ran the selective and full-range benchmark over 262,144 logical rows on an Apple M4 Pro. The selective filter returned the same row in both modes. Pushdown reduced rows delivered to the merge from 524,288 to 128, column-data bytes from 2,342,012 to 756, and median scan time from 61.8 ms to 13.0 ms. The full-range filter returned all rows in both modes, with a 4.4% increase in median scan time. - Formatting and full-workspace Clippy passed on Rust 1.98.0 with `--all-targets --workspace --features vortex -- -D warnings`. ### API and Format `CoreOptions::data_evolution_merged_read_stats_pushdown_enabled()` reads the Java-compatible table option `data-evolution.merged-read.stats-pushdown.enabled`. It defaults to `true`. Set it to `false` to disable merged-read statistics pruning. The reader uses metadata already stored in existing Parquet files. ### Documentation `docs/src/getting-started.md` describes when pruning applies, how it falls back and how to disable it. `docs/src/benchmark.md` includes the reproduction command, byte counts and seven-run median scan times for both filters. 🤖 Assisted with Codex. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
