JunRuiLee opened a new pull request, #427: URL: https://github.com/apache/paimon-rust/pull/427
## What Align the Rust Parquet reader with Java/parquet-mr for predicate-driven **page-level** pruning using the Parquet `ColumnIndex` / `OffsetIndex`. Today the reader only prunes at **row-group** granularity (`build_predicate_row_selection`). Java Paimon additionally gets page-level skipping through parquet-mr's `readNextFilteredRowGroup` (the `COLUMN_INDEX` layer of `RowGroupFilter`). arrow-rs does **not** do this automatically — it exposes the page index but leaves it to the caller to turn a predicate into a `RowSelection`. This PR builds that selection. > Note: this is not a full equivalent of the Java reader. It targets one specific capability — predicate-driven page-level pruning via ColumnIndex/OffsetIndex. Other parquet-mr layers (bloom-filter and dictionary row-group pruning) are out of scope here. ## How - Load the page index with `PageIndexPolicy::Optional`, and **only when a predicate exists** (otherwise the extra metadata read is pure overhead). `Optional` lets files without a page index fall through to row-group pruning instead of erroring. - `build_predicate_page_selection` evaluates each page's min/max/null_count through the **same** `StatsAccessor` evaluator as row-group pruning, so both layers share identical stats-safe semantics. - Pages are split per column chunk, so different columns in a row group may have different page counts/boundaries. Each predicate column is pruned against **its own** page layout, and the per-column selections are intersected (the top-level predicate list is a conjunction). - Fix `ArrowFileReader::get_metadata`, which forwarded only `metadata_options` and dropped the column/offset index policies — without this, `with_page_index_policy` would silently no-op against the custom `AsyncFileReader` and no page index would ever load. ## Fail-open by construction Page index is only ever used to *skip* pages that provably cannot match; when it cannot be trusted, the whole row group is kept and row-group pruning / the per-row filter decide. It **never panics and never affects old-file compatibility**. Fail-open cases: - no predicate; - page index missing (older files, or a writer that didn't emit it); - a column with only chunk-level statistics (`ColumnIndexMetaData::NONE`, whose accessors panic rather than return `None`); - a page-count mismatch between the column and offset indexes; - malformed offset page boundaries (non-zero start, non-monotonic, or out of range); - a predicate the page stats cannot decide (same conservative matrix as row-group pruning). ## Tests - Eq keeps only the matching page; Eq outside all pages skips everything; range keeps overlapping pages. - NotEq (conservative op) and missing page index both fall open. - **Regression:** predicate on a non-driver column whose page layout diverges from the first column — guards the per-column page-boundary fix. - **Regression:** predicate on a column with no `ColumnIndex` — must fail open, not panic. - Unit test for the offset-boundary validation helper. `cargo test -p paimon --lib` passes (993 tests); `cargo clippy --all-targets` clean; `cargo fmt` applied. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
