hhhizzz opened a new pull request, #9956:
URL: https://github.com/apache/arrow-rs/pull/9956
# Which issue does this PR close?
- Closes #8846
- Part of #7456
- Addresses #9591
- Related to #9589
- Follow-up to #8565
# Rationale for this change
`RowFilter` can be much slower than a full scan for fragmented selections,
especially when page indexes provide little pruning and the resulting
`ReadPlan` contains many tiny select/skip runs. #8565 shows an extreme case
where predicate pushdown is around 10x slower than scanning and filtering
afterwards.
This PR improves the `RowSelectionPolicy::Auto` path so it can make better
strategy decisions for fragmented row selections and avoid continuing with
row-filter pushdown when the observed shape suggests the pushdown path is
unlikely to help.
The main goal is to reduce the performance cliff without changing explicit
`Mask` / `Selectors` behavior.
While working on the Auto strategy, this PR also fixes a correctness issue
in the `Mask` execution path. With sparse page-loaded ranges, a mask-backed
read plan could previously attempt to consume selected rows outside the loaded
ranges and fail during decoding. This made `Mask` risky for some page-index /
fragmented-selection cases.
This PR makes the loaded row ranges explicit in the read plan and adds
coverage for sparse loaded-range execution. After this change, using `Mask`
should no longer hit this known failure mode.
# What changes are included in this PR?
- Adds structured `RowSelection` shape analysis and strategy decision
metrics.
- Improves `RowSelectionPolicy::Auto` so it can choose between mask and
selectors using selection shape and loaded page ranges.
- Adds sparse loaded-range tracking from page offsets so fragmented
page-loaded selections can avoid expensive mask execution.
- Adds adaptive post-filter fallback for Auto:
- observes the first row group;
- keeps pushdown for sparse/low-selectivity cases where it is still useful;
- switches later row groups to post-filter for high-selectivity or
fragmented moderate/high-selectivity cases.
- Preserves correctness around fallback:
- does not reuse sparse predicate chunks when rebuilding a base/full read;
- disables post-filter fallback for `try_next_reader` handoff paths;
- avoids evaluating `ArrowPredicate` twice for the same current row group
when caller-provided row selection is present.
- Simplifies fallback trigger metrics to shape-based reasons only, avoiding
unused timing/cache fields.
- Replaces loaded range intersection with a linear two-pointer merge.
- Extends `arrow_reader_row_filter` benchmarks to cover strategy-sensitive
cases.
- Fixes a known `Mask` correctness failure with sparse loaded page ranges.
- Preserves explicit `Mask` behavior while ensuring sparse loaded ranges are
tracked so mask execution does not read outside available page ranges.
# Are these changes tested?
Yes.
Unit / integration validation:
- `cargo fmt -p parquet -- --check`
- `git diff --check`
- `cargo test -p parquet --lib arrow::push_decoder`
- 38 passed
- `cargo test -p parquet --lib arrow::arrow_reader::read_plan`
- 28 passed
- `cargo test -p parquet --lib arrow::arrow_reader::selection`
- 25 passed
- `cargo test -p parquet --lib`
- 1132 passed
New focused tests cover:
- Auto fallback with caller-provided row selection does not evaluate the
current row group predicate twice.
- `try_next_reader` does not use post-filter fallback in normal reader
handoff mode.
- Sparse/current-row fallback does not reuse incompatible predicate chunks.
- Explicit `Mask` execution remains correct with sparse loaded page ranges.
- Auto switches away from `Mask` when sparse loaded ranges make selector
execution safer.
- Multi-column loaded row-range intersection remains correct after switching
to a linear merge.
- Auto strategy decisions for sparse loaded ranges and expensive fragmented
output.
Benchmark evidence from `arrow_reader_row_filter` comparing `origin/main` vs
this branch:
| case | origin/main | this branch | change |
|---|---:|---:|---:|
| `utf8View != ''`, all columns | 7.91 ms | 5.52 ms | +43.2% |
| `int64 > 90`, all columns | 7.29 ms | 5.34 ms | +36.5% |
| `int64 > 90`, exclude filter column | 6.97 ms | 5.23 ms | +33.3% |
| `utf8View != ''`, exclude filter column | 6.77 ms | 5.53 ms | +22.5% |
| `float64 > 99.0`, exclude filter column | 4.94 ms | 4.38 ms | +12.9% |
Across the 16 async `arrow_reader_row_filter` cases run, geometric mean
speedup was about 1.08x. The worst observed regression in that run was about
-3.3%.
# Are there any user-facing changes?
No intended breaking API changes.
`RowSelectionPolicy::Auto` may choose different internal execution
strategies than before. Explicit `Mask` and `Selectors` policies remain
available for callers that want fixed behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
