hhhizzz opened a new pull request, #10852:
URL: https://github.com/apache/arrow-rs/pull/10852
# Which issue does this PR close?
- Closes #10776.
# Rationale for this change
For a first predicate with no pre-existing `RowSelection`,
`ReadPlanBuilder::with_predicate_options` currently materializes the
predicate
results as RLE selectors. `RowSelectionPolicy::Auto` may subsequently choose
the mask strategy and convert those selectors back to a bitmap.
For scattered selections such as ClickBench Q25 this creates a large
temporary
selector vector even though the final strategy is already determined by the
run-count threshold.
# What changes are included in this PR?
- Add an internal `RowSelection::from_filters_auto` constructor.
- Build normalized selectors only until Auto's mask threshold is reached,
then
construct the mask directly from the predicate BooleanArrays.
- Preserve selector backing for selector-friendly inputs.
- Use this constructor only for a first predicate with no pre-existing
selection and `RowSelectionPolicy::Auto`; existing selection, explicit
Mask,
and explicit Selectors paths are unchanged.
- Preserve Auto in `prepare_selection_for_page_skipping` when there is no
selection yet, as there are no skipped pages to prepare in that state.
# Are these changes tested?
Yes.
Focused construction benchmark over 4,194,304 rows, split into 512
`BooleanArray`s of 8,192 rows each (median of three Criterion rounds):
These measurements used the separate experiment harness. Following the
repository guidance for benchmark additions, the benchmark-only code is not
included in this PR.
| Shape | Current Auto | Capped Auto | Change |
|---|---:|---:|---:|
| Q25-like 15% scattered | 14.675 ms | 506.9 us | 28.95x faster |
| Alternating run-1 | 48.595 ms | 513.8 us | 94.57x faster |
| Exact run-32 boundary | 431.4 us | 445.9 us | +3.35% |
| Clustered run-128 | 165.8 us | 154.5 us | -6.77% |
| Sparse run-32 | 66.2 us | 65.5 us | -1.04% |
An unconditional mask-first implementation was rejected because it regressed
selector-friendly run-128 by 30.7% and sparse input by 107.6%.
Three paired async-reader rounds with PageIndex disabled showed:
- Q25-like: -26.02%
- alternating run-1: -48.91%
- run-32: -1.87%
- run-128: -0.29%
- sparse: -0.30%
- all-selected: +1.45%
Correctness coverage includes a 512-case fixed-seed matrix spanning eight
thresholds, eight selectivities, four run shapes, aligned/non-byte-aligned
buffers, multiple BooleanArray splits, deterministic 31/32 threshold
boundaries, cross-filter run merging, empty inputs, and LIMIT/padding.
Validation on the current upstream `main` base:
- `parquet` library: 1,303 passed, 0 failed, 1 known baseline test filtered
- async `arrow_reader`: 124 passed, 0 failed, 1 ignored
- `cargo clippy -p parquet --all-targets --all-features -- -D warnings`:
passed
# Are there any user-facing changes?
No public interface changes. The capped constructor is `pub(crate)`, and the
selection semantics and explicit policy behavior are unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
