hhhizzz commented on issue #10776: URL: https://github.com/apache/arrow-rs/issues/10776#issuecomment-5423515360
PR for this issue: https://github.com/apache/arrow-rs/pull/10852 ## Core result The implementation constructs normalized selectors only until Auto's strict run-count threshold proves that the final strategy must be Mask. It then builds the bitmap directly from the predicate BooleanArrays. Selector-friendly inputs continue to use selectors. In a focused benchmark over 4,194,304 rows (512 x 8,192-row BooleanArrays), the median of three Criterion rounds was: | Shape | Current Auto | Capped Auto | Change | |---|---:|---:|---:| | Q25-like 15% scattered | 14.675 ms | 506.9 us | 28.95x faster | | Alternating run-1 | 48.595 ms | 513.8 us | 94.57x faster | | Exact run-32 boundary | 431.4 us | 445.9 us | +3.35% | | Clustered run-128 | 165.8 us | 154.5 us | -6.77% | | Sparse run-32 | 66.2 us | 65.5 us | -1.04% | Mask-first was not a safe general solution: it regressed run-128 by 30.7% and sparse input by 107.6% because selector-friendly inputs then had to lower a bitmap back into selectors. At the async Parquet-reader level, three paired rounds showed -26.0% for the Q25-like shape and -48.9% for alternating run-1. Run-32, run-128, sparse, and all-selected stayed within the 3-5% practical regression gate. The branch also fixes a reachability issue: page-skipping preparation previously resolved Auto to Selectors even when no selection existed, preventing the new first-predicate path from running. ## What I found after the #10852 fix The construction fix removes the Boolean -> selectors -> mask round trip, but it is only one part of the original end-to-end pushdown gap. I continued with a separate experiment to understand the remaining costs. ### 1. Predicate/output overlap For Q25, `SearchPhrase` is both the predicate column and the output column. The normal pushdown path still builds a selection, manages the predicate cache, and then creates an output reader around the same string column. A direct-output path can decode the predicate/output union once, evaluate the predicate, and return the filtered output directly. ### 2. Whole-row-group materialization was not acceptable The first direct-output prototype collected all filtered batches for a Row Group before returning. I replaced this for the DataFusion path with a streaming reader that owns the stateful `RowFilter` until row-group EOF and returns it to the decoder through a completion slot. Dropping an undrained reader fails closed instead of reusing a positionally stale `FnMut` predicate. ### 3. Ordered split predicates should remain Arrow-owned Combining predicates in DataFusion could regress when Arrow later rejected the direct path: the predicates had already lost sequential short-circuiting, but there was no direct-output benefit. The experiment therefore lets Arrow execute ordered split predicates, with each later predicate evaluating only the prior survivors, and composes the masks in original-row coordinates. ### 4. Regressions exposed missing admission signals Two useful counterexamples shaped the final experimental policy: - TPC-H Q13 selected 14.84M of 15.00M rows. A static byte-length rule admitted direct output, but post-filtering copied almost all output and coalesced two nearly full batches. Projecting output before filtering, bypassing coalescing for batches at least 75% full, and observing one Row Group reduced the repeatable 19-25% regression to about +1.6%. - ClickBench Q36 had `URL` as both output and a late predicate column. The union therefore looked like zero deferred output, but direct decoding front-loaded `URL` before selective numeric predicates. Detecting staged predicate projections and observing one Row Group reduced the 20-run average regression from +16.8% to +1.5%. The resulting experimental admission rule uses three signals: - actual mask-backed selectivity from the first eligible Row Group (8% <= selected rows < 50%); - deferred uncompressed bytes (1:1 cap for variable-width output, 5:1 for fixed-width output); - predicate projection topology, to preserve staged decoding when predicates require different columns. It also rejects actual pre-existing `RowSelection`, offset/limit, virtual columns, and nested top-level projections. Merely loading page-index metadata does not reject the path if no page-level selection was produced. ## End-to-end experimental result With the broader direct-output experiment enabled: - focused ClickBench Q25: ON 174.7 ms, original ON 184.6 ms, OFF 176.6 ms; - final ON/OFF result captures were byte-identical; - ClickBench 43/43: three-run median +1.8% (parity); - TPC-DS 99/99: median +0.7% (parity); - TPC-H 22/22: median -0.4% (parity/slightly faster); - all previously identified relevant regressions passed higher-iteration focused gates below 5%. I intentionally kept all of this broader work out of the #10776 branch. My current view is that the capped Auto construction is a focused fix for this issue, while streaming/adaptive direct output should be reviewed separately. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
