haohuaijin opened a new pull request, #10968: URL: https://github.com/apache/arrow-rs/pull/10968
# Which issue does this PR close? - Closes #10967. # Rationale for this change The mask-backed `RowSelection::and_then` implementation has high per-selected-row overhead for large, dense masks. Expanding the inner mask one word at a time is substantially faster for these inputs. # What changes are included in this PR? - Add a word-at-a-time dense mask expansion path. - Keep the existing set-index path for short or sparse masks. - Select the dense path for masks with at least 8,192 rows, approximately 75% outer selectivity, and approximately 5% inner selectivity. - Add unit tests and Criterion benchmark sweeps for random and clustered masks. # Are these changes tested? Yes. Tests cover bitmap offsets, threshold boundaries, and exhaustive 8-bit deposit inputs. ```console cargo test -p parquet --lib arrow::arrow_reader::selection::algebra::tests test result: ok. 22 passed; 0 failed ``` The new production code has 100% executable line and region coverage in the targeted LLVM coverage run. Benchmarks were run on an Apple M4 with Rust 1.97.1. The same benchmark harness was used for `main` and this PR. The sweep contains 62 cases. All 36 cases that select the dense path improved, with mean execution time reduced by 30.0% to 97.9%. | Distribution | Rows | Outer | Inner | `main` | PR | Change | |---|---:|---:|---:|---:|---:|---:| | Pseudo-random | 8,192 | 75% | 5% | 3,710 ns | 2,570 ns | -30.72% | | Clustered | 8,192 | 75% | 5% | 793 ns | 532 ns | -32.97% | | Pseudo-random | 8,192 | 99% | 99% | 16,377 ns | 546 ns | -96.67% | | Clustered | 65,536 | 99% | 99% | 129,890 ns | 2,699 ns | -97.92% | # Are there any user-facing changes? No. This is an internal performance optimization with no API or behavior changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
