Jo2234 opened a new pull request, #10368: URL: https://github.com/apache/arrow-rs/pull/10368
# Which issue does this PR close? - Closes #5097. # Rationale for this change `zip` currently uses `MutableArrayData`, which is efficient when a mask contains long runs but performs one extension per run. Fragmented array/array masks therefore pay substantial per-run overhead, while the specialized interleave kernels are faster for this shape. # What changes are included in this PR? - Count true runs in large masks and route fragmented array/array inputs to `interleave` - Keep the existing `MutableArrayData` path for inputs shorter than 1,024 rows, scalar inputs, and masks with fewer runs - Treat null mask entries as false before dispatch, preserving existing `zip` semantics - Extend the zip benchmark with a direct interleave comparison and a contiguous `true_then_false` case The run-count threshold is conservative: interleave is selected only above one true run per eight rows. Counting the complete bitmap also avoids sampling errors for masks whose fragmentation is unevenly distributed. # Are these changes tested? Yes. The new tests cover dispatch decisions for short, fragmented, sparse/dense, contiguous, unevenly fragmented, and offset masks, plus end-to-end fragmented array selection with null mask and input values. Validation: - `CARGO_BUILD_JOBS=2 cargo test -p arrow-select --all-features` (393 unit tests and 17 doctests passed) - `CARGO_BUILD_JOBS=2 cargo clippy -p arrow-select --all-targets --all-features -- -D warnings` - `CARGO_BUILD_JOBS=2 cargo clippy -p arrow --bench zip_kernels --features=async,test_utils -- -D warnings` - `cargo +stable fmt --all -- --check` - `CARGO_BUILD_JOBS=2 RUSTDOCFLAGS='-D warnings' cargo doc -p arrow-select --no-deps` Criterion used 30 samples, a 2-second warm-up, and a 5-second measurement. On 8,192-row array/array inputs with random 50%-true masks, the existing run path was forced by temporarily raising the local dispatch threshold; the benchmark-only control was then restored: | Input | Run path | Adaptive | Improvement | |---|---:|---:|---:| | i32 | 76.889 us | 33.676 us | 56.20% | | short strings | 101.600 us | 69.580 us | 31.52% | | long strings | 214.880 us | 125.560 us | 41.57% | | short bytes | 103.680 us | 70.169 us | 32.32% | | long bytes | 218.090 us | 123.040 us | 43.58% | | short string views | 85.114 us | 38.201 us | 55.12% | | longer string views | 84.633 us | 49.308 us | 41.74% | At the 1,024-row boundary, i32 measured 14.350 us on the forced run path versus 4.6005 us on the adaptive path. The contiguous `true_then_false` case remains on the run-oriented path. AI assistance was used to draft parts of the adaptive dispatch, tests, and benchmark changes. The resulting diff was reviewed against the `zip`, `MutableArrayData`, `BooleanBuffer`, and `interleave` implementations; an edge-sampling flaw found during review was replaced with a complete run count and covered by a regression test. The checks and benchmark results above were rerun on the final implementation. # Are there any user-facing changes? No API or semantic changes. Fragmented array/array `zip` calls use a faster internal implementation; scalar, short, and contiguous-mask calls retain the existing path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
