Rich-T-kid opened a new pull request, #11918: URL: https://github.com/apache/arrow-rs/pull/11918
# Why Follow-up to #11732 and #11867. Boolean coalescing currently materializes temporary filtered value and validity buffers before appending them to the in-progress builders. Now that NullBufferBuilder supports packed-word appends, the filtering and append steps can be fused. # What changed - Compress each 64-bit source word with the normalized filter mask. - Pack the selected value and validity bits into complete output words before appending them directly to the coalescer builders. - Preserve lazy validity allocation for non-null inputs. - Add coverage for sliced sources, unaligned output positions, nullable and non-null inputs, optimized and unoptimized predicates, word boundaries, empty and full selections, and oversized filters. # Results Apple M4 Max arm64, existing single_boolean coalesce filter matrix. Each value is the geometric mean of two Criterion medians using baseline to candidate to candidate to baseline ordering. Lower is better. | Nulls | Selectivity | Main | Changed | Delta | |---:|---:|---:|---:|---:| | 0% | 0.001 | 14.694 ms | 8.812 ms | -40.03% | | 0% | 0.005 | 2.968 ms | 2.124 ms | -28.45% | | 0% | 0.01 | 1.523 ms | 1.214 ms | -20.30% | | 0% | 0.1 | 460.4 us | 351.8 us | -23.60% | | 0% | 0.8 | 62.93 us | 60.82 us | -3.35% | | 10% | 0.001 | 22.606 ms | 9.721 ms | -57.00% | | 10% | 0.005 | 4.722 ms | 2.526 ms | -46.52% | | 10% | 0.01 | 2.482 ms | 1.570 ms | -36.74% | | 10% | 0.1 | 837.7 us | 645.5 us | -22.95% | | 10% | 0.8 | 115.7 us | 114.6 us | -0.96% | The equal-weight geometric-mean improvement across the matrix is 30.05%. The densest nullable case is effectively flat; its two controlled deltas were -1.84% and -0.06%. # Test coverage - cargo test -p arrow-select: 478 unit tests and 20 doctests passed. - cargo clippy -p arrow-select --all-targets -- -D warnings. - cargo +stable fmt --all -- --check. - git diff --check. - The pre-push workspace formatting, Clippy, and typo checks passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
