viirya opened a new pull request, #12024: URL: https://github.com/apache/arrow-rs/pull/12024
# Which issue does this PR close? - Closes #12023 # Rationale for this change `BooleanBufferBuilder::append_slice` sets one bit at a time behind a data-dependent branch. This is the path the Parquet reader takes to build `BooleanArray` values (`BooleanBuffer::from(&[bool])`), and it dominates reading BOOLEAN columns. See the issue for details. # What changes are included in this PR? `append_slice` now: 1. sets bits individually only until the write position is byte aligned (at most 7 values), 2. packs each following group of 8 bools into one byte with `u64::from_le_bytes(bytes).wrapping_mul(0x0102_0408_1020_4080) >> 56`, writing whole bytes, and 3. sets the remaining fewer than 8 values individually. The multiply works because each input byte is 0 or 1 (`bool as u8`): byte `i` lands on bit `56 + i`, and no two partial products overlap, so there are no carries. `from_le_bytes` makes the result the same on big-endian targets. Whole bytes are written only past the previous length, and those bytes were just zero-filled by `advance`, so overwriting them is safe, including after `truncate`. Also adds a `boolean_buffer_builder` criterion bench to arrow-buffer. # Are these changes tested? Yes. Two new unit tests: - `test_append_slice_all_byte_patterns` checks all 256 patterns of 8 bools. - `test_append_slice_offsets` checks every starting bit offset 0..16 against slice lengths 0..100, starting after a `truncate`, and followed by another `append`. Benchmarks, Apple M-series (aarch64), back-to-back runs of `main` and this PR, two rounds each: `cargo bench -p arrow-buffer --bench boolean_buffer_builder` (64Ki bools per call; offset 3 starts mid-byte) | input | offset | main | this PR | change | |---|---|---|---|---| | random | 0 | 164.0 µs | 2.35 µs | −98.6% | | random | 3 | 165.1 µs | 2.33 µs | −98.6% | | 90% true | 0 | 74.8 µs | 2.33 µs | −96.9% | | 90% true | 3 | 75.9 µs | 2.33 µs | −96.9% | | all true | 0 | 94.6 µs | 2.33 µs | −97.5% | | all true | 3 | 94.8 µs | 2.33 µs | −97.5% | The existing `bench_bool` in `arrow/benches/builder.rs` (`BooleanBuilder::append_slice`, 64 batches of 8Ki random bools): | bench | main | this PR | change | |---|---|---|---| | `bench_bool` | 761–773 µs | 19.6 µs | −97.4% | End to end, reading 1M rows of an optional BOOLEAN column (10% nulls, random values) from an in-memory Parquet file through `ParquetRecordBatchReader`: 4.95 ms → 2.10 ms (−57.5%). This uses a scenario benchmark I'll propose in a separate PR. # Are there any significant user-facing changes? No. This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
