sdf-jkl opened a new pull request, #11027: URL: https://github.com/apache/arrow-rs/pull/11027
# Which issue does this PR close? - Draft performance follow-up to #2276 and #2278. An issue can be filed and linked before marking this ready. # Rationale for this change `BitReader::get_batch` currently performs runtime bit-width dispatch for every complete native unpack block. For larger batches, this repeatedly enters the same jump table even though the bit width is constant for the entire call. # What changes are included in this PR? - Add multi-block unpack functions that dispatch on bit width once, then loop over const-generic unpack blocks. - Add a provided `FromBitpacked::unpack_batches` fallback and optimized implementations for the built-in integer, signed integer, and boolean types. - Route multiple complete blocks through the batched path while retaining the existing direct path for exactly one block and all existing smaller-block and scalar tail handling. - Preserve the zero-width behavior without allocating intermediate buffers. This is intentionally limited to unpacking. The related packing work remains separate. # Are these changes tested? Yes. Existing `get_batch` tests exhaustively cover supported bit widths, signed and unsigned integer types, booleans, zero width, and batch sizes including 128 and 129 values that exercise the multi-block path. Validated locally with: - `cargo test -p parquet --features experimental --lib` (1,374 passed) - `cargo clippy -p parquet --all-targets --features experimental -- -D warnings` - workspace `cargo fmt` check - the separate Parquet `cargo fmt` check Benchmark results will be added before this draft is marked ready. Earlier exploratory results were collected from a mixed development branch and are deliberately not attributed to this smaller patch. # Are there any user-facing changes? No behavior changes are intended. `FromBitpacked` gains a provided method, so existing implementations remain source-compatible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
