Shockp opened a new pull request, #51109:
URL: https://github.com/apache/arrow/pull/51109
### Rationale for this change
Definition levels in the Parquet column reader are currently decoded into
a buffer and then scanned separately with `std::count` to determine how
many physical values need to be decoded.
This adds an extra pass over the decoded definition levels.
For RLE-encoded levels, the number of matching values can often be
determined directly from the run value and run length while the output is
being materialized.
### What changes are included
- Add `GetBatchAndCount` support to the RLE and bit-packed decoders.
- For repeated RLE runs, compute the matching count directly from run
metadata without scanning the decoded output.
- For bit-packed runs, count the values immediately after unpacking them.
- Add `LevelDecoder::DecodeAndCount`, preserving the existing level
validation.
- Use the fused decode-and-count operation when reading Parquet definition
levels.
- Remove the separate `std::count` pass from the column reader.
- Add targeted benchmarks comparing the separate and fused approaches.
### Benchmarks
`parquet-column-reader-benchmark`, pinned to one CPU, 20 repetitions.
| Encoding | Max level | Batch | Repeat | Separate | Fused | Change |
|---|---:|---:|---:|---:|---:|---:|
| RLE | 1 | 1024 | 1 | 5555.5 ns | 5383.5 ns | 3.10% faster |
| RLE | 1 | 1024 | 7 | 13464.7 ns | 10972.1 ns | 18.51% faster |
| RLE | 1 | 1024 | 1024 | 4111.3 ns | 971.7 ns | 76.36% faster |
| RLE | 3 | 1024 | 1 | 5500.5 ns | 5003.0 ns | 9.04% faster |
| RLE | 3 | 2048 | 1 | 5384.0 ns | 4778.2 ns | 11.25% faster |
| RLE | 3 | 1024 | 7 | 14184.0 ns | 11833.4 ns | 16.57% faster |
| BitPack | 1 | 1024 | 1 | 4001.3 ns | 3953.2 ns | 1.20% faster |
| BitPack | 1 | 1024 | 7 | 3996.6 ns | 3949.6 ns | 1.18% faster |
| BitPack | 1 | 1024 | 1024 | 3974.4 ns | 3988.0 ns | 0.34% slower |
| BitPack | 3 | 1024 | 1 | 3940.1 ns | 3974.0 ns | 0.86% slower |
| BitPack | 3 | 2048 | 1 | 3895.6 ns | 3881.6 ns | 0.36% faster |
| BitPack | 3 | 1024 | 7 | 3956.2 ns | 3919.4 ns | 0.93% faster |
The improvement increases with RLE run length because matching values can
be counted from run metadata instead of rescanning the materialized output.
The bit-packed path remains approximately neutral.
Closes #48561.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
