AarryaSaraf opened a new pull request, #10691:
URL: https://github.com/apache/arrow-rs/pull/10691
# Which issue does this PR close?
None. This adds missing benchmark coverage; no library code changes.
# Rationale for this change
Every dictionary-encoded case in `arrow_reader` decodes ~20 byte values:
`build_dictionary_encoded_string_page_iterator` builds `"Dictionary value
{x}"`
at 1% unique. At that size the decode is dominated by per-key overhead — RLE
index decoding and the per-key bounds check — and the cost of gathering the
dictionary values into the output buffer is negligible.
Binary columns holding large payloads are also commonly dictionary encoded,
and
there the balance inverts: a page carries a handful of very large values
behind
dictionary keys, and the gather dominates instead. No benchmark in the crate
covers that shape today, so changes to `OffsetBuffer::extend_from_dictionary`
have nothing to be measured against.
# What changes are included in this PR?
One generator and one case in the existing `BinaryArray` group:
- `build_dictionary_encoded_large_value_page_iterator` — 16 unique values of
64 KiB, 256 values per page, mandatory (no NULLs), otherwise the same
row-group and page geometry as the existing generators.
- `arrow_array_reader/BinaryArray/dictionary encoded, mandatory, no NULLs,
large values`
128 MiB decoded per iteration. Measured on `main` (Apple M-series):
```
arrow_array_reader/BinaryArray/dictionary encoded, mandatory, no NULLs,
large values
time: [9.8350 ms 9.9838 ms 10.118 ms]
```
# Are these changes tested?
This is a benchmark. It asserts its decoded value count on each run, as the
surrounding cases do.
# Are there any user-facing changes?
No.
# AI disclosure
This benchmark was drafted with AI assistance and reviewed by me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
