Punisheroot opened a new pull request, #50710: URL: https://github.com/apache/arrow/pull/50710
## What changed This changes dictionary-encoded `BYTE_ARRAY` decoding for Binary, String, LargeBinary, and LargeString builders to: - decode dictionary indices up front into the decoder's reusable scratch buffer; - validate the decoded index count and dictionary bounds before appending; - compute the exact decoded value-data length; - reserve that data capacity once and use the builder's unsafe append path. BinaryView and StringView builders keep the existing bounded streaming path, since short view values are stored inline and do not benefit from reserving the dictionary values' total byte length. The patch also adds dictionary decode benchmarks with 10% and 50% null density and a regression test for out-of-range and truncated dictionary-index streams. ## Why The dictionary path previously could not derive a useful data-size estimate from page metadata, so every value used the checked append path and the value buffer grew incrementally. Predecoding the indices provides the exact payload size while hoisting the bounds checks out of the append loop. The trade-off is one reusable `int32_t` scratch entry per decoded non-null value and an additional pass over the indices. The scratch allocation is retained for reuse across decode calls. This addresses the dictionary-path exact-reservation proposal in GH-50703. ## Benchmarks Windows MSVC Release build, 65,536 values, baseline/optimized/optimized/baseline order, five randomized repetitions per run: | Null density | Binary time improvement | BinaryView control | Improvement normalized to control | | ---: | ---: | ---: | ---: | | 0% | 21.9% | -1.2% | 23.4% | | 10% | 25.5% | -3.9% | 30.6% | | 50% | 17.2% | -0.5% | 17.7% | BinaryView is the unchanged-path control. These are local decoder microbenchmarks rather than an end-to-end scan of the reporter's 6.5 GB file. ## Validation - `parquet-encoding-test.exe --gtest_filter=DictEncodingAdHoc.DenseDecodeRejectsInvalidOrTruncatedIndices`: passed - complete `parquet-encoding-test.exe` suite: 136/136 passed - optimized dictionary decode benchmark smoke run: passed - `git diff --check`: passed ## AI assistance disclosure AI assistance was used for code exploration, implementation drafting, and test/benchmark iteration. I reviewed and validated the resulting changes and can explain and take ownership of them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
