wgtmac opened a new pull request, #55197: URL: https://github.com/apache/arrow/pull/55197
## Summary Parquet LIST columns that are effectively fixed-length currently pay the full Dremel reconstruction cost on every read. This change adds a page-level fixed-length detector and a direct child-value path for vector-shaped LIST data. The reader validates all-present definition levels and the periodic repetition-level pattern, derives the list width, then reads values directly across pages and constructs offsets with a fixed stride. The detector handles V1/V2 page boundaries, including pages that continue a list from the previous page. The optimization is controlled by `ArrowReaderProperties::set_fixed_list_fast_path()` and is disabled by default. If enabled for a variable-width LIST and a later page changes width after values have been consumed, the reader returns an explicit error instructing callers to disable the property. ## Validation - `make -C cpp/build-release -j8 parquet` - `make -C cpp/build-release -j8 parquet_shared` - `make -C cpp/build-release -j8 parquet-benchmarks` - 158 relevant LIST/FixedSizeList reader tests passed - Release Google Benchmark built with Conan `benchmark/1.9.4` ## Benchmark Release, ZSTD, three repetitions, 4 MiB float payload per benchmark case: - ordinary LIST, PLAIN, width 16: 4.60 ms vs 11.15 ms, 2.42x - ordinary LIST, BSS, width 768: 0.94 ms vs 5.35 ms, 5.69x - FixedSizeList, PLAIN, width 768: 5.80 ms vs 8.58 ms, 1.48x - FixedSizeList, BSS, width 768: 2.55 ms vs 5.29 ms, 2.08x - ALP remains decode-dominated: ordinary LIST width 768 is 1.34x The detailed matrix and larger Hardwood-sized stress results are in `parquet_vector_reader_optimization.md`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
