dwsmith1983 opened a new issue, #12017: URL: https://github.com/apache/arrow-rs/issues/12017
### Is your feature request related to a problem or challenge? Reading Parquet list columns spends a large share of its time in `ListArrayReader::consume_batch`, which walks the definition and repetition levels and builds the list offsets and validity with a `match` per level. When NULLs, empty lists and values are mixed, those branches mispredict often, and a nested list pays that cost once per level of nesting. In a callgrind profile of the `arrow_array_reader/ListArray` benches on main, `consume_batch` accounts for 35% to 55% of the instructions in the read loop. Found while working on apache/datafusion-comet#6921, where nested lists in Iceberg tables make this path hot. ### Describe the solution you'd like Build the offsets and validity without branching on the level values: compute each level's contribution with boolean arithmetic, write offsets by index into a pre-sized buffer, and pack validity bits into a `u64` that is appended with `BooleanBufferBuilder::append_word`. Under callgrind this lowers the instruction count slightly and cuts branch mispredicts in the read loop by 34% to 92% across the ListArray benches. ### Describe alternatives you've considered None beyond the current loop. ### Additional context There is no bench for nested lists yet; a `List<List<Int32>>` case would cover the path where a parent threshold applies. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
