mightsleep opened a new issue, #11963: URL: https://github.com/apache/arrow-rs/issues/11963
### Describe the bug For list (repeated) columns, the Parquet writer can get the `null_pages` flags of the column index wrong. It decides that a page is all null by comparing its null count with its row count, but for lists the null count is per element, not per row. As a result: - a page whose lists each hold one null and one value (`[null, 7]`) is written as an all-null page, and its min/max are lost; - a page whose lists hold only nulls (`[null, null]`) is not recognised as a null page, and the writer then drops the column index for the whole column. Flat columns are not affected. ### To Reproduce Write a list<int32> column where every row is `[null, 7]`, and read its column index: page 0 is flagged as a null page and has no min/max. Write a column where every row is `[null, null]`: the file has no column index for that column. Both cases are tests in #11828. ### Expected behavior A page is a null page only when every element in it is null. Pages with any values keep their min/max, and all-null pages keep the column index intact. ### Additional context Nobody prunes list columns with the column index yet, so the wrong flags go unnoticed today. But pages of a list column always start at a row boundary, so the index could be used to skip pages for predicates on list elements, such as `array_has(tags, 'x')` or `ANY(values) > 10`. That only works if `null_pages` and min/max are right, and files written now carry these flags, so the sooner the writer is fixed, the better. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
