kszucs opened a new pull request, #131: URL: https://github.com/apache/parquet-testing/pull/131
Adds a file whose column chunk contains data pages with `num_values = 0`, the first page and one in the middle. Readers must skip such pages: arrow-rs and DuckDB read the file correctly, while Arrow C++ (pyarrow 26.0.0) treats the first such page as the end of the column chunk and silently returns 0 of the 3 rows. ### Why Such files exist in the wild: with content-defined chunking enabled, parquet-rs 58.1.0 to 60.1.0 writes an empty data page whenever a row group starts exactly on a chunk boundary, due to a bug in its writer (apache/arrow-rs#10929, fixed by apache/arrow-rs#11003). Nothing in the format requires a data page to hold values; only the OffsetIndex rules them out, since it requires strictly increasing `first_row_index` values, so this file has no page index. The existing files don't cover this, their data pages have values: `page_v2_empty_compressed.parquet` has a data page of 10 nulls (after an empty dictionary page) and `datapage_v2_empty_datapage.snappy.parquet` a data page of 1 null. Since all values are null, their encoded values sections are empty, which is what those files test with compression. ### Contents - `data/empty_data_pages.parquet`: a required INT64 column with the DataPageV1 pages `[]`, `[1, 2]`, `[]`, `[3]`, so it reads as `1, 2, 3`. - `data/empty_data_pages.cc`: the generator. Writers don't emit such pages on purpose, so it writes them one by one with the Arrow C++ `PageWriter` API. ### Verifying the layout `parquet-layout` from parquet-rs 58.x prints the page headers (from 59.0.0 on it only lists pages through the page index): ``` cargo install [email protected] --features cli --bin parquet-layout parquet-layout data/empty_data_pages.parquet ``` It shows four `data_page_v1` pages with `num_values` 0, 2, 0, 1 and `row_count: 3`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
