Anurag-S1ngh opened a new pull request, #4401: URL: https://github.com/apache/iceberg-python/pull/4401
## Summary Data files written by the PyArrow writer always had an empty `nan_value_counts` map, because the counts were never computed. The scan planner supports NaN counts, so files written by pyiceberg could never be pruned on `is_nan`, `not_nan`, or range filters, while files written by parquet-mr could. ## Changes - Count the non-null NaNs of the top-level float and double columns from the Arrow table the writer receives, since PyArrow does not expose a NaN count on its Parquet column statistics. - Accumulate the counts across batches and merge them into the `DataFileStatistics` returned by `ParquetFormatWriter.close()`. - Leave a column out of the map when it cannot be counted, which only means the file cannot be pruned on NaN counts. Nested float and double columns are not counted yet; that needs flattening through lists, structs, and maps and can follow separately. ## Testing - Added a test asserting the NaN counts of a written file, and one asserting `d IS NAN` now prunes a file without NaNs. - `make test` passes (4694 passed, 5 skipped). Closes #4396 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
