Remenby31 opened a new pull request, #4313: URL: https://github.com/apache/iceberg-python/pull/4313
Closes #4312 # Rationale for this change `data_file_statistics_from_parquet_metadata` built a column's bounds from the row groups that report a min/max and silently skipped a row group whose statistics have no min/max. PyArrow writes such statistics when a value is larger than 4 KB. The stored bounds then exclude that row group's values, and the metrics evaluator prunes the file from scans that should match it. With this change, a row group that holds non-null values but reports no min/max drops the column's bounds for the file. The column keeps its null and value counts, because the row group still reports them. A row group whose values are all null still has no effect on the bounds. ## Are these changes tested? Yes. `test_bounds_absent_when_a_row_group_with_values_has_no_min_max` writes two row groups per column. One column has a 5,000-character value in its second row group, and the other column has an all-null first row group. The test fails without the change: the first column keeps the bound `b'a'`. `tests/io/test_pyarrow_stats.py` and the unit tests of `tests/io/test_pyarrow.py` pass, and `prek` passes on the changed files. ## Are there any user-facing changes? Data files written from now on have no bounds for such a column, so scans no longer skip their matching rows. Files already written keep their bounds. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
