AP24110011695 opened a new pull request, #4412:
URL: https://github.com/apache/iceberg-python/pull/4412
## What's changed
Implements design A from #4396: `data_file_statistics_from_parquet_metadata`
returned an empty `nan_value_counts` dict unconditionally, so data files
written by the pyarrow writer never reported NaN counts — and NaN-aware scan
pruning stayed silently unavailable for them (`_InclusiveMetricsEvaluator`
degrades to `ROWS_MIGHT_MATCH` without nan counts, for `is_nan`, `not_nan` and
range predicates via `_contains_nans_only`).
pyarrow's statistics API does not expose NaN counts (checked on 26.0.1 — no
`nan_count`/`has_nan_count`), so the writer now counts them from the written
table itself:
- `ParquetFormatWriter.write()` accumulates `pa.compute.is_nan(...)` sums
per top-level float/double field (`is_nan` is null on nulls and `sum` skips
nulls — matching the spec's NaN-count semantics of counting non-null NaN values)
- the counts flow through a new optional `table_nan_value_counts` parameter
of `data_file_statistics_from_parquet_metadata`
- the metadata-only caller (`parquet_file_to_data_file`, used by
`add_files`) keeps its previous behavior — no NaN counts from metadata-only
input
Nested float fields keep the previous (empty) behavior for now; the metrics
plan and evaluator keys make them a natural follow-up once the top-level path
is settled.
## Test
`test_nan_value_counts_are_written` (mixed values + null) and
`test_nan_value_counts_are_written_across_row_groups` (all-NaN, two `write()`
calls) — both verified failing-first on main (`nan_value_counts == {}`) and
passing with the change; the second also pins the pruning win (`not_nan` now
prunes an all-NaN file via `_contains_nans_only`). Full
`tests/io/test_pyarrow_stats.py` (22) green; the pre-existing `azure`-fixture
failures in `test_pyarrow.py` are unrelated (missing optional dep in this
environment).
AI disclosure: drafted with AI assistance, verified locally by running the
failing-first cycle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]