AP24110011695 opened a new issue, #4396:
URL: https://github.com/apache/iceberg-python/issues/4396
### PyIceberg version
main @ fce8185 (2026-11-15)
### Describe the bug
`data_file_statistics_from_parquet_metadata` returns `nan_value_counts={}`
unconditionally — the dict is initialized and returned but never populated.
Data files written by the pyarrow writer (and files imported through
`parquet_file_to_data_file`) therefore never report NaN counts, while the
`DataFile` schema and the scan-planning metrics evaluator both fully support
them.
Probe on a table containing 2 NaNs per column (written with
`pq.write_table`, metadata read back and passed through the function):
```
null_value_counts: {1: 0, 2: 1} # correct
nan_value_counts: {} # always empty
# true non-null NaNs: field 1 = 2, field 2 = 2
```
**Consequence** — conservative, not wrong results:
`_InclusiveMetricsEvaluator` uses `nan_counts` for pruning (`visit_is_nan`:
`nan_counts.get(field_id) == 0 → ROWS_CANNOT_MATCH`; `_contains_nans_only` for
`not_nan`/`less_than`/`greater_than`). With empty nan counts every check
degrades to `ROWS_MIGHT_MATCH`, so:
- `is_nan` filters can never prune files, even files with zero NaNs in that
column are always scanned,
- `not_nan`/range filters never prune all-NaN or NaN-heavy files (the
sensor-data and ML-feature use case),
- files written by parquet-mr/Java *do* carry `nan_value_counts` and get
that pruning — the gap only affects pyiceberg-written tables.
This looks adjacent to #4313 (drop bounds when a row group has no min/max)
and #4385 (omit incomplete null counts): the same statistics-completeness
family, except NaN counts are not partially accumulated — they are never
accumulated at all.
### Fix options
pyarrow 26's `ColumnStatistics` exposes no `nan_count`/`has_nan_count`, so a
row-group-level accumulation mirroring #4385 is not possible with the public
API today:
**A. Compute at write time** — the writer sees the arrow table before
`close()`; NaN counts per float column are one vectorized op
(`pa.compute.is_nan(col)` sums to the non-null NaN count, which matches the
spec's NaN-count semantics) and can be threaded into
`data_file_statistics_from_parquet_metadata` (or merged into its returned
`DataFileStatistics`).
**B. Upstream** — expose `nan_count` from the parquet statistics in pyarrow
(the thrift `Statistics` message carries it since parquet-format 2.5), then
mirror the #4385 accumulation including a completeness guard.
Happy to send A as a PR if that design is preferred.
AI disclosure: this report was compiled with AI assistance; every claim
above is reproducible from the probe snippet and the cited code paths
(`pyiceberg/io/pyarrow.py` `data_file_statistics_from_parquet_metadata`,
`pyiceberg/expressions/visitors.py` `_InclusiveMetricsEvaluator`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]