hanke580 opened a new issue, #26490:
URL: https://github.com/apache/datafusion/issues/26490
### Describe the bug
Parquet writers leave NaN out of floating-point min/max statistics (the spec
requires it). So a column chunk holding
one distinct finite value plus some NaNs has statistics `min == max == x`
and `null_count == 0`, although its values are
not constant. DataFusion treats such a column as constant and substitutes
the statistic for the data. A plain
`SELECT * FROM 'file.parquet'` then returns `x` for every NaN row. There is
no error, and NaNs silently disappear.
pyarrow, DuckDB and Polars read the same file correctly.
This is not limited to pruning: `datafusion.execution.parquet.pruning =
false` and `enable_page_index = false` do not
change the result. `datafusion.execution.collect_statistics = false` does. A
`COPY … TO` from such a file writes the
wrong values into the new files.
The shortcut is `constant_columns_from_stats` / `constant_value_from_stats`
in
`datafusion/datasource-parquet/src/opener/mod.rs` (lines 1968 and 1999,
called at line 891): `min == max &&
!min.is_null() && null_count == Exact(0)`. For Float16/32/64, Parquet
min/max cannot prove the column is constant, because NaN is excluded from them.
So floating-point columns should not be treated as constant from these
statistics (unless a NaN count of 0 is known).
Draft PR #24804 (for #15812, pruning with NaN) would disable floating-point
min/max in a way that also covers this
("constant-column substitution"), but it is a draft and has no issue for
this user-visible symptom. I'm filing it so
the wrong-result case is tracked and tested.
### To Reproduce
```python
import pyarrow as pa, pyarrow.parquet as pq
pq.write_table(pa.table({'c0': pa.array([float('nan'), -8.6e-27])}),
'nan.parquet')
pq.write_table(pa.table({'c0': pa.array([1.5, float('nan')],
pa.float32())}), 'nan_f32.parquet')
pq.write_table(pa.table({'c0': pa.array([float('nan'), -8.6e-27])}),
'nan_nostats.parquet', write_statistics=False)
```
```
$ datafusion-cli -q -c "SELECT * FROM 'nan.parquet';" -c "SELECT count(*)
FILTER (WHERE isnan(c0)) AS nans FROM 'nan.parquet';"
+----------+
| c0 |
+----------+
| -8.6e-27 |
| -8.6e-27 | <-- this row is NaN in the file
+----------+
+------+
| nans |
+------+
| 0 |
+------+
$ datafusion-cli -q -c "SELECT * FROM 'nan_f32.parquet';"
| 1.5 |
| 1.5 | <-- NaN in the file
$ datafusion-cli -q -c "SELECT * FROM 'nan_nostats.parquet';"
| NaN |
| -8.6e-27 | <-- correct without statistics
$ datafusion-cli -q -c "SET datafusion.execution.collect_statistics =
false;" -c "SELECT * FROM 'nan.parquet';"
| NaN |
| -8.6e-27 | <-- correct
```
The same happens for files written by DataFusion's own `COPY`. `[NaN, 2.0,
3.0]` (min ≠ max) and `[NaN, NaN]` (no min/max
written) are read correctly.
Both column orders are affected. The pyarrow files above have
`TYPE_DEFINED_ORDER` and no `nan_count`. DataFusion/arrow-rs
60 writes `IEEE_754_TOTAL_ORDER` with `nan_count`, but still leaves NaN out
of min/max: `COPY (VALUES ('NaN'::double), (5.0))`
gives `min = max = 5.0, null_count = 0, nan_count = 1`, and that file is
read back as `5.0, 5.0`. So DataFusion ignores a
`nan_count` that the file does carry.
### Expected behavior
`SELECT * FROM 'nan.parquet'` returns `NaN, -8.6e-27`, as pyarrow, DuckDB
and Polars do. Float statistics with `min == max`
should not be used to replace the column by a constant, because NaN values
are not reflected in them.
### Additional context
- DataFusion main 290ad15 (2026-10-18), `datafusion-cli 55.1.0` built from
source; also 91cd7ffb (2026-10-15). pyarrow 26.0.0.
- Shapes like "one default value plus missing values as NaN" are common in
sensor and ML feature data.
- Related: #15812 and draft PR #24804.
### Willingness to contribute
None
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
