singhpratech opened a new issue, #51491:
URL: https://github.com/apache/arrow/issues/51491
### Describe the bug, including details regarding any error messages,
version, and platform.
Filtering a Parquet dataset on a float or double column can drop rows that
contain NaN. The row group is skipped based on its min/max statistics, but NaN
is not part of min/max.
```python
import pyarrow as pa, pyarrow.parquet as pq, pyarrow.dataset as ds,
pyarrow.compute as pc
t = pa.table({"x": pa.array([5.0, float("nan"), 5.0, 5.0])})
pq.write_table(t, "a.parquet") # statistics: min 5.0,
max 5.0
pc.filter(t, pc.not_equal(t["x"], 5.0)) # [nan]
pq.read_table("a.parquet", filters=[("x", "!=", 5.0)]) # [] -- row lost
pq.write_table(t, "b.parquet", write_statistics=False)
pq.read_table("b.parquet", filters=[("x", "!=", 5.0)]) # [nan]
u = pa.table({"x": pa.array([1.0, float("nan"), 10.0])})
pq.write_table(u, "c.parquet") # statistics: min 1.0,
max 10.0
ds.dataset("c.parquet").to_table(filter=~(ds.field("x") <= 10.0)) # [] --
in memory: [nan]
```
The same happens for `~(x == 5)`, `is_null(x, nan_is_null=True)` and
`x.isin([nan])`, and for `pc.is_nan(x)` when min == max and the column has no
nulls. Filters that are false for NaN (`<`, `<=`, `>`, `>=`, `==`) are not
affected.
Cause: `ParquetFileFragment::EvaluateStatisticsAsExpression`
(cpp/src/arrow/dataset/file_parquet.cc) turns the statistics into the guarantee
`x == min` when min == max, or `x >= min and x <= max`, optionally `or
is_null(x)`. Writers do not include NaN in min/max, so the guarantee does not
hold for NaN rows. `SimplifyWithGuarantee` then simplifies `x != 5`, `invert(x
<= 10)` and the others to false and the row group is skipped. GH-28074 (#15125)
handled NaN written as min or max, but not NaN values outside them.
Expected: the same rows as filtering in memory or reading without
statistics. Row groups can still be skipped for filters that are false for NaN.
Environment: pyarrow 25.0.1, macOS 26.6.2 arm64, Python 3.13. Also
reproduced on C++ main at 1f789ffd70.
### Component(s)
C++, Parquet, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
