rahulsmahadev opened a new pull request, #4346:
URL: https://github.com/apache/iceberg-python/pull/4346
# Rationale for this change
Scans drop NaN rows for predicates that NaN satisfies: `IsNaN`,
`NotEqualTo`, `NotIn`, and a NOT over a float comparison. Parquet statistics
exclude NaN, so a row group whose non-NaN values are all equal reports min ==
max. pyarrow prunes that row group for `is_nan(x)`, which is false for the
value 7.0, and the scan never returns the NaN row:
```python
# tbl: table with id (long) and x (double)
tbl.append(pa.table({"id": pa.array([0, 1], pa.int64()), "x":
pa.array([float("nan"), 7.0], pa.float64())}))
tbl.scan(row_filter=IsNaN("x")).to_arrow() # 0 rows; expected the row (0,
NaN)
```
The trigger is a float or double row group with no nulls whose non-NaN
values are all equal. Nulls in the same row group avoid it.
This change keeps NaN-sensitive predicates away from Parquet pruning.
`_task_to_record_batches` pushes down only the top-level AND conjuncts that
cannot hide NaN rows. A conjunct is NaN-sensitive if it is `IsNaN`,
`NotEqualTo`, or `NotIn` on a float or double column, or any predicate on such
a column under a NOT. When one is present, each batch is filtered with the full
expression after the read, as already happens for scans with positional
deletes. A NOT over a non-floating column keeps full pushdown.
Trade-off: NaN-sensitive conjuncts lose Parquet row-group pruning, and other
conjuncts keep it. Java's row-group filter takes the same conservative view:
for `notEq` and `notIn` it never drops a row group, and for `isNaN` it drops
one only when the column is all null (`ParquetMetricsRowGroupFilter`).
## Are these changes tested?
Yes. `test_task_to_record_batches_keeps_nan_rows_for_nan_sensitive_filters`
in `tests/io/test_pyarrow.py` covers `IsNaN`, `NotEqualTo`, `NotIn`, AND, OR, a
NOT over a float comparison, a NOT over an integer column, and `NotNaN`.
`test_scan_nan_row_with_equal_finite_statistics` in
`tests/catalog/test_catalog_behaviors.py` runs the public scan path on the
`memory`, `sql`, and `sql_without_rowcount` catalogs. Both fail without the fix
and pass with it.
## Are there any user-facing changes?
Bug fix only: scans return the NaN rows they previously dropped. No API
changes.
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
