rahulsmahadev commented on code in PR #4346:
URL: https://github.com/apache/iceberg-python/pull/4346#discussion_r4543070550
##########
pyiceberg/io/pyarrow.py:
##########
@@ -1721,6 +1733,29 @@ def _get_column_projection_values(
return projected_missing_fields
+def _has_nan_sensitive_predicate(expr: BooleanExpression, negated: bool =
False) -> bool:
+ """Return whether a predicate in the expression can match NaN values that
pyarrow may prune away.
+
+ Parquet statistics exclude NaN, so a row group whose non-NaN values are
all equal reports min == max.
+ pyarrow then prunes it for predicates that the NaN values satisfy: IsNaN,
NotEqualTo and NotIn on a
+ float or double column, and any predicate on such a column under a NOT.
+ """
+ if isinstance(expr, (And, Or)):
+ return _has_nan_sensitive_predicate(expr.left, negated) or
_has_nan_sensitive_predicate(expr.right, negated)
+ if isinstance(expr, Not):
Review Comment:
Opened #4350 for the `rewrite_not` change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
