Copilot commented on code in PR #3918:
URL: https://github.com/apache/iceberg-python/pull/3918#discussion_r3948136590
##########
pyiceberg/io/pyarrow.py:
##########
@@ -919,7 +919,10 @@ def visit_equal(self, term: BoundTerm, literal:
Literal[Any]) -> pc.Expression:
return pc.field(self._get_field_name(term)) ==
_convert_scalar(literal.value, term.ref().field.field_type)
def visit_not_equal(self, term: BoundTerm, literal: Literal[Any]) ->
pc.Expression:
- return pc.field(self._get_field_name(term)) !=
_convert_scalar(literal.value, term.ref().field.field_type)
+ # A null is not equal to the literal, but an Arrow comparison yields
null for it and
+ # the row would be dropped. Keep it explicitly to match the other
evaluators.
+ ref = pc.field(self._get_field_name(term))
+ return ref.is_null(nan_is_null=False) | (ref !=
_convert_scalar(literal.value, term.ref().field.field_type))
Review Comment:
`visit_not_equal` now explicitly treats nulls as matching (via `is_null(...)
or (ref != ...)`). This changes the semantics of a delete/overwrite filter like
`col != X`: null rows should now be considered *matching* the filter (and
therefore be removed/replaced). However, `_expression_to_complementary_pyarrow`
currently preserves nulls for predicates that the
`_NullNaNUnmentionedTermsCollector` considers "null-unmentioned" (including
`BoundNotEqualTo`), which will cause the complementary/preserve filter used
during rewrite deletes/overwrites to incorrectly keep null rows for `!=`
filters.
To keep delete/overwrite behavior consistent with the new `NotEqualTo`
semantics, update `_NullNaNUnmentionedTermsCollector.visit_not_equal` (and
likely `visit_not_in` for the same reason) so these predicates are treated as
explicitly handling nulls (and NaNs if applicable), preventing
`_expression_to_complementary_pyarrow` from OR-ing `is_null`/`is_nan` back into
the preserve filter. Adding a focused unit/integration test for
overwrite/delete with a nullable column and a `!=` row_filter would help
prevent regressions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
