dominikandreasseitz opened a new pull request, #4068: URL: https://github.com/apache/iceberg-python/pull/4068
Closes #4067 # Description Fixes `IsNull`/`NotNull` binding silently dropping rows/files for a required field nested under an optional ancestor struct. Related iceberg java/rust PRs: * [apache/iceberg#14270](https://github.com/apache/iceberg/pull/14270) * [apache/iceberg-rust#3292](https://github.com/apache/iceberg-rust/issues/3292) **Bug:** - `BoundIsNull`/`BoundNotNull.__new__` folded to `AlwaysFalse()`/`AlwaysTrue()` using only a field's own `required` flag. - A `required` leaf can still be absent if an *optional ancestor struct* containing it is null - `plan_files()` silently dropped matching files once `IsNull`/`NotNull` hit a required field under an optional parent. **Example:** ```python # schema: s is optional, its only child x is required # s = None for the one row -> s.x is absent too files = table.scan(row_filter=IsNull("s.x")).plan_files() len(files) # 0 <-- empty, should contain the file with the s = None row ``` **New behavior:** - `IsNull`/`NotNull` on a required field nested under an optional ancestor now correctly matches rows/files instead of silently pruning them. - No public API changes. ## Change - `pyiceberg/schema.py` — add `Schema.is_field_required_in_path(field_id)`, which walks the existing ancestor index to check the field *and every ancestor* is required. Mirrors Java's `TypeUtil.ancestorFields`/`allAncestorFieldsAreRequired`. - `pyiceberg/expressions/__init__.py` — remove the leaf-only fold from `BoundIsNull`/`BoundNotNull.__new__`; do the fold in `IsNull.bind()`/`NotNull.bind()` instead, matching where Java does it (`UnboundPredicate.bindUnaryOperation`) — no change to `BoundReference`. - `tests/test_schema.py`, `tests/expressions/test_expressions.py` — coverage for the fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
