dominikandreasseitz opened a new issue, #4067: URL: https://github.com/apache/iceberg-python/issues/4067
### Apache Iceberg version main (development) ### Please describe the bug 🐞 ### Description `IsNull`/`NotNull` binding silently drops rows/files for a required field nested under an optional ancestor struct. Related iceberg java/rust issues/PRs: * [apache/iceberg#14270](https://github.com/apache/iceberg/pull/14270) * [apache/iceberg-rust#3292](https://github.com/apache/iceberg-rust/issues/3292) **Bug:** - `BoundIsNull`/`BoundNotNull.__new__` folds to `AlwaysFalse()`/`AlwaysTrue()` using only a field's own `required` flag. - A `required` leaf can still be absent if an *optional ancestor struct* containing it is null - `plan_files()` silently drops matching files once `IsNull`/`NotNull` hits a required field under an optional parent. ### Example ```python from pyiceberg.catalog.sql import SqlCatalog from tempfile import TemporaryDirectory from pathlib import Path import pyarrow as pa with TemporaryDirectory() as tmp: cat = SqlCatalog("local", uri="sqlite:///:memory:", warehouse=Path(tmp).as_uri()) cat.create_namespace("ns") # schema: s is optional, its only child x is required dtype = pa.struct([pa.field("x", pa.int64(), nullable=False)]) data = pa.table({"s": pa.array([None], type=dtype)}) table = cat.create_table("ns.t", schema=data.schema) table.append(data) # s = None for the one row -> s.x is absent too import pyiceberg.expressions as e files = list(table.scan(row_filter=e.IsNull("s.x")).plan_files()) print(len(files)) # 0 <-- empty, should contain the file with the s = None row ``` ### Proposed fix - Add `Schema.is_field_required_in_path(field_id)` — walks the existing ancestor index to check the field *and every ancestor* is required. - Remove the leaf-only fold from `BoundIsNull`/`BoundNotNull.__new__`; do the fold in `IsNull.bind()`/`NotNull.bind()` instead, where the schema is already in scope. No change to `BoundReference`. - No public API changes. ### Willingness to contribute - [x] I can contribute a fix for this bug independently - [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community - [ ] I cannot contribute a fix for this bug at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
