hcrosse opened a new issue, #4277: URL: https://github.com/apache/iceberg-python/issues/4277
### Apache Iceberg version main (development) ### Please describe the bug 🐞 A NaN partition value prevents deleting or replacing the data file that contains it. Even deleting every row or overwriting the whole table fails. This is a regression in 0.12.0. The reproduction below needs main, because 0.12.0 cannot `append` a NaN partition. I reproduced this on main at `e3564a54c10960307147377c578680276865ed18`, with Python 3.12 and PyArrow 25.0.1 on Linux/amd64. The table has an identity partition on a `double` column. Deletes also fail with a `float` column. Install from main, for example `pip install "pyiceberg[pyarrow,sql-sqlite] @ git+https://github.com/apache/iceberg-python@e3564a54c10960307147377c578680276865ed18"`. The reproduction uses a SQLite catalog and a temporary warehouse. It appends two rows, one with a NaN partition value, then tries to delete that partition: ```python import tempfile import pyarrow as pa from pyiceberg.catalog.sql import SqlCatalog from pyiceberg.partitioning import PartitionField, PartitionSpec from pyiceberg.schema import Schema from pyiceberg.transforms import IdentityTransform from pyiceberg.types import DoubleType, IntegerType, NestedField warehouse = tempfile.mkdtemp() catalog = SqlCatalog("default", uri=f"sqlite:///{warehouse}/catalog.db", warehouse=f"file://{warehouse}") catalog.create_namespace("default") schema = Schema( NestedField(1, "id", IntegerType(), required=False), NestedField(2, "value", DoubleType(), required=False), ) partition_spec = PartitionSpec(PartitionField(source_id=2, field_id=1000, transform=IdentityTransform(), name="value")) tbl = catalog.create_table("default.nan_partition", schema=schema, partition_spec=partition_spec) tbl.append(pa.Table.from_pydict({"id": [1, 2], "value": [1.0, float("nan")]}, schema=schema.as_arrow())) print(tbl.scan().to_arrow().to_pylist()) tbl.delete("value is nan") ``` Output, with the traceback shortened: ```text [{'id': 1, 'value': 1.0}, {'id': 2, 'value': nan}] Traceback (most recent call last): File "repro.py", line 25, in <module> tbl.delete("value is nan") ... File ".../pyiceberg/table/update/snapshot.py", line 541, in _build_delete_files_partition_predicate self._transaction._build_partition_predicate( File ".../pyiceberg/table/__init__.py", line 424, in _build_partition_predicate EqualTo(Reference(partition_field), partition_record[pos]) ... File ".../pyiceberg/expressions/literals.py", line 98, in __init__ raise ValueError("Cannot create expression literal from NaN.") ValueError: Cannot create expression literal from NaN. ``` Expected: the delete succeeds and leaves only the row whose partition value is `1.0`. Actual: `ValueError: Cannot create expression literal from NaN.` I also tested a four-row table with two rows in each partition: - `delete("value is nan")`, which would drop the whole NaN file, and `delete("id = 3")`, which would rewrite it, both fail. - `delete(AlwaysTrue())` fails too. - `overwrite(df)` fails even when the new data has no NaN, and so do overwrites filtered to the NaN file (`"value is nan"`, `"id = 3"`). - `upsert` fails when updating a row in the NaN partition. - `dynamic_partition_overwrite` fails when the incoming data contains a NaN partition value. After each failure, reloading the table returns the same snapshot and rows, with no new metadata files. The failed rewrites and dynamic overwrite each leave one unreferenced data file. Operations that touch only non-NaN files succeed. Dynamic overwrite also succeeds when NaN is present only in an untouched partition. [`_build_partition_predicate`](https://github.com/apache/iceberg-python/blob/e3564a54c10960307147377c578680276865ed18/pyiceberg/table/__init__.py#L408-L431) emits `IsNull` for a `None` value and `EqualTo` for any other value. [Expression literals reject NaN](https://github.com/apache/iceberg-python/blob/e3564a54c10960307147377c578680276865ed18/pyiceberg/expressions/literals.py#L90-L98), so that `EqualTo` raises. Two callers reach it. [`_build_delete_files_partition_predicate`](https://github.com/apache/iceberg-python/blob/e3564a54c10960307147377c578680276865ed18/pyiceberg/table/update/snapshot.py#L531-L546) passes the partitions of the files being deleted. [`dynamic_partition_overwrite`](https://github.com/apache/iceberg-python/blob/e3564a54c10960307147377c578680276865ed18/pyiceberg/table/__init__.py#L636-L648) passes the partitions of the incoming data. A narrow fix would be to emit `IsNaN` for a NaN value, next to the existing `IsNull` branch. https://github.com/apache/iceberg-python/pull/3011 added the snapshot path that builds partition predicates for manifest pruning. I also committed the NaN data file directly with `fast_append` instead of `append`. On that table, `delete("value is nan")`, `delete(AlwaysTrue())` and `overwrite(df)` succeed on 0.11.1 but raise this error on 0.12.0. Tables whose NaN partitions were written by another engine should fail the same way, but I did not test another engine. https://github.com/apache/iceberg-python/pull/4030 fixed writing NaN partitions and is not yet released. On 0.12.0, `append` raises `ZeroDivisionError` before reaching this bug. On 0.12.0 and 0.11.1, that error also hides the rewrite, `upsert` and dynamic-overwrite cases. Java's [`ManifestFilterManager`](https://github.com/apache/iceberg/blob/5a2ccf5ba830838ecf182bccb516d5ee48f8b8ed/core/src/main/java/org/apache/iceberg/ManifestFilterManager.java#L175-L187) and [`BaseReplacePartitions`](https://github.com/apache/iceberg/blob/5a2ccf5ba830838ecf182bccb516d5ee48f8b8ed/core/src/main/java/org/apache/iceberg/BaseReplacePartitions.java#L50-L55) collect deleted partitions in a set and [match manifests against that set](https://github.com/apache/iceberg/blob/5a2ccf5ba830838ecf182bccb516d5ee48f8b8ed/core/src/main/java/org/apache/iceberg/ManifestFilterManager.java#L430-L442), so they never build an expression literal from a partition value. I read the Java source but did not run it. https://github.com/apache/iceberg-python/issues/4130 is another NaN problem in `delete()`. There, missing NaN counts make the strict evaluator drop NaN rows. In the whole-file delete reproduced here, the partition predicate raises before anything is written. ### Willingness to contribute - [ ] I can contribute a fix for this bug independently - [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community - [ ] I cannot contribute a fix for this bug at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
