linzch3 opened a new pull request, #4283:
URL: https://github.com/apache/iceberg-python/pull/4283
Closes #4277
# Rationale for this change
`Transaction._build_partition_predicate` emits `IsNull` for a `None`
partition value and `EqualTo` for anything else. Expression literals reject
NaN, so as soon as a delete or overwrite touches a NaN identity partition it
fails with `ValueError: Cannot create expression literal from NaN.` This
affects `delete()` (both dropping and rewriting the file), `overwrite()`,
`upsert` and `dynamic_partition_overwrite`. Both callers
(`_build_delete_files_partition_predicate` and `dynamic_partition_overwrite`)
go through this helper.
This PR takes the narrow fix suggested in the issue: a float NaN partition
value now becomes `IsNaN(Reference(field))`, next to the existing `IsNull`
branch.
# Are these changes tested?
Yes. `test_delete_and_overwrite_nan_identity_partition` in
`tests/catalog/test_catalog_behaviors.py` runs against each catalog fixture and
covers the cases from the issue on a `double` identity partition:
- `delete("id = 3")`, which rewrites the NaN file
- `delete("value is nan")`, which drops the NaN file
- `dynamic_partition_overwrite` with NaN incoming data
- a full `overwrite`
Without the fix it fails with the NaN literal error.
Ran locally:
- `pytest tests/ -m unit --ignore=tests/integration`: 4475 passed. The only
failures are 4 Kerberos tests in `test_hive.py`, which also fail on `main` here
because a Kerberos module is missing in my environment.
- `prek` (ruff, ruff-format, mypy, codespell) on the changed files
# Are there any user-facing changes?
Yes, a bug fix: deletes and overwrites on tables with a NaN identity
partition no longer raise.
This change was prepared with an AI coding assistant (Grok Bot) and tested
locally as described above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
