rahulsmahadev opened a new pull request, #4370:
URL: https://github.com/apache/iceberg-python/pull/4370
# Rationale for this change
Scan planning gets uuid identity partitions wrong. On a table partitioned by
`identity(u)` on a `uuid` column:
```python
tbl.scan(row_filter=EqualTo("u", str(first))).plan_files() # no files;
count() is 0
tbl.scan(row_filter=In("u", {str(first), str(second)})).plan_files() # no
files
tbl.scan(row_filter=NotEqualTo("u", str(first))).count() # counts the
excluded partition too
```
Manifests are read with uuid partition values as `uuid.UUID`, while a
`UUIDLiteral` holds the 16 bytes, so the partition evaluator and the residual
evaluator compare a `UUID` with `bytes` and never find them equal. `EqualTo`
and `In` plan no files, and `NotEqualTo` and `NotIn` plan every file. Engines
that use PyIceberg's planning (`plan_files()`) get the same wrong set of files,
and `count()` is wrong because it trusts the residual.
This change gives partition values the representation literals use (a uuid
becomes its 16 bytes) before the partition and residual evaluators compare
them. It only applies to specs with a uuid partition field; other partition
values are passed through unchanged.
Follow-up to #4360, which noted this as a separate fix. Reading the matching
files still hits the PyArrow `ArrowNotImplementedError` of #2372: `to_arrow()`
raises it, and so does `count()` when it has to read the files (a row predicate
that the partition does not resolve, or delete files). For `=` and `in` filters
this replaces silently returning no rows.
## Are these changes tested?
Yes. `test_scan_plans_uuid_identity_partitions` in
`tests/io/test_pyarrow.py`, next to the append test from #4360, checks the
planned partitions and `count()` for `=`, `in`, `!=` and `not in` filters on a
uuid identity partition. It fails on main (the `=` filter plans no files) and
passes with this change.
## Are there any user-facing changes?
Bug fix: the partition and residual evaluators compare uuid partition values
with uuid literals correctly. No API changes.
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]