rahulsmahadev opened a new pull request, #4370:
URL: https://github.com/apache/iceberg-python/pull/4370

   # Rationale for this change
   
   Scan planning gets uuid identity partitions wrong. On a table partitioned by 
`identity(u)` on a `uuid` column:
   
   ```python
   tbl.scan(row_filter=EqualTo("u", str(first))).plan_files()   # no files; 
count() is 0
   tbl.scan(row_filter=In("u", {str(first), str(second)})).plan_files()  # no 
files
   tbl.scan(row_filter=NotEqualTo("u", str(first))).count()     # counts the 
excluded partition too
   ```
   
   Manifests are read with uuid partition values as `uuid.UUID`, while a 
`UUIDLiteral` holds the 16 bytes, so the partition evaluator and the residual 
evaluator compare a `UUID` with `bytes` and never find them equal. `EqualTo` 
and `In` plan no files, and `NotEqualTo` and `NotIn` plan every file. Engines 
that use PyIceberg's planning (`plan_files()`) get the same wrong set of files, 
and `count()` is wrong because it trusts the residual.
   
   This change gives partition values the representation literals use (a uuid 
becomes its 16 bytes) before the partition and residual evaluators compare 
them. It only applies to specs with a uuid partition field; other partition 
values are passed through unchanged.
   
   Follow-up to #4360, which noted this as a separate fix. Reading the matching 
files still hits the PyArrow `ArrowNotImplementedError` of #2372: `to_arrow()` 
raises it, and so does `count()` when it has to read the files (a row predicate 
that the partition does not resolve, or delete files). For `=` and `in` filters 
this replaces silently returning no rows.
   
   ## Are these changes tested?
   
   Yes. `test_scan_plans_uuid_identity_partitions` in 
`tests/io/test_pyarrow.py`, next to the append test from #4360, checks the 
planned partitions and `count()` for `=`, `in`, `!=` and `not in` filters on a 
uuid identity partition. It fails on main (the `=` filter plans no files) and 
passes with this change.
   
   ## Are there any user-facing changes?
   
   Bug fix: the partition and residual evaluators compare uuid partition values 
with uuid literals correctly. No API changes.
   
   This pull request and its description were written by Isaac.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to