linzch3 opened a new pull request, #4419:
URL: https://github.com/apache/iceberg-python/pull/4419

   Closes #4417
   
   # Rationale for this change
   
   `pyarrow_to_schema` only used field IDs when every column had one (`_HasIds` 
used `all`). If any column lacked an ID, it ignored all IDs and fell back to 
the name mapping. That fails when physical names differ from the mapped names, 
which is what happens with Databricks UniForm files that attach IDs to data 
columns but not to row-tracking columns.
   
   I changed `_HasIds` to match Java's `ParquetSchemaUtil.hasIds` (`any`), and 
`_ConvertToIceberg` now drops fields without IDs, including lists or maps whose 
element, key, or value is missing an ID.
   
   ## Are these changes tested?
   
   The issue's example is a unit test in `tests/io/test_pyarrow_visitor.py`, 
with and without a name mapping. I also added a nested case: a map missing 
child IDs is dropped, and a struct keeps only the children that have IDs. 
Existing `_HasIds` tests for partial IDs now expect `True`. Two 
schema-compatibility tests that rebuild a nested field from `as_arrow()` now 
keep field IDs on that field, so they still test a missing nested column rather 
than the new prune path.
   
   `pytest tests/io/test_pyarrow_visitor.py` and the related 
schema-compatibility tests passed. `make lint` passed.
   
   ## Are there any user-facing changes?
   
   Files that already have IDs on every column are unchanged. Files with IDs on 
only some columns are now read by ID, and columns without IDs are ignored, even 
if a name mapping is present. That also applies to schema-compatibility checks 
on write: an extra column without an ID is dropped rather than rejected. I can 
change that if you'd rather keep the old write-path behavior.
   
   Happy to adjust the approach if you'd prefer something else.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to