linzch3 opened a new pull request, #4419: URL: https://github.com/apache/iceberg-python/pull/4419
Closes #4417 # Rationale for this change `pyarrow_to_schema` only used field IDs when every column had one (`_HasIds` used `all`). If any column lacked an ID, it ignored all IDs and fell back to the name mapping. That fails when physical names differ from the mapped names, which is what happens with Databricks UniForm files that attach IDs to data columns but not to row-tracking columns. I changed `_HasIds` to match Java's `ParquetSchemaUtil.hasIds` (`any`), and `_ConvertToIceberg` now drops fields without IDs, including lists or maps whose element, key, or value is missing an ID. ## Are these changes tested? The issue's example is a unit test in `tests/io/test_pyarrow_visitor.py`, with and without a name mapping. I also added a nested case: a map missing child IDs is dropped, and a struct keeps only the children that have IDs. Existing `_HasIds` tests for partial IDs now expect `True`. Two schema-compatibility tests that rebuild a nested field from `as_arrow()` now keep field IDs on that field, so they still test a missing nested column rather than the new prune path. `pytest tests/io/test_pyarrow_visitor.py` and the related schema-compatibility tests passed. `make lint` passed. ## Are there any user-facing changes? Files that already have IDs on every column are unchanged. Files with IDs on only some columns are now read by ID, and columns without IDs are ignored, even if a name mapping is present. That also applies to schema-compatibility checks on write: an extra column without an ID is dropped rather than rejected. I can change that if you'd rather keep the old write-path behavior. Happy to adjust the approach if you'd prefer something else. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
