dgokeeffe opened a new issue, #4417:
URL: https://github.com/apache/iceberg-python/issues/4417

   ### Apache Iceberg version
   
   0.12.0 (also fails on `main` at ce5a195 and on 0.9.1)
   
   ### Please describe the bug 🐞
   
   PyIceberg can't read a Parquet file in which only some columns have field 
IDs.
   
   `pyarrow_to_schema` uses field IDs only when every column has one (`_HasIds` 
uses `all`). If any column lacks an ID, it ignores all IDs and falls back to 
the name mapping, so the read fails when the physical column names differ from 
the mapped names.
   
   Iceberg Java treats a file as having IDs if any column has one 
(`ParquetSchemaUtil.hasIds`) and prunes the columns without IDs.
   
   Minimal example, no catalog needed:
   
   ```python
   import pyarrow as pa
   from pyiceberg.io.pyarrow import pyarrow_to_schema
   from pyiceberg.table.name_mapping import MappedField, NameMapping
   
   file_schema = pa.schema([
       pa.field("col-2b93b371", pa.string(), metadata={"PARQUET:field_id": 
"1"}),
       pa.field("_row-id-col-922d6355", pa.int64()),  # no field id
   ])
   name_mapping = NameMapping([MappedField(field_id=1, names=["store_id"])])
   pyarrow_to_schema(file_schema, name_mapping)
   ```
   
   ```
   ValueError: Could not find field with name: col-2b93b371
   ```
   
   Expected: a schema with field 1, ignoring the column without an ID.
   
   This comes from a Databricks Delta table with UniForm (Iceberg v2), column 
mapping and row tracking, read through the Unity Catalog Iceberg REST catalog. 
Its data files have:
   
   - 13 data columns with physical names like `col-<uuid>`, each with a field ID
   - 2 row tracking columns, `_row-id-col-<uuid>` and 
`_row-commit-version-col-<uuid>`, without field IDs
   
   `tbl.scan().to_arrow()` fails with the error above on every file. Spark with 
the Iceberg 1.12.0 runtime, Polars `scan_iceberg` and DuckDB read the same 
table correctly.
   
   ### Willingness to contribute
   
   - [ ] I can contribute a fix for this bug independently
   - [x] I would be willing to contribute a fix for this bug with guidance from 
the Iceberg community
   - [ ] I cannot contribute a fix for this bug at this time
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to