dgokeeffe opened a new issue, #4417:
URL: https://github.com/apache/iceberg-python/issues/4417
### Apache Iceberg version
0.12.0 (also fails on `main` at ce5a195 and on 0.9.1)
### Please describe the bug 🐞
PyIceberg can't read a Parquet file in which only some columns have field
IDs.
`pyarrow_to_schema` uses field IDs only when every column has one (`_HasIds`
uses `all`). If any column lacks an ID, it ignores all IDs and falls back to
the name mapping, so the read fails when the physical column names differ from
the mapped names.
Iceberg Java treats a file as having IDs if any column has one
(`ParquetSchemaUtil.hasIds`) and prunes the columns without IDs.
Minimal example, no catalog needed:
```python
import pyarrow as pa
from pyiceberg.io.pyarrow import pyarrow_to_schema
from pyiceberg.table.name_mapping import MappedField, NameMapping
file_schema = pa.schema([
pa.field("col-2b93b371", pa.string(), metadata={"PARQUET:field_id":
"1"}),
pa.field("_row-id-col-922d6355", pa.int64()), # no field id
])
name_mapping = NameMapping([MappedField(field_id=1, names=["store_id"])])
pyarrow_to_schema(file_schema, name_mapping)
```
```
ValueError: Could not find field with name: col-2b93b371
```
Expected: a schema with field 1, ignoring the column without an ID.
This comes from a Databricks Delta table with UniForm (Iceberg v2), column
mapping and row tracking, read through the Unity Catalog Iceberg REST catalog.
Its data files have:
- 13 data columns with physical names like `col-<uuid>`, each with a field ID
- 2 row tracking columns, `_row-id-col-<uuid>` and
`_row-commit-version-col-<uuid>`, without field IDs
`tbl.scan().to_arrow()` fails with the error above on every file. Spark with
the Iceberg 1.12.0 runtime, Polars `scan_iceberg` and DuckDB read the same
table correctly.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]