haizhou-zhao opened a new issue, #4288:
URL: https://github.com/apache/iceberg-python/issues/4288
### Apache Iceberg version
0.12.0 (latest release); also reproduced on `main` (e3564a5)
### Please describe the bug 🐞
Reading an Avro file fails with `ResolveError` when the file schema has a
map, list, or struct field that is not in the read schema:
```
pyiceberg.exceptions.ResolveError: File/read schema are not aligned for map,
got None
at pyiceberg/avro/resolver.py SchemaPartnerAccessor.map_key_partner
```
Fields are resolved by ID, so a field that exists only in the file should be
read and dropped. That already works for primitives:
`ReadSchemaResolver.struct` gives them a `None` position. For nested types,
though, `SchemaPartnerAccessor.field_partner`, `list_element_partner`,
`map_key_partner`, and `map_value_partner` raise when the partner is `None`.
This comes up with manifests from writers that add optional fields to
`data_file`. For example, one writer adds `map<int, binary>` fields with IDs
1250 and 1280. PyIceberg then can't `scan()` the table, while Iceberg
Java/Spark, DuckDB, and Snowflake read it fine.
Minimal reproduction: write a manifest entry with fastavro, using
`MANIFEST_ENTRY_SCHEMAS[2]` plus an extra optional `map<int, binary>` field in
`data_file`. Then read it with `AvroFile(read_schema=MANIFEST_ENTRY_SCHEMAS[2],
...)`.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
