kevinjqliu opened a new issue, #4116: URL: https://github.com/apache/iceberg-python/issues/4116
### Apache Iceberg version main (development). This affects all versions. ### Please describe the bug 🐞 When two or more partition fields have the same `decimal(P,S)` type (e.g. `identity(a), identity(b)` where both are `decimal(10,2)`), the type `decimal_10_2` shows up twice in the manifest's Avro schema. The [Avro spec](https://avro.apache.org/docs/1.12.0/specification/#names) only allows one definition per name, so the second use should be a reference, i.e. just `"decimal_10_2"`. PyIceberg gets this backwards: - ❌ **Can't read the reference form.** `AvroSchemaConversion` treats `"decimal_10_2"` as an unknown type, so it fails on manifests written by Java, and by iceberg-rust since apache/iceberg-rust#3354. - ❌ **Writes the repeated form.** `iceberg_to_avro` writes the full definition every time, which strict readers like apache-avro (Rust) 0.22 reject. Repro: ```python from pyiceberg.utils.schema_conversion import AvroSchemaConversion dec = {"type": "fixed", "name": "decimal_10_2", "size": 5, "logicalType": "decimal", "precision": 10, "scale": 2} AvroSchemaConversion().avro_to_iceberg({"type": "record", "name": "r102", "fields": [ {"name": "a", "type": ["null", dec], "field-id": 1000, "default": None}, {"name": "b", "type": ["null", "decimal_10_2"], "field-id": 1001, "default": None}]}) # TypeError: Type not recognized: decimal_10_2 ``` The same applies to `fixed(L)` (`fixed_L`) and `uuid` (`uuid_fixed`). We should match Java and iceberg-rust: lenient on read, strict on write. - **read:** accept both the reference form and the repeated form, since manifests in the repeated form already exist - **write:** define each named type once, then refer to it by name More background, including how Java and iceberg-rust handle this, is in apache/iceberg-rust#3466. ### Willingness to contribute - [ ] I can contribute a fix for this bug independently - [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community - [ ] I cannot contribute a fix for this bug at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
