kevinjqliu opened a new issue, #3466: URL: https://github.com/apache/iceberg-rust/issues/3466
### Apache Iceberg Rust version v0.10.1 and v0.11.0-rc.2. Fixed on `main` by #3354. ### Describe the bug **Trigger:** a partition spec with two or more fields of the same `decimal(P,S)` or the same `fixed(L)`. For example: - `identity(a), identity(b)` where `a` and `b` are both `decimal(10,2)` - `identity(price), truncate(100, price)` (`truncate` keeps the decimal type) - a v1 table that replaced `identity(price)` with `truncate(price)`, which keeps `void(price)` with the source type Each type is written as an Avro `fixed` with a name (`decimal_10_2`, `fixed_16`). The [Avro spec](https://avro.apache.org/docs/1.12.0/specification/#names) says *"A schema or protocol may not contain multiple definitions of a fullname"*, so a second use must refer to the type by name. A manifest's schema can take one of two forms (the data is encoded the same way in both): **Reference form (compliant)** ```json {"name": "a", "type": ["null", {"type": "fixed", "name": "decimal_10_2", "size": 5, "logicalType": "decimal", "precision": 10, "scale": 2}]}, {"name": "b", "type": ["null", "decimal_10_2"]} ``` **Repeated form (not compliant)** ```json {"name": "a", "type": ["null", {"type": "fixed", "name": "decimal_10_2", ...}]}, {"name": "b", "type": ["null", {"type": "fixed", "name": "decimal_10_2", ...}]} ``` A *strict* reader reads only the reference form. A *lenient* reader reads both. **iceberg-rust ≤ 0.11 (apache-avro 0.21)** - Writes the repeated form for `decimal`, so strict readers such as Java ≤ 1.6 (Avro 1.11) can't read these manifests: `Can't redefine: decimal_10_2`. - Can't write two `fixed(L)` partition fields of the same length: apache-avro 0.21 rejects the schema. - Can't read the repeated form for `fixed`, for example from PyIceberg. **iceberg-rust `main` (#3354)** - Writes the reference form (`define_once` in `avro/schema.rs`). - Reads both forms. apache-avro 0.22 is strict, so `avro/named_types.rs` rewrites a repeated form into the reference form before reading. #### Java Java writes the reference form and, from 1.7, reads both. - **Write:** [`TypeToSchema.primitive`](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/avro/TypeToSchema.java) names the types `fixed_L`, `decimal_P_S`, and `uuid_fixed`. Avro's [`writeNameRef`](https://github.com/apache/avro/blob/main/lang/java/avro/src/main/java/org/apache/avro/Schema.java) writes every later use as a reference. - **Read:** - Java ≥ 1.7 (Avro 1.12): lenient. [`ParseContext.put`](https://github.com/apache/avro/blob/main/lang/java/avro/src/main/java/org/apache/avro/ParseContext.java) accepts an identical repeated definition. - Java ≤ 1.6 (Avro 1.11): strict. It rejects any repeated definition. #### PyIceberg PyIceberg writes the repeated form and can't read the reference form. - **Write:** `iceberg_to_avro` repeats the definition for every use of `decimal` and `fixed`. - **Read:** `AvroSchemaConversion._convert_schema` doesn't resolve name references. It fails on manifests written by Java and by iceberg-rust `main`: ```python from pyiceberg.utils.schema_conversion import AvroSchemaConversion dec = {"type": "fixed", "name": "decimal_10_2", "size": 5, "logicalType": "decimal", "precision": 10, "scale": 2} AvroSchemaConversion().avro_to_iceberg({"type": "record", "name": "r102", "fields": [ {"name": "a", "type": ["null", dec], "field-id": 1000, "default": None}, {"name": "b", "type": ["null", "decimal_10_2"], "field-id": 1001, "default": None}]}) # TypeError: Type not recognized: decimal_10_2 ``` - This is a follow-up to apache/iceberg-python#14, which made these types named `fixed` but didn't handle a name used more than once. #### Summary | | Writes | Reads reference form | Reads repeated form | |---|---|---|---| | Java ≥ 1.7 | Reference | ✅ | ✅ | | Java ≤ 1.6 | Reference | ✅ | ❌ | | iceberg-rust `main` | Reference | ✅ | ✅ | | iceberg-rust ≤ 0.11 | Repeated (`decimal`); write fails for `fixed` | ✅ | `decimal` ✅, `fixed` ❌ | | PyIceberg | Repeated | ❌ | ✅ | ### To Reproduce 1. Create a table partitioned by `identity(a), identity(b)` where `a` and `b` are both `decimal(10,2)`, and append a data file with iceberg-rust ≤ 0.11. 2. Read the manifest with Java ≤ 1.6: `SchemaParseException: Can't redefine: decimal_10_2`. 3. Change the types to `fixed(16)`: writing the manifest fails in iceberg-rust ≤ 0.11. ### Expected behavior Be strict when writing and lenient when reading: - Writers define each named type once and refer to it by name afterwards. - Readers read both forms, because manifests in the repeated form already exist in tables. iceberg-rust `main` does both. Follow-ups: - [ ] iceberg-rust: the fallback comment in `spec/manifest/mod.rs` describes the repeated form as legacy iceberg-rust output, but PyIceberg still writes it. Correct the comment and log at `debug!` instead of `warn!`. - [ ] PyIceberg: resolve name references when reading (`AvroSchemaConversion`). - [ ] PyIceberg: define each named type once when writing (`iceberg_to_avro`). - [ ] Spec (optional): Appendix A → Avro doesn't say how Avro types are named or how a reused type is written. Add that writers use the reference form and readers accept both. ### Willingness to contribute I can contribute a fix for this bug independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
