kevinjqliu opened a new issue, #708:
URL: https://github.com/apache/avro-rs/issues/708
### Problem
Some Avro writers repeat the full definition of a named type instead of
referring to it by name after the first use. For example, PyIceberg and
iceberg-rust 0.21 do this for two Iceberg partition fields that are both
`decimal(10,2)`:
```json
{"type": "record", "name": "r", "fields": [
{"name": "a", "type": {"type": "fixed", "name": "decimal_10_2", "size": 5,
"logicalType": "decimal", "precision": 10, "scale": 2}},
{"name": "b", "type": {"type": "fixed", "name": "decimal_10_2", "size": 5,
"logicalType": "decimal", "precision": 10, "scale": 2}}
]}
```
The [spec](https://avro.apache.org/docs/1.12.0/specification/#names) doesn't
allow this, but these files already exist and won't go away until someone
rewrites them. In 0.22:
```rust
let schema = Schema::parse_str(REPEATED)?; // Ok
ResolvedSchema::try_from(&schema)?; // Err: Two named schema defined
for same fullname: decimal_10_2.
```
`Reader::new` on a file with this schema in its header fails with the same
error, unless a reader schema is passed.
### What Java does
Java Avro 1.12 accepts a repeat if it's identical to the first definition,
and only rejects one that differs
([`ParseContext.put`](https://github.com/apache/avro/blob/main/lang/java/avro/src/main/java/org/apache/avro/ParseContext.java)):
```java
if (alreadyKnownSchema != null) {
if (!schema.equals(alreadyKnownSchema)) throw new
SchemaParseException("Can't redefine: " + fullName);
}
```
### Proposal
During name resolution, treat a repeated definition that's equal to the
first one as a reference, and keep the error when it differs. That's lenient on
read, like Java, and a repeat that differs is still caught as genuinely
ambiguous.
### Why
iceberg-rust upgraded to 0.22 in apache/iceberg-rust#3354. To keep reading
existing manifests, we had to add a workaround that rewrites the schema JSON in
the file header (repeats become references) and reads the file again
([`named_types.rs`](https://github.com/apache/iceberg-rust/blob/main/crates/iceberg/src/avro/named_types.rs)).
With this change, plus #575 for the other custom code, we could move all of it
into avro-rs. More background in apache/iceberg-rust#3466.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
