JosephLenton commented on issue #654:
URL: https://github.com/apache/avro-rs/issues/654#issuecomment-5526164453
Cheers @martin-g for taking a look. I'll try to explain it. To be clear, it
doesn't need to be a custom attribute. What I'm asking for is:
- `logicalType` is accessible again in _some_ way (on `attributes` or as
`pub logicalType: Option<String>`),
- When you serialise a Schema, the `logicalType` is preserved (matching the
behaviour in Java Avro).
## The Issue
The Iceberg spec has a `map` type which is implemented as an array. This is
the example Json I posted above in the Rust test. A map schema is basically:
* an `array`,
* it holds a `record` with the fields `key` and `value`. i.e. It's an array
of key / value pairs.
* And it has `logicalType: map`.
The example schema from the Iceberg spec is this:
```json
{
"type": "array",
"logicalType": "map", // <-- marks it as being a map.
"items": {
"type": "record",
"name": "k12_v13",
"fields": [
{
"name": "key",
"type": "int",
"field-id": 12
},
{
"name": "value",
"type": "string",
"field-id": 13
}
]
}
}
```
## Where it fails in Iceberg
The failing test I have is here:
https://github.com/JosephLenton/iceberg-rust/blob/main/crates/iceberg/src/avro/schema.rs#L922
(this is from main and it's the same on my own branch).
What happens is:
1. We parse the Json Schema,
2. we receive an `avro::Schema::Array`,
3. then we then check if it has `logicalType: map` here:
https://github.com/JosephLenton/iceberg-rust/blob/main/crates/iceberg/src/avro/schema.rs#L365-L371
4. Before: it has the `logicalType`, so we make it a map.
5. Now: the type is missing!
AFAIK because the `logicalType` is getting stripped out, I cannot solve this
in Avro 0.22.0. I could look at the raw json but it gets hacky.
^ I'm aware what I wrote above is quite long. I can repeat it again in
shorter comments if it's too much to go over. Let me know if I'm making sense
or not, and I can try to clarify stuff.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
