JosephLenton opened a new issue, #654:
URL: https://github.com/apache/avro-rs/issues/654
Hey, in version 0.22.0 the attribute `logicalType` was dropped entirely. I
believe this is a regression, and is incorrect behaviour. I am pretty new to
Avro, so I'm looking for some help on asserting if this is right. I have
replicated the behaviour in both the Java and Rust Avro projects, and the
results are different.
# Steps to reproduce
* You create a valid Json schema which contains a `logicalType` attribute.
* You parse the Json into an Avro type.
* You then serialise the schema back to Json.
## What do I expect?
* The `logicalType` should still be present.
* The behaviour should match the Java implementation.
## What happened?
* The `logicalType` is missing.
* The behaviour does not match the Java implementation.
* The `logicalType` is entirely dropped, making it impossible to correctly
identify `map` types for Iceberg.
# Context
I am trying to upgrade the Iceberg repo from Avro 0.21.0 to 0.22.0. One test
there is failing, which is for map types. There is no Avro equivalent, so the
Iceberg spec asks for an array of key/value pairs + adding `logicalType: map`
to the schema. As Avro 0.22.0 drops `logicalType`, this makes it impossible to
identify the map type.
1. I am trying to upgrade the Iceberg project to use Avro 0.22.0, and find
that the map tests now fail. Looking at the Iceberg spec, the maps are arrays
with a `logicalType: map` added. The tests break because the parser drops the
`logicalType`, making it impossible to know for certainty it is meant to be a
map.
2. I believe the Java version preserved the `logicalType`. The Java version
has tests which perform a round trip and asserts what they started and ended
with are the same. This implies the `logicalType` is preserved. See:
https://github.com/apache/avro/blob/main/lang/java/avro/src/test/java/org/apache/avro/TestLogicalType.java#L190-L195
3. The original PR is about fixing **duplicate** outputs of `logicalType`,
not removing all of them. It seems odd to remove them all.
# Tests to reproduce
## For Java
Here is a *passing* test for the Java Apache Avro repo at
(https://github.com/apache/avro). It's a bit noisy. The key part is
`logicalType` is at the end of the Json schema, and turning it from the schema
to string yields the original schema.,
```java
@Test
void logicalTypeTestReplication() {
String rawSchema = "{\n" + " \"type\" : \"array\",\n" + " \"items\" :
{\n" + " \"type\" : \"record\",\n"
+ " \"name\" : \"k12_v13\",\n" + " \"fields\" : [ {\n" + "
\"name\" : \"key\",\n"
+ " \"type\" : \"int\",\n" + " \"field-id\" : 12\n" + "
}, {\n" + " \"name\" : \"value\",\n"
+ " \"type\" : \"string\",\n" + " \"field-id\" : 13\n" + "
} ]\n" + " },\n"
+ " \"logicalType\" : \"map\"\n" + "}";
Schema schema = SchemaParser.parseSingle(rawSchema);
String output = schema.toString(true);
assertEquals(rawSchema, output, "logicalType and custom properties should
survive parsing");
}
```
## For Rust
Here is the same test for your repo:
```rust
#[test]
fn it_should_preserve_map_logical_type_on_outer_item() -> TestResult {
let raw_schema = r#"{
"type": "array",
"logicalType": "map",
"items": {
"type": "record",
"name": "k12_v13",
"fields": [
{
"name": "key",
"type": "int",
"field-id": 12
},
{
"name": "value",
"type": "string",
"field-id": 13
}
]
}
}"#;
let schema = Schema::parse_str(raw_schema)?;
let output = serde_json::to_string_pretty(&schema).unwrap();
pretty_assertions::assert_eq!(r#"{
"type": "array",
"logicalType": "map", // <-- this is missing!
"items": {
"type": "record",
"name": "k12_v13",
"fields": [
{
"name": "key",
"type": "int",
"field-id": 12
},
{
"name": "value",
"type": "string",
"field-id": 13
}
]
}
}"#, output);
let logical_type = schema
.custom_attributes()
.unwrap()
.get("logicalType");
// !!! This fails !!!
assert!(logical_type.is_some());
Ok(())
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
