jackylee-ch opened a new pull request, #724:
URL: https://github.com/apache/paimon-rust/pull/724
Reading a table whose data files are Avro fails or loses data for three
column
types that Java Paimon happily writes.
`TIME`, `BLOB` and `MULTISET` have no arm in the Avro reader's
`build_column`,
so a query over such a column aborts with `Avro reader does not support data
type: ...` even though the same column reads fine from Parquet, `.row` and
Mosaic. Java writes all three: `AvroSchemaConverter` maps `TIME` to an int
with
the `time-millis` logical type (rejecting precision above 3), `BLOB` to bytes
alongside `BINARY`/`VARBINARY`, and `MULTISET<T>` to a map from the element
to
an int count.
A map whose key is not a string is worse than a hard failure. Avro supports
string keys natively only, so Java encodes those as an array of `{key,
value}`
records (`AvroSchemaConverter#isArrayMap`). `build_map_column` only matched
`Value::Map` and the fallback arm pushed an unchanged offset, so every such
row
decoded to a non-null map with zero entries, silently dropping the data.
**Fix**: add the three missing type arms, and decode the array-map shape into
the same Arrow `Map` array. Entries missing either field are skipped rather
than
shifting the offsets of later rows. `MAP` and `MULTISET` declare key
nullability
differently in `paimon_type_to_arrow`, so the shared helper takes both flags
explicitly and the new tests go through `build_target_arrow_schema` to pin
it.
`VECTOR` is still unsupported here: Java maps it to an Avro array, but its
Arrow type is `FixedSizeList` rather than `List`, so it needs its own
builder.
Happy to follow up separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
