linliu-code opened a new pull request, #664:
URL: https://github.com/apache/hudi-rs/pull/664
**Stacked on #639–#663** — review only the last commit.
## The bug
`avro_to_arrow/schema.rs` modelled an Avro map as:
```rust
DataType::Dictionary(Box::new(DataType::Utf8), Box::new(value_type))
```
**An Arrow dictionary key must be an integer type.** `Dictionary(Utf8, V)`
is not a valid Arrow type — not merely an odd choice. And it cannot reconcile
against the `Map` a parquet base file actually carries:
```
evolution: unsupported container combination
Map("key_value": non-null Struct("key": non-null Utf8, "value": Int32),
unsorted)
-> Dictionary(Utf8, Int32) for field 'map_null_val'
```
So any table with a map column fails as soon as a log block has to merge
with its base file. **This affects the existing reader too** — it is a shared
conversion, and a map column has never been readable through either.
## The fix
An Avro map becomes `Map(key_value: struct<key: string, value: V>,
sorted=false)` — the same shape and the same entry-field name the parquet
reader produces, so the two agree by name.
On the array side, `build_map_array` constructs a real `MapArray`. Entries
are materialized as two-field records (`key`, `value`) so the existing struct
machinery builds both children; that is also why `child_schema_lookup` now
registers those two positions, since a struct-valued map needs its own fields
resolvable underneath them.
Entries are emitted **in key order**. Avro maps are unordered and the Arrow
type says `sorted = false`, but a stable order keeps a read reproducible rather
than dependent on hash iteration.
## What it unblocks
`harness_null_container_elements` — maps and lists containing NULL elements
— now passes. That was the hardest of the three map cases.
The other two (`harness_all_data_types`, `harness_mixed_column_types`) still
fail, but **on something else**: a decimal column in a log block reads as NULL
where the Spark snapshot has a value. The map error was failing those cases
earlier and hiding it. They stay pinned, now carrying that finding instead.
This also removes the reason base-file-only slices are routed away from the
merge-on-read engine in #659 — that guard exists purely because every parquet
fixture here has a map column.
## Tests
Two direct conversion tests: a map of primitives produces `Map` with a
non-nullable `key_value` entry struct and a non-nullable `Utf8` key; a map of
records keeps the value as a nested struct rather than collapsing it.
Plus the harness case above, which exercises the array builder end to end
against a Spark snapshot.
Full workspace green: 1178 lib + 79 table-read + 39 datafusion + 21 + 12.
Ignored 8 → 7.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
