zhangshenghang opened a new pull request, #12770:
URL: https://github.com/apache/seatunnel/pull/12770
### Purpose of this pull request
`HudiCatalog#getTable` converts the Avro table schema to SeaTunnel types via
`SchemaUtil.convertSeaTunnelType`. Both complex-type branches of that converter
are broken, so **reading any Hudi table that contains a MAP or ARRAY column
crashes with `AvroRuntimeException`**:
**1. MAP columns** — `convertMapType` builds the key type with
`schema.getElementType()`:
```java
private static MapType convertMapType(String field, Schema schema) {
return new MapType(
convertSeaTunnelType(field, schema.getElementType()), // throws
for MAP schemas
convertSeaTunnelType(field, schema.getValueType()));
}
```
In Avro (the project pins 1.11.1) only `ArraySchema` implements
`getElementType()`; the base implementation throws `AvroRuntimeException("Not
an array")` for every other type, so this line throws for *every* MAP column
(verified empirically against avro-1.11.1:
`Schema.createMap(...).getElementType()` → `AvroRuntimeException: Not an
array`). Per the Avro specification map keys are always strings, so the key
type is now `STRING` and the value type keeps using `getValueType()`.
**2. ARRAY columns** — the ARRAY case unwraps the element type at the call
site, but `convertListType` then unwraps it a *second* time:
```java
case ARRAY:
return convertListType(field, schema.getElementType()); // passes the
ELEMENT schema
...
private static ArrayType convertListType(String field, Schema schema) {
switch (schema.getElementType().getType()) { // calls
getElementType() on the element
```
For `array<string>` the switch calls `getElementType()` on the *string*
schema and throws `AvroRuntimeException: Not an array: \"string\"`.
`convertListType` now receives the array schema itself and unwraps it once,
symmetric with `convertMapType`.
Both paths are exercised from `HudiCatalog.convertSchema`
(HudiCatalog.java:334), i.e. table-schema reads through the Hudi catalog — a
plain `array<string>` or `map<string,string>` column was enough to fail the
read.
### Does this PR introduce _any_ user-facing change?
No API/config change. Behaviour change: Hudi tables containing MAP or ARRAY
columns can now be read through the Hudi catalog instead of failing with
`AvroRuntimeException`. MAP keys convert to `STRING` (Avro map keys are always
strings); ARRAY columns convert to the corresponding primitive array types as
before.
### How was this patch tested?
Added `SchemaUtilTest` (unit tests, no cluster needed): MAP columns with
string / int / nullable-union / array / decimal values and a record containing
a MAP column all convert correctly, plus an `array<string>` sanity case. The
array-valued and plain-array tests fail with `AvroRuntimeException` on the
pre-fix code and pass after the fix; the MAP tests likewise crashed before the
fix. Full module suite passes: `connector-hudi` — 23 tests, 0 failures.
`spotless:check` passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
