voonhous opened a new issue, #19782:
URL: https://github.com/apache/hudi/issues/19782
### Describe the problem you faced
On the AVRO record type, a table column `array<struct<...>>` whose element
struct has a SINGLE field is written as a 2-level parquet list (`repeated group
array { <field> }`, parquet-avro's default
`parquet.avro.write-old-list-structure=true`) and cannot be read back by Hudi's
own `HoodieAvroParquetReader` when parquet-avro is older than 1.15. Multi-field
element structs and primitive elements read fine.
Reproduced with and without variant shredding, so it is independent of
#19777 (found while reviewing it: the value-level round trip there had to add a
second field to the element struct to be readable).
### To Reproduce
1. Write through `HoodieAvroWriteSupport` (any AVRO record-type write; a
direct `HoodieAvroParquetWriter` is enough) with schema `record { id int,
items: array<record { v: int }> }` and one row `items = [{v: 1}]`.
2. Read the file with `new HoodieAvroParquetReaderBuilder<>(path)` (or
`HoodieAvroParquetReader.getIndexedRecordIterator(tableSchema)`).
Result on parquet-avro 1.13.1 (hudi-hadoop-common default, Spark 3.5
bundles):
```
java.lang.ClassCastException: optional int32 v is not a group (element
struct { v: int })
org.apache.parquet.io.InvalidRecordException: Parquet/Avro schema mismatch:
Avro field 'metadata' not found (element struct { v: variant })
```
Setting `AvroReadSupport.setAvroReadSchema` / `setRequestedProjection` does
not help. Writing the same shape with
`parquet.avro.write-old-list-structure=false` (3-level `list/element`)
round-trips correctly.
### Root cause
`org.apache.parquet.avro.AvroRecordConverter.isElementType(Type
repeatedType, Schema elementSchema)` has to decide whether a single-field
repeated group is the element itself or a synthetic wrapper. In parquet-avro
1.13.1 it falls through to
`SchemaCompatibility.checkReaderWriterCompatibility(elementSchema,
CONVERTER.convert(repeatedType))`, which compares Avro RECORD NAMES: the
converted group is named `array`, the reader's element record carries the
table's name, so the check fails and the group is treated as a wrapper.
parquet-avro 1.15+ added `|| repeatedType.getName().equals("array")` to the
short-circuit, which resolves the 2-level layout by name, so Spark 4.1+ bundles
are unaffected.
`HoodieAvroReadSupport` (HUDI-7874, #11450) detects the file's list layout
and switches `WRITE_OLD_LIST_STRUCTURE` for the converter, but its regression
suite `TestParquetReaderCompatibility` only uses primitive elements
(`array<long>`), so the single-field struct element was never covered.
### Fix options
1. `HoodieAvroReadSupport`: resolve the single-field 2-level case by the
group name (`array`, `<list>_tuple`) before delegating to parquet-avro,
mirroring what 1.15+ does (the legacy Hive guard already applies this rule:
`HoodieParquetInputFormat.isSyntheticListLevel`).
2. Write 3-level lists on the Avro path by default
(`parquet.avro.write-old-list-structure=false`, what the row writer does via
`hoodie.parquet.writelegacyformat.enabled=false`). Changes the on-disk layout
of new files; readers handle both.
3. Bump parquet to 1.15+ for the modules that still ship 1.13.1.
Whichever is chosen, `TestParquetReaderCompatibility` should gain a
single-field struct element case, and nullable/null array elements on the
2-level layout (`NPE: Array contains a null element`) can be pinned in the same
place.
### Environment
Hudi master (efe02e1621c1 and later), parquet-avro 1.13.1; not reproducible
on parquet-avro 1.15.2 / 1.16.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
