voonhous opened a new pull request, #20324: URL: https://github.com/apache/hudi/pull/20324
### Describe the issue this Pull Request addresses Fixes #20307. A COW upsert on a table whose shredded variant column has a DECIMAL leaf fails with `AvroTypeException: Undefined schema: k102.typed_value`. `AvroSchemaConverterWithTimestampNTZ` (Hudi's copy of parquet-java 1.13.1's `AvroSchemaConverter`) names every Avro `fixed` after its parquet field with no namespace. Shredding names every leaf `typed_value`, so a DECIMAL `fixed` leaf collides with its enclosing `typed_value` record, and the read schema no longer survives the `toString`/parse round trip in `HoodieAvroParquetReader`. ### Summary and Changelog - Port PARQUET-2456 (apache/parquet-java#1304, released in 1.14.0): records and `fixed` types share one name counter, and repeated names get a `<name><count>` namespace. - Apply the same to DECIMAL on `fixed_len_byte_array`, which Hudi's copy builds in its logical-type branch instead of `convertFIXED_LEN_BYTE_ARRAY`. - Restore the INT96 doc string, which the `HoodieSchema` migration passed in the namespace slot (`HoodieSchema.createFixed` takes the namespace second). - Tests in `TestAvroSchemaConverter`: upstream's `testReuseNamesArrays`, plus a #20307-shaped schema that must survive `toString`/parse unchanged. Both fail on master with `Can't redefine`. ### Impact Fixes the Avro read path for any parquet footer with repeated `fixed` or `fixed`/record names (COW merge, MOR compaction, Hive/MR reads, bootstrap). Only the second and later occurrences of a name change. ### Risk Level low <details><summary>Why footer-side name changes are safe</summary> Footer-derived `fixed` names already differed from table-schema names (Spark names decimals `<ns>.<record>.fixed`), so every full-name comparison between them (`HoodieAvroUtils` rewrite, `HoodieArrayWritableSchemaUtils`, `HoodieSchemaCompatibilityChecker` with `checkNaming`) already took its "names differ" path, which preserves values. </details> ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
