prashantwason opened a new pull request, #20483:
URL: https://github.com/apache/hudi/pull/20483
### Describe the issue this Pull Request addresses
Hive sync fails for any table whose schema has a field that is a union of
more than one non-null type, such as `["null","int","string"]`, `["long",
{"type":"record",...}]`, or the single-type union `["string"]`:
```
java.lang.UnsupportedOperationException: Cannot convert type UNION
at
org.apache.hudi.hive.util.HiveSchemaUtil.convertField(HiveSchemaUtil.java:263)
...
at
org.apache.hudi.hive.util.HiveSchemaUtil.getSchemaDifference(HiveSchemaUtil.java:77)
at org.apache.hudi.hive.HiveSyncTool.syncSchema(HiveSyncTool.java:482)
```
This is a regression from #14344, which moved meta sync from the parquet
`MessageType` to `HoodieSchema`. The parquet writer encodes such a union as a
group of optional `member<N>` fields (see
`AvroSchemaConverterWithTimestampNTZ#convertUnionToGroupType` and
parquet-avro's `AvroSchemaConverter`). The parquet-schema-based sync therefore
registered it as `struct<member0:...,member1:...>`.
`HoodieSchema.getNonNullType()` only unwraps `[null, X]`, so these unions reach
`HiveSchemaUtil.convertField` and `SparkSchemaUtils.convertFieldType`. Neither
has a `UNION` case, so they throw.
### Summary and Changelog
Hive sync no longer fails on schemas with multi-type unions. They are synced
as structs of `member<N>` fields, matching both the data files and the schema
that older releases registered. Tables already synced by an older release see
no schema change.
- `HiveSchemaUtil`: map `UNION` to `struct<member0:...,memberN:...>`,
skipping null members.
- `SparkSchemaUtils`: map `UNION` to a struct of nullable `member<N>`
fields. This is how Spark reads the parquet group, and it builds the
`spark.sql.sources.schema` table property that `HiveSyncTool.syncProperties`
writes right after `syncSchema`.
- Tests:
- `TestHiveSchemaUtil#testUnionTypeConversion` covers nullable, non-null
and single-type unions, a union with a record member, and an array of unions.
- `TestHiveSchemaUtil#testSchemaDiffForUnionFieldsSyncedFromParquetSchema`
checks that a table synced from the parquet schema has no schema difference.
- `TestSparkSchemaUtils#testConvertUnionType`.
### Impact
Fixes Hive sync for tables with multi-type union fields. No public API or
config change.
### Risk Level
low. Only `UNION` schemas left after `getNonNullType()` are affected, and
they previously caused sync to throw. Nullable `[null, X]` fields are
unaffected.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
