prashantwason opened a new pull request, #20354:
URL: https://github.com/apache/hudi/pull/20354
### Describe the issue this Pull Request addresses
Hive sync fails for any table whose schema contains a field declared with
the Avro primitive type `"null"` (as opposed to a nullable union such as
`["null","string"]`). Such fields are valid Avro and are common in schemas
inferred from sample data, e.g.:
```json
{"name": "clientPromotion", "type": "null", "default": null}
```
The sync fails with:
```
org.apache.hudi.exception.HoodieException: Got runtime exception when hive
syncing <table>
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:187)
Caused by: java.lang.UnsupportedOperationException: Cannot convert type NULL
at
org.apache.hudi.hive.util.HiveSchemaUtil.convertField(HiveSchemaUtil.java:258)
at
org.apache.hudi.hive.util.HiveSchemaUtil.createHiveStruct(HiveSchemaUtil.java:275)
...
at
org.apache.hudi.hive.util.HiveSchemaUtil.hoodieSchemaToMapSchema(HiveSchemaUtil.java:177)
at
org.apache.hudi.hive.util.HiveSchemaUtil.getSchemaDifference(HiveSchemaUtil.java:77)
at org.apache.hudi.hive.HiveSyncTool.syncSchema(HiveSyncTool.java:482)
```
This is a regression from #14344, which moved meta sync from the parquet
`MessageType` to `HoodieSchema`. The parquet Avro converter skips `"null"` type
fields ("Avro nulls are not encoded, unless they are null unions"), so they
never reached the Hive schema before. `HiveSchemaUtil.convertField` and
`SparkSchemaUtils.convertFieldType` have no case for `NULL`, and
`HoodieSchema.getNonNullType()` only unwraps unions, so a `"null"` type field
now hits the `default` branch and throws.
### Summary and Changelog
Hive sync no longer fails on schemas with `"null"` type fields. These fields
are skipped, so the synced schema matches the columns in the data files, which
is what Hive sync produced before #14344. Tables already synced by an older
release see no schema change.
- `HiveSchemaUtil`: skip `"null"` type fields in `hoodieSchemaToMapSchema`
(top-level columns) and `createHiveStruct` (nested struct fields).
- `SparkSchemaUtils`: skip `"null"` type fields in
`convertToSparkSchemaJson`, which builds the `spark.sql.sources.schema` table
property. `HiveSyncTool.syncProperties` calls it right after `syncSchema`, so
it would otherwise throw `Cannot convert NULL to spark sql type`.
- Tests: `TestHiveSchemaUtil#testNullTypeFieldsAreSkipped`,
`TestHiveSchemaUtil#testSchemaDiffIgnoresNullTypeFields` and
`TestSparkSchemaUtils#testConvertSchemaSkipsNullTypeFields` cover top-level and
nested `"null"` type fields. All three fail without the fix.
### Impact
Fixes Hive sync for tables with `"null"` type fields. No public API or
config change.
### Risk Level
low. The change only affects fields whose type is exactly `"null"`, which
previously caused sync to throw. Nullable unions are unaffected.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
