prashantwason commented on code in PR #20483:
URL: https://github.com/apache/hudi/pull/20483#discussion_r4471689076
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/util/HiveSchemaUtil.java:
##########
@@ -259,11 +259,25 @@ private static String convertField(final HoodieSchema
fieldSchema, boolean suppo
case BLOB:
case VARIANT:
return createHiveStruct(nonNullType.getFields(), supportTimestamp,
doFormat);
+ case UNION:
+ // A union of multiple non-null types is written to parquet as a group
of optional "member<N>" fields,
Review Comment:
Done in 6416e5011170. The comment now describes only the current mapping,
and the #14344 history stays in the PR description.
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/util/HiveSchemaUtil.java:
##########
@@ -259,11 +259,25 @@ private static String convertField(final HoodieSchema
fieldSchema, boolean suppo
case BLOB:
case VARIANT:
return createHiveStruct(nonNullType.getFields(), supportTimestamp,
doFormat);
+ case UNION:
+ // A union of multiple non-null types is written to parquet as a group
of optional "member<N>" fields,
+ // so it is synced as a struct of those members, the same as the
parquet schema based sync before #14344.
+ return createHiveStruct(getUnionMemberFields(nonNullType),
supportTimestamp, doFormat);
default:
throw new UnsupportedOperationException("Cannot convert type " +
nonNullType.getType());
}
}
+ private static List<HoodieSchemaField> getUnionMemberFields(HoodieSchema
unionSchema) {
Review Comment:
Done in 6416e5011170. Both converters now use a single helper,
`UnionSchemaUtils.getUnionMemberFields` in hudi-sync-common, so the `member<N>`
numbering is defined in one place. I also dropped the null filter:
`HiveSchemaUtil.convertField` and `SparkSchemaUtils.convertFieldType` both call
`getNonNullType()` first, so the union no longer has a null branch when it
reaches the helper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
