wombatu-kun commented on code in PR #19834:
URL: https://github.com/apache/hudi/pull/19834#discussion_r3953843350
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java:
##########
@@ -574,7 +574,12 @@ private static HoodieSchema
pruneDataSchemaInternal(HoodieSchema dataSchema, Hoo
return
HoodieSchema.createMap(pruneDataSchema(dataSchema.getValueType(),
requiredSchema.getValueType(), Collections.emptySet()));
case UNION:
- throw new IllegalArgumentException("Data schema is a union");
+ // A union is a leaf as far as pruning goes: Avro resolves a branch by
its type, so dropping a
+ // branch changes the column's type instead of narrowing it. Hand back
the data schema unpruned,
+ // which is what the default arm below already does when Spark
projects a single member out of
+ // the member-struct encoding of a union. This also covers a plain
record whose fields happen to
+ // be named member0..memberN, which HoodieSparkSchemaConverters reads
back as a union.
+ return dataSchema;
Review Comment:
`SELECT choice.member0` on a union with a record branch still throws "Data
schema is not a record": Spark's pruned `struct<member0>` converts back to
`["null", <record>]`, whose `getNonNullType()` is a RECORD, so the switch
dispatches on RECORD and never reaches this arm. Hoisting the
`dataSchema.getType() == UNION` check above the switch would cover the record,
array and map branches too.
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/internal/convert/InternalSchemaConverter.java:
##########
@@ -130,7 +130,7 @@ private static void collectColNamesFromSchema(HoodieSchema
schema, Deque<String>
return;
case UNION:
- collectColNamesFromSchema(schema.getNonNullType(), visited, resultSet);
+ schema.getTypes().forEach(branch -> collectColNamesFromSchema(branch,
visited, resultSet));
Review Comment:
Walking every branch emits leaf names for branches `visitSchemaToBuildType`
discards - it keeps only the first non-null one - so on a schema-on-read table
`pruneInternalSchema` fails with "cannot prune col: x.y which does not exist in
hudi table" for a union whose record, array or map branch is not first. Walking
only the first non-null branch would keep the two sides in step.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
