voonhous commented on code in PR #19834:
URL: https://github.com/apache/hudi/pull/19834#discussion_r3949001676
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchema.java:
##########
@@ -1379,9 +1379,13 @@ public boolean isSchemaNull() {
}
/**
- * If this is a union schema, returns the non-null type. Otherwise, returns
this schema.
+ * Strips the null branch from a nullable union. For {@code ["null", T]} (in
either order) this returns
+ * {@code T}. For a union with two or more non-null branches it returns a
union of just those branches,
Review Comment:
Fixed: the javadoc now names both, `[T]` comes back as-is (still a union)
and `["null"]` throws, and `TestHoodieSchema` pins them. The three walkers no
longer read that contract off the result at all; they ask
`HoodieSchema#isComplexUnion()` first (see the `ValueType` thread).
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java:
##########
@@ -574,7 +574,12 @@ private static HoodieSchema
pruneDataSchemaInternal(HoodieSchema dataSchema, Hoo
return
HoodieSchema.createMap(pruneDataSchema(dataSchema.getValueType(),
requiredSchema.getValueType(), Collections.emptySet()));
case UNION:
- throw new IllegalArgumentException("Data schema is a union");
+ // A union is a leaf as far as pruning goes: Avro resolves a branch by
its type, so dropping a
+ // branch changes the column's type instead of narrowing it. Hand back
the data schema unpruned,
+ // which is what the default arm below already does when Spark
projects a single member out of
+ // the member-struct encoding of a union. This also covers a plain
record whose fields happen to
+ // be named member0..memberN, which HoodieSparkSchemaConverters reads
back as a union.
+ return dataSchema;
Review Comment:
Reachable, and nothing keeps the struct whole: the format extends
`ParquetFileFormat`, so Spark's own `SchemaPruning` prunes the member struct,
and `canBeUnion` turns the pruned `struct<member1>` back into a union. The
reader then emits every member while the output projection is bound to Spark's
narrower struct, so leaves shift by ordinal: `SELECT choice.member1` returned
the byte length of `member0`. A one-member projection already did this on
master through the `default` arm; this arm would have extended it to several
members.
Fixed: `HoodieFileGroupReaderBasedFileFormat` now binds its output
projection to the shape the reader emits and drops the unrequested inner fields
by name at every depth
(`SparkSchemaTransformUtils.generateNestedPruningProjection`, next to the
null-padding one). BLOB and VARIANT, which `pruneDataSchema` also keeps whole,
go through the same path. `TestNestedSchemaPruningOptimization` covers COW and
MOR with a log file; the table is written through the Avro write client because
a Spark write cannot land such a union (its writer schema round-trips through
InternalSchema, which keeps the first branch), a separate bug noted in the
description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
