voonhous commented on code in PR #19834:
URL: https://github.com/apache/hudi/pull/19834#discussion_r3940232953
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaRepair.java:
##########
@@ -246,7 +246,7 @@ public static boolean hasTimestampMillisField(HoodieSchema
tableSchema) {
return hasTimestampMillisField(tableSchema.getValueType());
case UNION:
- return hasTimestampMillisField(tableSchema.getNonNullType());
+ return
tableSchema.getTypes().stream().anyMatch(HoodieSchemaRepair::hasTimestampMillisField);
Review Comment:
Good catch, pulled it in.
`pruneDataSchemaInternal`'s UNION arm now returns the data schema instead of
throwing. A union is a leaf as far as pruning goes: Avro resolves a branch by
its type, so dropping one changes the column's type rather than narrowing it.
The `default` arm right below already does exactly that -- when Spark projects
a single member out of the member-struct encoding, the required schema reduces
to `["null","string"]` -> `STRING` and the pruner returns the full union today.
The union arm was the odd one out.
It also fixes the `canBeUnion` false positive: a plain record whose fields
are all nullable and named `member0..memberN` comes back from Spark as a union
while the data schema still holds the record, and that mismatch threw here too.
`TestSchemaConverters#testMemberStructUnionSurvivesReadPathSchemaSteps`
covers the two schema steps `buildReaderWithPartitionValues` takes before it
opens a file -- `hasTimestampMillisField`, then `pruneDataSchema` -- on a table
schema carrying `["null","string","int"]`. The second fails without the fix
with exactly the message you quoted.
Worth flagging that this throw is older than the rest of the PR: it came
over from `AvroSchemaUtils.pruneDataSchema` in #13654, so it is a long-standing
limitation rather than one of the migration regressions. And I have not run a
full write-then-read round trip, so the PR text now says the read-path schema
steps pass rather than claiming the table reads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
