voonhous commented on code in PR #19582:
URL: https://github.com/apache/hudi/pull/19582#discussion_r3765891732
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/HoodieMergeHelper.java:
##########
@@ -93,8 +94,15 @@ public void runMerge(HoodieTable<?, ?, ?, ?> table,
// Check whether the writer schema is simply a projection of the file's
one, ie
// - Its field-set is a proper subset (of the reader schema)
// - There's no schema evolution transformation necessary
+ // Shredded variant columns are aligned to the writer side first: their
footer-derived file
+ // schema surfaces a plain {metadata, value, typed_value} record where the
writer schema has a
+ // variant, which can never pass the strict check even though the reader
reconstructs such
+ // columns when handed a variant-bearing requested schema. Without the
alignment recordSchema
+ // degenerates to the footer schema, reconstruction cannot anchor on a
variant requested
+ // column, and the rewrite below silently drops typed_value (#19567).
boolean isPureProjection = schemaEvolutionTransformerOpt.isEmpty()
- && HoodieSchemaCompatibility.isStrictProjectionOf(readerSchema,
writerSchema);
+ && HoodieSchemaCompatibility.isStrictProjectionOf(
+ VariantSchemaUtils.alignShreddedVariants(readerSchema,
writerSchema), writerSchema);
Review Comment:
Folded in. The Summary now has a bullet for the `runMerge` change itself -
why detection alone was not enough, and that aligning the columns flips
`isPureProjection` so the merge reads at the writer schema and skips the
rewrite - and Impact spells out what that means for the shared path:
`alignShreddedVariants` returns the file schema instance untouched unless a
file column has the exact shredded shape and the matching requested column is a
variant, so non-variant and ordinary unshredded variant tables get the same
schema object and the same verdict as before.
The residual is no longer a residual. Chasing your wording for it, the
added-column half turned out not to need schema-on-read at all: `recordSchema`
fell back to the raw footer schema on that branch too, so a plain `ALTER TABLE
ADD COLUMNS` before the bin-pack dropped `typed_value` the same way. Fixed in
the same place by aligning the reader schema where it is derived rather than
inside the projection check, with an end-to-end test for the added-column leg.
What is left is the general limit on variant schema evolution (#18285), which
the body now points at instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
