voonhous commented on code in PR #19582:
URL: https://github.com/apache/hudi/pull/19582#discussion_r3765889652
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/HoodieMergeHelper.java:
##########
@@ -93,8 +94,15 @@ public void runMerge(HoodieTable<?, ?, ?, ?> table,
// Check whether the writer schema is simply a projection of the file's
one, ie
// - Its field-set is a proper subset (of the reader schema)
// - There's no schema evolution transformation necessary
+ // Shredded variant columns are aligned to the writer side first: their
footer-derived file
+ // schema surfaces a plain {metadata, value, typed_value} record where the
writer schema has a
+ // variant, which can never pass the strict check even though the reader
reconstructs such
+ // columns when handed a variant-bearing requested schema. Without the
alignment recordSchema
+ // degenerates to the footer schema, reconstruction cannot anchor on a
variant requested
Review Comment:
Good catch, and it was not just the schema-on-read case. The alignment sat
inside the `isStrictProjectionOf` call, so it only reached one branch of
`recordSchema = isPureProjection ? writerSchema : readerSchema`. Any time the
writer schema is not a strict projection of the file's, `recordSchema` fell
back to the raw footer schema, reconstruction had no variant column to anchor
on, and `rewriteRecordWithNewSchema` copied `{metadata, value}` by name and
dropped `typed_value`. The transformer you point at hits it, but so does a
plain `ALTER TABLE ADD COLUMNS` with no schema-on-read config at all, which
makes it the same silent corruption as #19567 rather than a separate
combination.
So not a follow-up - fixed here. `alignShreddedVariants` now runs once where
the reader schema is derived, so both branches and the transformer reason about
the same variant-bearing form. That also removes the inline alignment from the
projection check, so `runMerge` reads the way it did before. Added an
end-to-end test for the added-column leg: shredded base file, `ALTER TABLE ADD
COLUMNS`, then the bin-pack merge, with rows from the first commit keeping
their variants and picking up a null for the new column.
Variant schema evolution more broadly is still limited (#18285); what is
closed here is the base file being silently emptied while merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
