voonhous commented on code in PR #19582:
URL: https://github.com/apache/hudi/pull/19582#discussion_r3774151090
##########
hudi-common/src/main/java/org/apache/hudi/common/avro/VariantSchemaUtils.java:
##########
@@ -82,4 +83,78 @@ public static HoodieSchema
stripVariantShredding(HoodieSchema schema) {
schema.getAvroSchema().getDoc(),
newFields);
}
+
+ /**
+ * Whether this column sits shredded on disk and must be read in that shape
and reconstructed to
+ * serve the requested schema. A file schema that kept its variant logical
type answers via
+ * {@link HoodieSchema.Variant#isShredded()}; but a file schema derived from
converting the
+ * parquet footer MessageType loses the logical type (variant groups come
back as plain records),
+ * so the on-disk side is detected by SHAPE, anchored by the requested side:
the requested column
+ * (from the table schema, logical type intact) must be a variant for the
shape match to count,
+ * leaving plain user structs of the same shape alone (#19567).
+ */
+ public static boolean isShreddedVariantTarget(HoodieSchema
requestedFieldSchema, HoodieSchema fileFieldSchema) {
+ HoodieSchema file = fileFieldSchema.getNonNullType();
+ if (file.getType() == HoodieSchemaType.VARIANT && ((HoodieSchema.Variant)
file).isShredded()) {
+ return true;
+ }
+ HoodieSchema requested = requestedFieldSchema.getNonNullType();
+ return requested.getType() == HoodieSchemaType.VARIANT &&
isShreddedVariantShape(file);
+ }
+
+ /**
+ * Returns {@code fileSchema} with each top-level shredded variant column
(per
+ * {@link #isShreddedVariantTarget}) replaced by its requested counterpart,
for projection or
+ * compatibility checks against {@code requestedSchema}. A footer-derived
shredded variant column
+ * surfaces as a plain {@code {metadata, value, typed_value}} record and so
can never look like a
+ * projection source of the requested variant, even though the readers
reconstruct it (see
+ * HoodieVariantReconstruction). Returns {@code fileSchema} as-is when
nothing matches.
+ */
+ public static HoodieSchema alignShreddedVariants(HoodieSchema fileSchema,
HoodieSchema requestedSchema) {
Review Comment:
Flipped to `(file, requested)`, so this class and
`HoodieVariantReconstruction.create` now read the same way; both call sites
updated and the two params documented. No behaviour change.
Worth noting the failure mode is not hypothetical: while checking this I ran
the reconstruction suite against a stale `hudi-common` in `.m2`, the flipped
call bound to the old signature, and 4/8 tests went red on inverted detection
with nothing at compile time to warn me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
