wombatu-kun commented on code in PR #19582:
URL: https://github.com/apache/hudi/pull/19582#discussion_r3773302205
##########
hudi-common/src/main/java/org/apache/hudi/common/avro/VariantSchemaUtils.java:
##########
@@ -82,4 +83,78 @@ public static HoodieSchema
stripVariantShredding(HoodieSchema schema) {
schema.getAvroSchema().getDoc(),
newFields);
}
+
+ /**
+ * Whether this column sits shredded on disk and must be read in that shape
and reconstructed to
+ * serve the requested schema. A file schema that kept its variant logical
type answers via
+ * {@link HoodieSchema.Variant#isShredded()}; but a file schema derived from
converting the
+ * parquet footer MessageType loses the logical type (variant groups come
back as plain records),
+ * so the on-disk side is detected by SHAPE, anchored by the requested side:
the requested column
+ * (from the table schema, logical type intact) must be a variant for the
shape match to count,
+ * leaving plain user structs of the same shape alone (#19567).
+ */
+ public static boolean isShreddedVariantTarget(HoodieSchema
requestedFieldSchema, HoodieSchema fileFieldSchema) {
+ HoodieSchema file = fileFieldSchema.getNonNullType();
+ if (file.getType() == HoodieSchemaType.VARIANT && ((HoodieSchema.Variant)
file).isShredded()) {
+ return true;
+ }
+ HoodieSchema requested = requestedFieldSchema.getNonNullType();
+ return requested.getType() == HoodieSchemaType.VARIANT &&
isShreddedVariantShape(file);
+ }
+
+ /**
+ * Returns {@code fileSchema} with each top-level shredded variant column
(per
+ * {@link #isShreddedVariantTarget}) replaced by its requested counterpart,
for projection or
+ * compatibility checks against {@code requestedSchema}. A footer-derived
shredded variant column
+ * surfaces as a plain {@code {metadata, value, typed_value}} record and so
can never look like a
+ * projection source of the requested variant, even though the readers
reconstruct it (see
+ * HoodieVariantReconstruction). Returns {@code fileSchema} as-is when
nothing matches.
+ */
+ public static HoodieSchema alignShreddedVariants(HoodieSchema fileSchema,
HoodieSchema requestedSchema) {
+ if (fileSchema.getType() != HoodieSchemaType.RECORD ||
requestedSchema.getType() != HoodieSchemaType.RECORD) {
+ return fileSchema;
+ }
+ List<HoodieSchemaField> newFields = new ArrayList<>();
+ boolean changed = false;
+ for (HoodieSchemaField fileField : fileSchema.getFields()) {
Review Comment:
alignShreddedVariants and the reconstruction it anchors both walk only
top-level fields, while HoodieRowParquetWriteSupport.processNestedDataType
shreds variants at any depth, so a variant nested in a struct still loses
typed_value on this merge and does not reach the fail-fast branches either. Is
nested deliberately out of scope here, or worth a follow-up issue?
##########
hudi-common/src/main/java/org/apache/hudi/common/avro/VariantSchemaUtils.java:
##########
@@ -82,4 +83,78 @@ public static HoodieSchema
stripVariantShredding(HoodieSchema schema) {
schema.getAvroSchema().getDoc(),
newFields);
}
+
+ /**
+ * Whether this column sits shredded on disk and must be read in that shape
and reconstructed to
+ * serve the requested schema. A file schema that kept its variant logical
type answers via
+ * {@link HoodieSchema.Variant#isShredded()}; but a file schema derived from
converting the
+ * parquet footer MessageType loses the logical type (variant groups come
back as plain records),
+ * so the on-disk side is detected by SHAPE, anchored by the requested side:
the requested column
+ * (from the table schema, logical type intact) must be a variant for the
shape match to count,
+ * leaving plain user structs of the same shape alone (#19567).
+ */
+ public static boolean isShreddedVariantTarget(HoodieSchema
requestedFieldSchema, HoodieSchema fileFieldSchema) {
+ HoodieSchema file = fileFieldSchema.getNonNullType();
+ if (file.getType() == HoodieSchemaType.VARIANT && ((HoodieSchema.Variant)
file).isShredded()) {
+ return true;
+ }
+ HoodieSchema requested = requestedFieldSchema.getNonNullType();
+ return requested.getType() == HoodieSchemaType.VARIANT &&
isShreddedVariantShape(file);
+ }
+
+ /**
+ * Returns {@code fileSchema} with each top-level shredded variant column
(per
+ * {@link #isShreddedVariantTarget}) replaced by its requested counterpart,
for projection or
+ * compatibility checks against {@code requestedSchema}. A footer-derived
shredded variant column
+ * surfaces as a plain {@code {metadata, value, typed_value}} record and so
can never look like a
+ * projection source of the requested variant, even though the readers
reconstruct it (see
+ * HoodieVariantReconstruction). Returns {@code fileSchema} as-is when
nothing matches.
+ */
+ public static HoodieSchema alignShreddedVariants(HoodieSchema fileSchema,
HoodieSchema requestedSchema) {
Review Comment:
isShreddedVariantTarget takes (requested, file) while alignShreddedVariants
and HoodieVariantReconstruction.create both take (file, requested), so this
class's two public methods order the same HoodieSchema pair opposite ways and a
swapped call still compiles. Could isShreddedVariantTarget flip to match?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
