voonhous commented on code in PR #19582:
URL: https://github.com/apache/hudi/pull/19582#discussion_r3774156674
##########
hudi-common/src/main/java/org/apache/hudi/common/avro/VariantSchemaUtils.java:
##########
@@ -82,4 +83,78 @@ public static HoodieSchema
stripVariantShredding(HoodieSchema schema) {
schema.getAvroSchema().getDoc(),
newFields);
}
+
+ /**
+ * Whether this column sits shredded on disk and must be read in that shape
and reconstructed to
+ * serve the requested schema. A file schema that kept its variant logical
type answers via
+ * {@link HoodieSchema.Variant#isShredded()}; but a file schema derived from
converting the
+ * parquet footer MessageType loses the logical type (variant groups come
back as plain records),
+ * so the on-disk side is detected by SHAPE, anchored by the requested side:
the requested column
+ * (from the table schema, logical type intact) must be a variant for the
shape match to count,
+ * leaving plain user structs of the same shape alone (#19567).
+ */
+ public static boolean isShreddedVariantTarget(HoodieSchema
requestedFieldSchema, HoodieSchema fileFieldSchema) {
+ HoodieSchema file = fileFieldSchema.getNonNullType();
+ if (file.getType() == HoodieSchemaType.VARIANT && ((HoodieSchema.Variant)
file).isShredded()) {
+ return true;
+ }
+ HoodieSchema requested = requestedFieldSchema.getNonNullType();
+ return requested.getType() == HoodieSchemaType.VARIANT &&
isShreddedVariantShape(file);
+ }
+
+ /**
+ * Returns {@code fileSchema} with each top-level shredded variant column
(per
+ * {@link #isShreddedVariantTarget}) replaced by its requested counterpart,
for projection or
+ * compatibility checks against {@code requestedSchema}. A footer-derived
shredded variant column
+ * surfaces as a plain {@code {metadata, value, typed_value}} record and so
can never look like a
+ * projection source of the requested variant, even though the readers
reconstruct it (see
+ * HoodieVariantReconstruction). Returns {@code fileSchema} as-is when
nothing matches.
+ */
+ public static HoodieSchema alignShreddedVariants(HoodieSchema fileSchema,
HoodieSchema requestedSchema) {
+ if (fileSchema.getType() != HoodieSchemaType.RECORD ||
requestedSchema.getType() != HoodieSchemaType.RECORD) {
+ return fileSchema;
+ }
+ List<HoodieSchemaField> newFields = new ArrayList<>();
+ boolean changed = false;
+ for (HoodieSchemaField fileField : fileSchema.getFields()) {
Review Comment:
Fixed here rather than deferred. You are right about the reach of it, and it
is wider than the two methods you point at: `stripVariantShredding`,
`alignShreddedVariants` and the reconstruction loop all walked `getFields()`
once, so a nested shredded variant matched nothing, `create()` reported no
target, and the column read at the unshredded schema. Same silent loss as the
top-level defect, and, as you say, no fail-fast either, since nothing at the
top level looks shredded.
Both halves recurse now:
- `VariantSchemaUtils` gained one shared recursion over records, array
elements and map values. `alignShreddedVariants` keeps its contract (same
instance back when nothing matches) and gained a dual, `toShreddedReadSchema`,
which is the schema to read the file at.
- `HoodieVariantReconstruction` swapped its flat `isTarget[]` plus parallel
sub-schema arrays for a rebuild plan built once in `create()`: a variant node
per target, record/array/map nodes descending into it, and `null` - pass the
value through untouched - wherever nothing below is shredded, so non-variant
tables walk the path they did before.
Tests are unit level: nested reconstruction across all three container kinds
(verified red on the `assertNotNull` with only the main-source changes
stashed), the nested unshredded twin that must stay disengaged, and a nested
alignment case in `TestHoodieSchemaCompatibility`. No end-to-end twin on
purpose - `applyForcedShreddingSchema` and the row writer's forced-schema hook
are both top-level only, so no fixture can produce a nested shredded base file
today.
Still open, and out of scope here: the avro **write** side does not re-shred
nested variants, so a rewrite leaves them unshredded. Lossless, unlike the read
gap, but it means a nested column silently loses its shredding on merge. PR
body updated for the nested change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
