voonhous commented on code in PR #18961:
URL: https://github.com/apache/hudi/pull/18961#discussion_r3472579906
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -872,16 +878,25 @@ private Type convertField(HoodieSchema fieldSchema,
StructField structField, Typ
.named(MAP_REPEATED_NAME))
.named(structField.name());
} else if (dataType instanceof StructType) {
+ StructType nestedStruct = (StructType) dataType;
Types.GroupBuilder<GroupType> groupBuilder =
Types.buildGroup(repetition);
- Arrays.stream(((StructType) dataType).fields()).forEach(field -> {
+ Arrays.stream(nestedStruct.fields()).forEach(field -> {
// Note: Cannot use HoodieSchemaField::schema method reference due to
Java 17 compilation ambiguity
HoodieSchema nestedFieldSchema = Option.ofNullable(resolvedSchema)
.flatMap(s -> s.getField(field.name()))
.map(f -> f.schema())
.orElse(null);
groupBuilder.addField(convertField(nestedFieldSchema, field));
});
- return groupBuilder.named(structField.name());
+ // A shredded variant column reaches here as a marked struct (the
VariantType was replaced by
+ // its {metadata, value, typed_value} shredding schema). Tag the parquet
group with the VARIANT
+ // logical type so external readers recognize it as a Variant, matching
Spark's native shredded
+ // parquet schema. No-op on Spark 4.0/3.x (the annotation only exists in
parquet 1.16+).
+ Types.GroupBuilder<GroupType> finalBuilder =
Review Comment:
Good call, renamed to `taggedBuilder`. Agreed that `.named()` is the
finalizing step, so `finalBuilder` was misleading.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
