kbuci commented on code in PR #18702:
URL: https://github.com/apache/hudi/pull/18702#discussion_r3211752597
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/parquet/ParquetSchemaConverter.java:
##########
@@ -190,6 +201,38 @@ public static MessageType
convertToParquetMessageType(String name, RowType rowTy
return new MessageType(name, types);
}
+ /**
+ * Checks whether the group carries the Parquet {@code VARIANT} logical type
annotation.
+ * Uses class-name matching so this compiles against parquet-java versions
that predate the
+ * {@code VariantLogicalTypeAnnotation} class (< 1.15.2).
+ */
+ private static boolean hasVariantAnnotation(LogicalTypeAnnotation
logicalType) {
Review Comment:
Yeah based on my understanding, in order to make sure a Flink-written
variant field (or any variant field written without the parquet variant
annotation — which is all current Hudi writes since the code has a TODO to add
it like you pointed out) is re-read as variant in Flink, we would need a
subsequent PR like #18539 that add HoodieSchema for detection or we need to
actually attach the annotation at write time.
Maybe we can add a TODO comment here, just so that readers know this state
of Flink variant integration (in case for some reason we don't fully "fix" this
by 1.3)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
