voonhous commented on code in PR #17833:
URL: https://github.com/apache/hudi/pull/17833#discussion_r2929821918
##########
hudi-common/src/main/java/org/apache/hudi/internal/schema/convert/InternalSchemaConverter.java:
##########
@@ -441,6 +455,24 @@ private static HoodieSchema
visitInternalSchemaToBuildHoodieSchema(Type type, Ma
*/
private static HoodieSchema
visitInternalRecordToBuildHoodieRecord(Types.RecordType recordType,
List<HoodieSchema> fieldSchemas, String recordNameFallback) {
List<Types.Field> fields = recordType.fields();
+
+ // Detect Variant round-trip: sentinel negative IDs with value/metadata
fields
+ if (fields.size() == 2) {
+ Types.Field field0 = fields.get(0);
+ Types.Field field1 = fields.get(1);
+ boolean hasNegativeIds = field0.fieldId() < 0 && field1.fieldId() < 0;
+ boolean hasVariantFields =
(field0.name().equals(HoodieSchema.Variant.VARIANT_VALUE_FIELD)
+ &&
field1.name().equals(HoodieSchema.Variant.VARIANT_METADATA_FIELD))
+ || (field0.name().equals(HoodieSchema.Variant.VARIANT_METADATA_FIELD)
+ &&
field1.name().equals(HoodieSchema.Variant.VARIANT_VALUE_FIELD));
+
+ if (hasNegativeIds && hasVariantFields) {
+ // TODO: Flesh out schema evolution for Variant types #18285
Review Comment:
Haven't really tested it out, scope here is unshredded only. There is no
other logical type a variant can be evolved into for now.
Since it is a RECORD of 2 byte fields, i suppose the only manipulation a
user can do is change the internal byte fields to int for type changes.
Then again, do we want to handle that? What sort of use case would require
that sort of manipulation - if user wants to be destructive with their data?
Adding, deleting and renaming fields should still be supported though, but
that hasn't been tested yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
