n3nash commented on a change in pull request #2927:
URL: https://github.com/apache/hudi/pull/2927#discussion_r629682415
##########
File path:
hudi-common/src/main/java/org/apache/hudi/common/table/TableSchemaResolver.java
##########
@@ -353,6 +360,89 @@ public static boolean isSchemaCompatible(String oldSchema,
String newSchema) {
return isSchemaCompatible(new Schema.Parser().parse(oldSchema), new
Schema.Parser().parse(newSchema));
}
+ /**
+ * Get latest schema either from incoming schema or table schema.
+ * @param incomingSchema incoming batch's schema.
+ * @param convertTableSchemaToAddNamespace {@code true} if table schema
needs to be converted to add namespace. false otherwise.
+ * @param converterFn converter function to be called over table schema.
+ * @return the latest schema.
+ */
+ public Schema getLatestSchema(Schema incomingSchema, boolean
convertTableSchemaToAddNamespace,
+ Function1<Schema, Schema> converterFn) {
+ Schema latestSchema = incomingSchema;
+ try {
+ if (isTimelineNonEmpty()) {
+ Schema tableSchema = getTableAvroSchemaWithoutMetadataFields();
+ if (convertTableSchemaToAddNamespace) {
+ tableSchema = converterFn.apply(tableSchema);
+ }
+ if (incomingSchema.getFields().size() < tableSchema.getFields().size()
&& isSchemaSubset(tableSchema, incomingSchema)) {
Review comment:
We don't allow changing the data types of columns. We just add new
columns when using AVRO schema rules for Hudi so implementing comparator should
be possible. I'm OK to skip implementing the comparator for this PR in the
interest to land it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
