danny0405 commented on code in PR #12598:
URL: https://github.com/apache/hudi/pull/12598#discussion_r1908102789
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/clustering/ClusteringOperator.java:
##########
@@ -344,6 +347,22 @@ private Iterator<RowData>
readRecordsForGroupBaseFiles(List<ClusteringOperation>
return new ConcatenatingIterator<>(iteratorsForPartition);
}
+ /**
+ * Since there exists discrepancies between flink and spark dealing with
nullability of primary key field,
+ * and there may be some files written by spark, we force update the
nullability of record key to nullable,
+ * so that schema validating would not fail.
+ *
+ * @param schema the original schema to be updated
+ * @return schema that has nullability constraints reconciled
+ */
+ private Schema reconcileSchemaWithRecordKeyNullability(Schema schema) {
+ Option<String[]> recordKeyOp =
table.getMetaClient().getTableConfig().getRecordKeyFields();
Review Comment:
The default record key field is `uuid`, we should check the validity first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
