kumarpritam863 commented on code in PR #18739:
URL: https://github.com/apache/iceberg/pull/18739#discussion_r4588545496
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DataConverter.java:
##########
@@ -64,6 +68,42 @@ static DataConverter identity() {
return object -> object;
}
+ /** Passes data that has the schema of the table on, after checking its
required columns. */
+ static DataConverter identity(Schema tableSchema) {
+ List<Types.NestedField> columns = tableSchema.columns();
+ int count = 0;
+ for (Types.NestedField column : columns) {
+ if (column.isRequired()) {
+ count++;
+ }
+ }
+
+ if (count == 0) {
+ return identity();
+ }
+
+ int[] indices = new int[count];
+ String[] names = new String[count];
+ int next = 0;
+ for (int i = 0; i < columns.size(); i++) {
+ if (columns.get(i).isRequired()) {
+ indices[next] = i;
+ names[next] = columns.get(i).name();
+ next++;
+ }
+ }
+
+ return new RequiredColumnsConverter(indices, names);
+ }
+
+ /**
+ * The writers only read the key of rows that delete, so only rows that
write a record must carry
+ * values for the required columns.
+ */
+ static boolean writesRecord(RowKind kind) {
+ return kind == RowKind.INSERT || kind == RowKind.UPDATE_AFTER;
Review Comment:
Thanks @mxm for pointing this out. I was wondering to move this check out of
the converter into a helper in DynamicSinkUtil, called after conversion at the
two sites that already hold those values. The converter then loses the identity
wrapper, the required-column converter and writesRecord. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]