mxm commented on code in PR #18739:
URL: https://github.com/apache/iceberg/pull/18739#discussion_r4552292057
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DataConverter.java:
##########
@@ -64,6 +68,42 @@ static DataConverter identity() {
return object -> object;
}
+ /** Passes data that has the schema of the table on, after checking its
required columns. */
+ static DataConverter identity(Schema tableSchema) {
+ List<Types.NestedField> columns = tableSchema.columns();
+ int count = 0;
+ for (Types.NestedField column : columns) {
+ if (column.isRequired()) {
+ count++;
+ }
+ }
+
+ if (count == 0) {
+ return identity();
+ }
+
+ int[] indices = new int[count];
+ String[] names = new String[count];
+ int next = 0;
+ for (int i = 0; i < columns.size(); i++) {
+ if (columns.get(i).isRequired()) {
+ indices[next] = i;
+ names[next] = columns.get(i).name();
+ next++;
+ }
+ }
+
+ return new RequiredColumnsConverter(indices, names);
+ }
+
+ /**
+ * The writers only read the key of rows that delete, so only rows that
write a record must carry
+ * values for the required columns.
+ */
+ static boolean writesRecord(RowKind kind) {
+ return kind == RowKind.INSERT || kind == RowKind.UPDATE_AFTER;
Review Comment:
Does this hold outside upsert mode? Only an upsert DELETE is written by key.
Without upsert, the equality delete writer writes the entire row, and without
equality fields the INSERT-only writer writes the row as data.
Could we skip the check only for upsert records with equality fields?
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicSinkUtil.java:
##########
@@ -21,15 +21,45 @@
import java.util.Collections;
import java.util.Set;
import javax.annotation.Nullable;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
import org.apache.iceberg.Schema;
-import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
import org.apache.iceberg.relocated.com.google.common.collect.Sets;
import org.apache.iceberg.types.Types;
class DynamicSinkUtil {
private DynamicSinkUtil() {}
+ /**
+ * Validates that a record's upsert configuration is writable, before it is
shuffled to the
+ * writer. Upsert requires non-empty equality fields, and every partition
source column must be an
+ * equality field. Throws {@link IllegalStateException} otherwise so the
convert/update stage can
Review Comment:
NIT: The Javadoc still says `IllegalStateException`, the method throws
`DynamicRecordException` now.
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/TableUpdater.java:
##########
@@ -153,8 +153,12 @@ private TableMetadataCache.ResolvedSchemaInfo
findOrCreateSchema(
LOG.info(
"Triggering schema update for table {} {} to {}", identifier,
tableSchema, schema);
UpdateSchema updateApi = table.updateSchema();
- EvolveSchemaVisitor.visit(
- identifier, updateApi, tableSchema, schema, caseSensitive,
dropUnusedColumns);
+ try {
+ EvolveSchemaVisitor.visit(
+ identifier, updateApi, tableSchema, schema, caseSensitive,
dropUnusedColumns);
+ } catch (IllegalArgumentException e) {
+ throw new DynamicRecordException(e.getMessage(), e);
Review Comment:
Is the list of `DynamicRecordException` complete? `updateApi.commit()` below
and `CompareSchemasVisitor` also throw record-caused
`IllegalArgumentException`s.
For example: The record has `a` as a struct, the table has `a` as `int`. It
fails with "Not a struct type: int".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]