Copilot commented on code in PR #4574:
URL: https://github.com/apache/flink-cdc/pull/4574#discussion_r4254388841
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-fluss/src/main/java/org/apache/flink/cdc/connectors/fluss/sink/FlussMetaDataApplier.java:
##########
@@ -218,13 +225,76 @@ private void applyAddColumnTable(AddColumnEvent event) {
Admin admin = connection.getAdmin()) {
TableId tableId = event.tableId();
TablePath tablePath = new TablePath(tableId.getSchemaName(),
tableId.getTableName());
- admin.alterTable(tablePath, tableChanges, true).get();
+ applyAddColumnChanges(admin, tablePath, tableChanges);
} catch (Exception e) {
LOG.error("Failed to apply schema change {}", event, e);
throw new RuntimeException(e);
}
}
+ @VisibleForTesting
+ void applyAddColumnChanges(Admin admin, TablePath tablePath,
List<TableChange> tableChanges)
+ throws ExecutionException, InterruptedException {
+ try {
+ List<TableChange> pendingChanges = tableChanges;
+ while (!pendingChanges.isEmpty()) {
+ try {
+ admin.alterTable(tablePath, pendingChanges, false).get();
+ return;
+ } catch (ExecutionException e) {
+ if (!ExceptionUtils.findThrowable(e,
InvalidAlterTableException.class)
+ .isPresent()) {
+ throw e;
+ }
+ List<TableChange> missingColumns =
+
getMissingColumns(admin.getTableInfo(tablePath).get(), tableChanges);
+ // Schema reads may lag behind ADD when served by another
ZooKeeper session.
+ for (int retry = 0;
+ missingColumns.size() >= pendingChanges.size()
+ && retry < MAX_SCHEMA_READ_RETRIES;
+ retry++) {
+ Thread.sleep(SCHEMA_READ_RETRY_DELAY_MILLIS);
+ missingColumns =
+ getMissingColumns(
+ admin.getTableInfo(tablePath).get(),
tableChanges);
+ }
+ // Retry only when matching existing columns reduce the
pending additions.
+ if (missingColumns.size() >= pendingChanges.size()) {
+ throw e;
+ }
+ pendingChanges = missingColumns;
+ }
+ }
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw e;
+ }
+ }
+
+ private List<TableChange> getMissingColumns(
+ TableInfo tableInfo, List<TableChange> tableChanges) {
+ RowType rowType = tableInfo.getRowType();
+ List<TableChange> missingColumns = new ArrayList<>();
+ for (TableChange change : tableChanges) {
+ TableChange.AddColumn addition = (TableChange.AddColumn) change;
+ int index = rowType.getFieldIndex(addition.getName());
+ if (index < 0) {
+ missingColumns.add(change);
Review Comment:
A duplicated ADD request can be mistaken for a successful replay when one
matching target column already exists. For example, with changes `[name INT,
name INT]` and an existing `name INT`, both entries resolve to the same field,
`missingColumns` becomes empty, and this method returns success even though
Fluss would reject the event as an invalid duplicate. Reject duplicate names
while classifying the requested additions (and cover the pre-existing-column
case in the duplicate-additions test).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
