sd4324530 commented on code in PR #4626:
URL: https://github.com/apache/flink-cdc/pull/4626#discussion_r4602925375
##########
flink-cdc-common/src/main/java/org/apache/flink/cdc/common/event/visitor/SchemaChangeEventVisitor.java:
##########
@@ -45,6 +46,34 @@ public static <T, E extends Throwable> T visit(
@Nonnull VisitorHandler<TruncateTableEvent, T, E>
truncateTableEventVisitor,
@Nonnull VisitorHandler<AlterTableCommentEvent, T, E>
alterTableCommentEventVisitor)
throws E {
+ return visit(
+ event,
+ addColumnVisitor,
+ alterColumnTypeEventVisitor,
+ createTableEventVisitor,
+ dropColumnEventVisitor,
+ dropTableEventVisitor,
+ renameColumnEventVisitor,
+ truncateTableEventVisitor,
+ alterTableCommentEventVisitor,
+ reorderColumnEvent -> {
Review Comment:
**Problem:** The 8-argument overloads now throw
`UnsupportedSchemaChangeEventException` for reorder, but the only in-tree
caller, `HudiMetadataApplier` (`HudiMetadataApplier.java:123-160`), catches
`Exception` and wraps it in a plain `RuntimeException`.
`SchemaCoordinator.shouldIgnoreException` only tolerates
`UnsupportedSchemaChangeEventException` in `TRY_EVOLVE`, so Hudi jobs fail
instead of keeping the sink schema. Hudi is also missing from the new
documentation list.
**Suggestion:**
- Update Hudi to use the 10-arg overload with an explicit `reorder -> throw
new UnsupportedSchemaChangeEventException(event)` handler, and do not wrap
`SchemaEvolveException`.
- Add Hudi to the docs.
- Consider deprecating the 8-arg overloads so third-party sinks migrate.
##########
flink-cdc-common/src/main/java/org/apache/flink/cdc/common/utils/SchemaUtils.java:
##########
@@ -186,6 +189,75 @@ private static Schema applyDropColumnEvent(DropColumnEvent
event, Schema oldSche
return oldSchema.copy(columns);
}
+ private static Schema applyReorderColumnEvent(ReorderColumnEvent event,
Schema oldSchema) {
+ LinkedList<Column> columns = new LinkedList<>(oldSchema.getColumns());
+ for (ReorderColumnEvent.ColumnMove move : event.getColumnMoves()) {
+ moveColumn(columns, move, event);
+ }
+ return oldSchema.copy(columns);
+ }
+
+ private static void moveColumn(
+ LinkedList<Column> columns,
+ ReorderColumnEvent.ColumnMove move,
+ ReorderColumnEvent event) {
+ Column column = removeMovedColumn(columns, move, event);
+ switch (move.getPosition()) {
+ case FIRST:
+ columns.addFirst(column);
+ return;
+ case LAST:
+ columns.addLast(column);
+ return;
+ case BEFORE:
+ insertColumnRelative(columns, column, move, event, false);
+ return;
+ case AFTER:
+ insertColumnRelative(columns, column, move, event, true);
+ return;
+ default:
+ throw new IllegalArgumentException(
+ String.format(
+ "Unable to apply ReorderColumnEvent %s:
unsupported position %s",
+ event, move.getPosition()));
+ }
+ }
+
+ private static Column removeMovedColumn(
+ LinkedList<Column> columns,
+ ReorderColumnEvent.ColumnMove move,
+ ReorderColumnEvent event) {
+ for (int i = 0; i < columns.size(); i++) {
+ Column candidate = columns.get(i);
+ if (candidate.getName().equals(move.getColumnName())) {
+ columns.remove(i);
+ return candidate;
+ }
+ }
+ throw new IllegalArgumentException(
Review Comment:
**Problem:** This throws for a moved column (line 253 does the same for a
missing anchor) that is absent from the evolved schema. That happens for a
supported configuration: exclude `add.column` (or `rename.column`) but keep
`reorder.column`. The original schema has the column, the evolved schema does
not, and applying the reorder fails here.
`SchemaCoordinator.shouldIgnoreException` does not tolerate
`IllegalArgumentException`, so even `TRY_EVOLVE` fails. The new E2E
`testExcludedReorderAndAddColumns` excludes both types, so it does not cover
this.
**Suggestion:** Rewrite/filter reorder moves against the current evolved
schema in `SchemaDerivator.normalizeSchemaChangeEvents` (drop moves whose moved
or reference column is absent, and drop the event if nothing remains), or throw
`UnsupportedSchemaChangeEventException`. Add a test that excludes only
`add.column`.
##########
flink-cdc-common/src/main/java/org/apache/flink/cdc/common/event/AddColumnEvent.java:
##########
@@ -94,26 +122,72 @@ public ColumnWithPosition(Column addColumn) {
existedColumnName = null;
}
+ /**
+ * @deprecated Use {@link #of(Column,
org.apache.flink.cdc.common.event.ColumnPosition,
+ * String)} instead.
+ */
+ @Deprecated
public ColumnWithPosition(
Column addColumn, ColumnPosition position, @Nullable String
existedColumnName) {
this.addColumn = addColumn;
this.position = position;
this.existedColumnName = existedColumnName;
}
+ public static ColumnWithPosition of(
+ Column addColumn,
+ org.apache.flink.cdc.common.event.ColumnPosition position,
+ @Nullable String referenceColumnName) {
+ Preconditions.checkArgument(addColumn != null, "addColumn must not
be null");
+ Preconditions.checkArgument(position != null, "position must not
be null");
+ boolean relativePosition =
+ position ==
org.apache.flink.cdc.common.event.ColumnPosition.BEFORE
+ || position ==
org.apache.flink.cdc.common.event.ColumnPosition.AFTER;
+ Preconditions.checkArgument(
+ relativePosition == (referenceColumnName != null),
+ relativePosition
+ ? "Column %s with position %s must define
referenceColumnName"
+ : "Column %s with position %s must not define
referenceColumnName",
+ addColumn.getName(),
+ position);
+ Preconditions.checkArgument(
+ !addColumn.getName().equals(referenceColumnName),
+ "Added column %s cannot reference itself",
+ addColumn.getName());
+ return new ColumnWithPosition(
+ addColumn, ColumnPosition.fromColumnPosition(position),
referenceColumnName);
+ }
+
public Column getAddColumn() {
return addColumn;
}
+ /**
+ * @deprecated Use {@link #getColumnPosition()} instead.
+ */
+ @Deprecated
public ColumnPosition getPosition() {
return position;
}
+ public org.apache.flink.cdc.common.event.ColumnPosition
getColumnPosition() {
Review Comment:
**Problem:** `getColumnPosition()` dereferences `position` without a null
check, but the still-public deprecated constructor at line 130 accepts a null
`position`. Code that previously relied on the null-safe
`AddColumnEvent.ColumnPosition.LAST.equals(e.getPosition())` check in
`SchemaUtils.transformSchemaChangeEvent` will now throw an NPE.
**Suggestion:** Make this method null-safe (return null, or `checkNotNull`
with a clear message), or validate `position` in the deprecated constructor.
Low likelihood, but it is a behavior change.
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-starrocks/src/main/java/org/apache/flink/cdc/connectors/starrocks/sink/StarRocksMetadataApplier.java:
##########
@@ -356,4 +361,173 @@ private void applyDropTable(DropTableEvent
dropTableEvent) {
throw new SchemaEvolveException(dropTableEvent, e.getMessage(), e);
}
}
+
+ private void applyReorderColumn(ReorderColumnEvent reorderColumnEvent)
Review Comment:
**Problem:** MySQL parses `MODIFY COLUMN ... AFTER` into an
`AlterColumnTypeEvent` followed by this `ReorderColumnEvent`
(`CustomAlterTableParserListener.java:450-471`). `applyAlterColumnType` (line
324) calls `StarRocksEnrichedCatalog.alterColumnType`, which submits an
asynchronous StarRocks schema-change job and returns without waiting
(`StarRocksEnrichedCatalog.java:121-152`). This method then immediately issues
the position `ALTER`. StarRocks allows only one schema change per table, so the
second `ALTER` is rejected with `SCHEMA_CHANGE`, and the job fails
deterministically. This also happens for a pure position change, because the
type event is always emitted.
**Suggestion:** Make `alterColumnType` wait for its job like `executeAlter`,
wait for any in-flight column alter job before the first move, or combine type
and position into one `ALTER`. Add a test that applies `AlterColumnTypeEvent`
and then `ReorderColumnEvent`; the current IT only sends the reorder.
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/main/java/org/apache/flink/cdc/connectors/mysql/source/parser/CustomAlterTableParserListener.java:
##########
@@ -431,6 +451,24 @@ public void
exitAlterByModifyColumn(MySqlParser.AlterByModifyColumnContext ctx)
new AlterColumnTypeEvent(currentTable,
typeMapping);
alterColumnTypeEvent.addColumnComment(columnName,
column.comment());
changes.add(alterColumnTypeEvent);
+ if (columnName != null && ctx.FIRST() != null) {
+ changes.add(
+ new ReorderColumnEvent(
+ currentTable,
+ Collections.singletonList(
+
ReorderColumnEvent.ColumnMove.first(columnName))));
+ } else if (columnName != null && ctx.AFTER() != null) {
Review Comment:
**Problem:** Column identifiers in MySQL are case-insensitive, but
`parseName` preserves the DDL casing. With `isTableIdCaseInsensitive=false`
(`lower_case_table_names=0`, the Linux default), `MODIFY COLUMN NAME ... AFTER
ID`, or a case-only `CHANGE COLUMN name Name ... FIRST`, emits a move for
`NAME`/`Name` while the tracked schema contains the create-time `name`.
`SchemaUtils.removeMovedColumn` / `insertColumnRelative` use exact `equals`, so
the job fails. The new case-normalization test only uses
`isTableIdCaseInsensitive=true`.
**Suggestion:** Resolve the parsed names against the existing table schema
case-insensitively and emit the canonical column name. Add a test with
`isTableIdCaseInsensitive=false` and mixed-case DDL.
##########
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/PostTransformOperator.java:
##########
@@ -282,6 +283,11 @@ private Optional<Event> processSchemaChangeEvent(
if (hasAsteriskMap.getOrDefault(tableId, true)) {
// See comments in PreTransformOperator#cacheChangeSchema method.
+ if (event instanceof ReorderColumnEvent) {
Review Comment:
**Problem:** Non-asterisk projections still silently misalign. When
`hasAsteriskMap` is false, `PreTransformOperator.cacheChangeSchema` calls
`SchemaUtils.transformSchemaChangeEvent(false, ...)`, which returns
`Optional.empty()` for `ReorderColumnEvent` (`SchemaUtils.java:379-383`), so
the event never reaches `PostTransformOperator`. However,
`cachePreTransformProcessor` (`PreTransformOperator.java:259`) rebuilds the
pre-transform processor from the reordered source schema, so the rows emitted
by `PreTransformOperator` change order while this operator pre-schema stays the
same. Every subsequent record is then decoded positionally with the old schema,
and values are silently swapped.
**Suggestion:** Handle reorder in the non-asterisk branch as well — for
example, after rebuilding the processor, derive `oldPreSchema`/`newPreSchema`
and forward `ReorderColumnEventTransformer.transformReorderColumnEvent(...)`,
or keep the pre-transform output order stable. Add a test with a non-asterisk
projection such as `id, age` and a move of `age`; the current test only covers
`*` (`testReorderWithPreWildcardAndComputedColumn`).
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-starrocks/src/main/java/org/apache/flink/cdc/connectors/starrocks/sink/StarRocksMetadataApplier.java:
##########
@@ -356,4 +361,173 @@ private void applyDropTable(DropTableEvent
dropTableEvent) {
throw new SchemaEvolveException(dropTableEvent, e.getMessage(), e);
}
}
+
+ private void applyReorderColumn(ReorderColumnEvent reorderColumnEvent)
+ throws SchemaEvolveException {
+ TableId tableId = reorderColumnEvent.tableId();
+ StarRocksTable currentTable;
+ try {
+ currentTable =
+ catalog.getTable(tableId.getSchemaName(),
tableId.getTableName()).orElse(null);
+ } catch (StarRocksCatalogException e) {
+ throw new SchemaEvolveException(reorderColumnEvent, "Failed to
load table metadata", e);
+ }
+ if (currentTable == null) {
+ throw new SchemaEvolveException(reorderColumnEvent, "Table does
not exist", null);
+ }
+
+ List<StarRocksColumn> targetColumns =
+ applyColumnMoves(currentTable.getColumns(),
reorderColumnEvent);
+ validateKeyColumnsRemainFirst(currentTable, targetColumns,
reorderColumnEvent);
+ if (sameColumnOrder(currentTable, targetColumns)) {
+ return;
+ }
+
+ List<StarRocksColumn> appliedColumns = new
ArrayList<>(currentTable.getColumns());
+ Set<String> tableKeys =
+ currentTable.getTableKeys().orElseGet(ArrayList::new).stream()
+ .collect(Collectors.toSet());
+ try {
+ for (int index = 0; index < targetColumns.size(); index++) {
Review Comment:
**Problem:** This loop turns one column move into up to n-1 separate
`ALTER`s: moving `a` to the end of `[id,a,b,c,d]` produces `b AFTER id`, `c
AFTER b`, `d AFTER c` instead of a single `a AFTER d`. Each
`alterColumnPosition` is a full StarRocks schema change, is waited on
synchronously with the full `schema-change.timeout`, and opens 3-4 new JDBC
connections (`StarRocksEnrichedCatalog.java:231-247, 249-281, 303-348`). The
poll loop also aborts on the first `SQLException` instead of retrying like
`executeAlter`. For a wide table this can block the metadata path for a very
long time and leave the table partially reordered.
**Suggestion:**
- Apply the event moves directly (one `ALTER` per `ColumnMove`), or compute
a minimal move set from current to target.
- Reuse one connection for polling, add connect/query timeouts and
transient-error retries, and use one overall deadline.
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-starrocks/src/main/java/org/apache/flink/cdc/connectors/starrocks/sink/StarRocksMetadataApplier.java:
##########
@@ -356,4 +361,173 @@ private void applyDropTable(DropTableEvent
dropTableEvent) {
throw new SchemaEvolveException(dropTableEvent, e.getMessage(), e);
}
}
+
+ private void applyReorderColumn(ReorderColumnEvent reorderColumnEvent)
+ throws SchemaEvolveException {
+ TableId tableId = reorderColumnEvent.tableId();
+ StarRocksTable currentTable;
+ try {
+ currentTable =
+ catalog.getTable(tableId.getSchemaName(),
tableId.getTableName()).orElse(null);
+ } catch (StarRocksCatalogException e) {
+ throw new SchemaEvolveException(reorderColumnEvent, "Failed to
load table metadata", e);
+ }
+ if (currentTable == null) {
+ throw new SchemaEvolveException(reorderColumnEvent, "Table does
not exist", null);
+ }
+
+ List<StarRocksColumn> targetColumns =
+ applyColumnMoves(currentTable.getColumns(),
reorderColumnEvent);
+ validateKeyColumnsRemainFirst(currentTable, targetColumns,
reorderColumnEvent);
+ if (sameColumnOrder(currentTable, targetColumns)) {
+ return;
+ }
+
+ List<StarRocksColumn> appliedColumns = new
ArrayList<>(currentTable.getColumns());
+ Set<String> tableKeys =
+ currentTable.getTableKeys().orElseGet(ArrayList::new).stream()
+ .collect(Collectors.toSet());
+ try {
+ for (int index = 0; index < targetColumns.size(); index++) {
+ StarRocksColumn targetColumn = targetColumns.get(index);
+ if (appliedColumns
+ .get(index)
+ .getColumnName()
+ .equals(targetColumn.getColumnName())) {
+ continue;
+ }
+
+ String precedingColumn =
+ index == 0 ? null : targetColumns.get(index -
1).getColumnName();
+ catalog.alterColumnPosition(
+ tableId.getSchemaName(),
+ tableId.getTableName(),
+ targetColumn,
+ tableKeys.contains(targetColumn.getColumnName()),
+ precedingColumn,
+ schemaChangeConfig.getTimeoutSecond());
+ appliedColumns.remove(targetColumn);
+ appliedColumns.add(index, targetColumn);
+ }
+ LOG.info("Successful to apply reorder column, event: {}",
reorderColumnEvent);
+ } catch (StarRocksCatalogException e) {
+ if (isTargetOrderPresent(tableId, targetColumns)) {
+ LOG.info(
+ "Successful to apply reorder column after a failed
request, event: {}",
+ reorderColumnEvent,
+ e);
+ return;
+ }
+ throw new SchemaEvolveException(
+ reorderColumnEvent, "Failed to apply reorder column", e);
+ }
+ }
+
+ private static StarRocksColumn findColumn(List<StarRocksColumn> columns,
String columnName) {
+ if (columnName == null) {
+ return null;
+ }
+ for (StarRocksColumn column : columns) {
+ if (column.getColumnName().equals(columnName)) {
+ return column;
+ }
+ }
+ return null;
+ }
+
+ private List<StarRocksColumn> applyColumnMoves(
+ List<StarRocksColumn> currentColumns, ReorderColumnEvent
reorderColumnEvent) {
+ if (reorderColumnEvent.getColumnMoves() == null) {
+ throw new SchemaEvolveException(
+ reorderColumnEvent, "Column moves must not be null", null);
+ }
+
+ List<StarRocksColumn> targetColumns = new ArrayList<>(currentColumns);
+ for (ReorderColumnEvent.ColumnMove move :
reorderColumnEvent.getColumnMoves()) {
+ if (move == null || move.getColumnName() == null ||
move.getPosition() == null) {
+ throw new SchemaEvolveException(reorderColumnEvent, "Invalid
column move", null);
+ }
+ StarRocksColumn movedColumn = findColumn(targetColumns,
move.getColumnName());
+ if (movedColumn == null) {
+ throw new SchemaEvolveException(
+ reorderColumnEvent,
+ "Referenced column does not exist: " +
move.getColumnName(),
+ null);
+ }
+ targetColumns.remove(movedColumn);
+
+ int targetIndex;
+ switch (move.getPosition()) {
+ case FIRST:
+ targetIndex = 0;
+ break;
+ case LAST:
+ targetIndex = targetColumns.size();
+ break;
+ case BEFORE:
+ case AFTER:
+ StarRocksColumn anchorColumn =
+ findColumn(targetColumns,
move.getReferenceColumnName());
+ if (anchorColumn == null) {
+ throw new SchemaEvolveException(
+ reorderColumnEvent,
+ "Referenced anchor column does not exist: "
+ + move.getReferenceColumnName(),
+ null);
+ }
+ targetIndex = targetColumns.indexOf(anchorColumn);
+ if (move.getPosition() == ColumnPosition.AFTER) {
+ targetIndex++;
+ }
+ break;
+ default:
+ throw new SchemaEvolveException(
+ reorderColumnEvent, "Unsupported column position",
null);
+ }
+ targetColumns.add(targetIndex, movedColumn);
+ }
+ return targetColumns;
+ }
+
+ private void validateKeyColumnsRemainFirst(
+ StarRocksTable table,
+ List<StarRocksColumn> targetColumns,
+ ReorderColumnEvent reorderColumnEvent) {
+ List<String> tableKeys =
table.getTableKeys().orElseGet(ArrayList::new);
+ for (int index = 0; index < tableKeys.size(); index++) {
+ if
(!tableKeys.get(index).equals(targetColumns.get(index).getColumnName())) {
+ throw new SchemaEvolveException(
Review Comment:
**Problem:** Key-order violations (here) and missing columns in
`applyColumnMoves` (line 438) throw a plain `SchemaEvolveException`.
`SchemaCoordinator.shouldIgnoreException` only tolerates
`UnsupportedSchemaChangeEventException` in `TRY_EVOLVE`, so these moves fail
the job even in tolerant mode. This contradicts the new docs, which say
`TRY_EVOLVE` keeps the sink schema.
**Suggestion:** Throw `UnsupportedSchemaChangeEventException` (or a
dedicated tolerated subtype) for unsupported moves. Paimon `checkArgument`
(`PaimonMetadataApplier.java:561`) and the `SchemaUtils` missing-column path
(lines 237/253) need the same treatment.
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-starrocks/src/test/java/org/apache/flink/cdc/connectors/starrocks/sink/StarRocksMetadataApplierTest.java:
##########
@@ -62,13 +72,233 @@ public void setup() {
.put(TABLE_CREATE_NUM_BUCKETS.key(), "10")
.put("table.create.properties.replication_num", "5")
.build());
- SchemaChangeConfig schemaChangeConfig =
SchemaChangeConfig.from(configuration);
- TableCreateConfig tableCreateConfig =
TableCreateConfig.from(configuration);
+ this.schemaChangeConfig = SchemaChangeConfig.from(configuration);
+ this.tableCreateConfig = TableCreateConfig.from(configuration);
this.catalog = new MockStarRocksCatalog();
this.metadataApplier =
new StarRocksMetadataApplier(catalog, tableCreateConfig,
schemaChangeConfig);
}
+ @Test
+ void testReorderColumnSupportsMovesAndPreservesMetadata() throws Exception
{
Review Comment:
**Problem:** This test cannot catch metadata loss:
`MockStarRocksCatalog.alterColumnPosition` (`MockStarRocksCatalog.java:200`)
ignores the passed `StarRocksColumn`/`isKey` and just reorders the existing
column objects, so dropping DEFAULT/nullability/COMMENT in
`loadColumnDefault`/`buildAlterColumnPositionSql` leaves it green. The IT
(`StarRocksMetadataApplierITCase.java:737`) also sends only the reorder event
and has no defaults/comments.
**Suggestion:** Make the mock apply the restated definition (or assert on
real `information_schema` values), and add an IT with DEFAULT/COMMENT and a
preceding `AlterColumnTypeEvent`. Also cover defaults like
`CURRENT_TIMESTAMP(3)`, which `formatDefaultValue` currently quotes as a string.
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-paimon/src/main/java/org/apache/flink/cdc/connectors/paimon/sink/PaimonMetadataApplier.java:
##########
@@ -502,4 +506,91 @@ private Map<String, String> filterBlobOptions(
private boolean isSupportedTypeForBlob(DataType dataType) {
return dataType.isAnyOf(BINARY_STRING, CHARACTER_STRING);
}
+
+ private void applyReorderColumn(ReorderColumnEvent event) throws
SchemaEvolveException {
+ try {
+ Identifier identifier =
+ new Identifier(event.tableId().getSchemaName(),
event.tableId().getTableName());
+ FileStoreTable table = (FileStoreTable)
catalog.getTable(identifier);
+ List<String> simulatedColumnNames = new
ArrayList<>(table.rowType().getFieldNames());
+ List<SchemaChange> tableChangeList = new ArrayList<>();
+ for (ReorderColumnEvent.ColumnMove columnMove :
event.getColumnMoves()) {
+ SchemaChange.Move move = toPaimonMove(columnMove);
+ if (applyColumnMove(simulatedColumnNames, columnMove)) {
+
tableChangeList.add(SchemaChange.updateColumnPosition(move));
+ }
+ }
+ if (!tableChangeList.isEmpty()) {
+ catalog.alterTable(identifier, tableChangeList, true);
+ }
+ } catch (Catalog.TableNotExistException
+ | Catalog.ColumnAlreadyExistException
+ | Catalog.ColumnNotExistException e) {
+ throw new SchemaEvolveException(event, e.getMessage(), e);
+ }
+ }
+
+ private SchemaChange.Move toPaimonMove(ReorderColumnEvent.ColumnMove
columnMove) {
+ String columnName = columnMove.getColumnName();
+ switch (columnMove.getPosition()) {
+ case FIRST:
+ return SchemaChange.Move.first(columnName);
+ case LAST:
+ return SchemaChange.Move.last(columnName);
+ case BEFORE:
+ return SchemaChange.Move.before(
+ columnName,
+ checkNotNull(
+ columnMove.getReferenceColumnName(),
+ "Existing column name must be provided for
BEFORE position"));
+ case AFTER:
+ return SchemaChange.Move.after(
+ columnName,
+ checkNotNull(
+ columnMove.getReferenceColumnName(),
+ "Existing column name must be provided for
AFTER position"));
+ default:
+ throw new IllegalArgumentException(
+ "Unknown column position: " +
columnMove.getPosition());
+ }
+ }
+
+ private boolean applyColumnMove(
+ List<String> simulatedColumnNames, ReorderColumnEvent.ColumnMove
columnMove) {
+ List<String> reorderedColumnNames = new
ArrayList<>(simulatedColumnNames);
+ checkArgument(
Review Comment:
**Problem:** Move application is now implemented four times with slightly
different semantics: `SchemaUtils.applyReorderColumnEvent`
(`SchemaUtils.java:192-259`), this method,
`StarRocksMetadataApplier.applyColumnMoves` (lines 438-491), and
`ReorderColumnEventTransformer`. Missing columns here throw
`IllegalArgumentException`, which `TRY_EVOLVE` does not tolerate.
**Suggestion:** Extract a shared helper (for example
`applyMoves(List<String>, List<ColumnMove>)` returning the reordered names, or
a normalized/minimal move list on `ReorderColumnEvent`) so no-op handling and
missing-column policy stay consistent. Use
`UnsupportedSchemaChangeEventException` for unsupported moves.
##########
docs/content/docs/core-concept/schema-evolution.md:
##########
@@ -146,6 +147,12 @@ The following YAML configuration is set to include
`CreateTableEvent` and column
```yaml
sink:
- include.schema.changes: [create.table, column] # This matches CreateTable,
AddColumn, AlterColumnType, RenameColumn, and DropColumn Events
+ include.schema.changes: [create.table, column] # This matches CreateTable,
AddColumn, AlterColumnType, RenameColumn, ReorderColumn, and DropColumn Events
exclude.schema.changes: [drop.column] # This excludes DropColumn Events
```
+
+### Column order changes
+
+The MySQL pipeline source emits `ReorderColumnEvent` for `MODIFY COLUMN` and
`CHANGE COLUMN` statements with `FIRST` or `AFTER` clauses. Renaming, type
changes, and column moves in a compound DDL are applied in statement order. The
`reorder` and `column` families both include `reorder.column`.
+
+Kafka updates its serialization schema. Paimon applies physical column moves,
while StarRocks only accepts moves that keep primary key columns first and in
their existing order. Doris, Fluss, Iceberg, MaxCompute, and OceanBase reject
physical column reordering explicitly. In `EVOLVE` mode, an unsupported move
fails the job rather than silently misaligning field values. In `TRY_EVOLVE`
mode, the framework retains the sink schema and maps subsequent records to that
schema. `LENIENT` mode does not reorder the sink columns; record values are
mapped to the unchanged column order. Explicitly excluding `reorder.column`
also keeps the sink column order unchanged.
Review Comment:
**Problem:** There are three documentation gaps:
1. Hudi is missing from the list of sinks that reject reorder explicitly;
`HudiMetadataApplier` currently fails `TRY_EVOLVE` as described in the review
comment on `SchemaChangeEventVisitor`.
2. The `TRY_EVOLVE` sentence is only true for sinks that throw
`UnsupportedSchemaChangeEventException`, but StarRocks validation, Paimon
missing columns and `SchemaUtils` currently throw other exceptions. Either fix
those or qualify the sentence.
3. The StarRocks connector page
(`docs/content/docs/connectors/pipeline-connectors/starrocks.md:257-261`) still
says only add/drop/rename/alter-type and "new column always last". Document
reorder support, the key-order constraint, and the one-schema-change-at-a-time
behavior.
**Suggestion:** Also consider an upgrade note: the default accepted set is
`ALL` (`SinkDef.java:56-57`), so existing EVOLVE pipelines will now fail or
physically reorder on the next `MODIFY ... AFTER`.
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-starrocks/src/main/java/org/apache/flink/cdc/connectors/starrocks/sink/StarRocksMetadataApplier.java:
##########
@@ -356,4 +361,173 @@ private void applyDropTable(DropTableEvent
dropTableEvent) {
throw new SchemaEvolveException(dropTableEvent, e.getMessage(), e);
}
}
+
+ private void applyReorderColumn(ReorderColumnEvent reorderColumnEvent)
+ throws SchemaEvolveException {
+ TableId tableId = reorderColumnEvent.tableId();
+ StarRocksTable currentTable;
+ try {
+ currentTable =
Review Comment:
**Problem:** `catalog.getTable()` reads `information_schema.COLUMNS` without
`ORDER BY ORDINAL_POSITION`. `StarRocksTable` is documented as sorted by
ordinal position, but the builder does not sort it. The entire logic here
(`applyColumnMoves`, `sameColumnOrder`, `precedingColumn`) and the `tableKeys`
order depend on the list being in physical order.
**Suggestion:** Sort the columns by `getOrdinalPosition()` (and derive
`tableKeys` accordingly) before using them, or add an ordered query/override in
`StarRocksEnrichedCatalog`.
##########
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/ReorderColumnEventTransformer.java:
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.cdc.runtime.operators.transform;
+
+import org.apache.flink.cdc.common.annotation.Internal;
+import org.apache.flink.cdc.common.event.ReorderColumnEvent;
+import org.apache.flink.cdc.common.schema.Schema;
+import org.apache.flink.cdc.common.utils.Preconditions;
+
+import javax.annotation.CheckReturnValue;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+/** Transforms source reorder events into moves that apply to post-transformed
schemas. */
+@Internal
+public final class ReorderColumnEventTransformer {
+
+ private ReorderColumnEventTransformer() {}
+
+ @CheckReturnValue
+ public static Optional<ReorderColumnEvent> transformReorderColumnEvent(
+ ReorderColumnEvent event, Schema schemaBeforeChange, Schema
schemaAfterChange) {
+ List<String> oldColumnNames = schemaBeforeChange.getColumnNames();
+ List<String> newColumnNames = schemaAfterChange.getColumnNames();
+ if (oldColumnNames.equals(newColumnNames)) {
+ return Optional.empty();
+ }
+
+ Preconditions.checkArgument(
+ oldColumnNames.size() == newColumnNames.size()
+ && new HashSet<>(oldColumnNames).equals(new
HashSet<>(newColumnNames)),
+ "Unable to transform ReorderColumnEvent %s between schemas
with different columns: %s and %s",
+ event,
+ oldColumnNames,
+ newColumnNames);
+
+ List<String> stableAnchors = longestCommonSubsequence(oldColumnNames,
newColumnNames);
+ Map<String, Integer> newColumnPositions = new HashMap<>();
+ for (int i = 0; i < newColumnNames.size(); i++) {
+ newColumnPositions.put(newColumnNames.get(i), i);
+ }
+ List<ReorderColumnEvent.ColumnMove> columnMoves = new ArrayList<>();
+ if (stableAnchors.isEmpty()) {
Review Comment:
**Problem:** This branch is unreachable: two non-empty permutations of the
same column set always have an LCS of at least one element, and equal lists
return at line 45. It has no test coverage.
**Suggestion:** Remove it. The `targetIndex = 0` initialization in the
following loop already handles the first-anchor case.
##########
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/ReorderColumnEventTransformer.java:
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.cdc.runtime.operators.transform;
+
+import org.apache.flink.cdc.common.annotation.Internal;
+import org.apache.flink.cdc.common.event.ReorderColumnEvent;
+import org.apache.flink.cdc.common.schema.Schema;
+import org.apache.flink.cdc.common.utils.Preconditions;
+
+import javax.annotation.CheckReturnValue;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+/** Transforms source reorder events into moves that apply to post-transformed
schemas. */
+@Internal
+public final class ReorderColumnEventTransformer {
+
+ private ReorderColumnEventTransformer() {}
+
+ @CheckReturnValue
+ public static Optional<ReorderColumnEvent> transformReorderColumnEvent(
+ ReorderColumnEvent event, Schema schemaBeforeChange, Schema
schemaAfterChange) {
+ List<String> oldColumnNames = schemaBeforeChange.getColumnNames();
+ List<String> newColumnNames = schemaAfterChange.getColumnNames();
+ if (oldColumnNames.equals(newColumnNames)) {
+ return Optional.empty();
+ }
+
+ Preconditions.checkArgument(
+ oldColumnNames.size() == newColumnNames.size()
+ && new HashSet<>(oldColumnNames).equals(new
HashSet<>(newColumnNames)),
+ "Unable to transform ReorderColumnEvent %s between schemas
with different columns: %s and %s",
+ event,
+ oldColumnNames,
+ newColumnNames);
+
+ List<String> stableAnchors = longestCommonSubsequence(oldColumnNames,
newColumnNames);
+ Map<String, Integer> newColumnPositions = new HashMap<>();
+ for (int i = 0; i < newColumnNames.size(); i++) {
+ newColumnPositions.put(newColumnNames.get(i), i);
+ }
+ List<ReorderColumnEvent.ColumnMove> columnMoves = new ArrayList<>();
+ if (stableAnchors.isEmpty()) {
+ for (int i = newColumnNames.size() - 1; i >= 0; i--) {
+
columnMoves.add(ReorderColumnEvent.ColumnMove.first(newColumnNames.get(i)));
+ }
+ } else {
+ int targetIndex = 0;
+ for (int anchorIndex = 0; anchorIndex < stableAnchors.size();
anchorIndex++) {
+ String anchor = stableAnchors.get(anchorIndex);
+ int anchorTargetIndex = newColumnPositions.get(anchor);
+ if (anchorIndex == 0) {
+ for (int i = anchorTargetIndex - 1; i >= targetIndex; i--)
{
+
columnMoves.add(ReorderColumnEvent.ColumnMove.first(newColumnNames.get(i)));
+ }
+ } else {
+ for (int i = targetIndex; i < anchorTargetIndex; i++) {
+ columnMoves.add(
+ ReorderColumnEvent.ColumnMove.before(
+ newColumnNames.get(i), anchor));
+ }
+ }
+ targetIndex = anchorTargetIndex + 1;
+ }
+ for (int i = targetIndex; i < newColumnNames.size(); i++) {
+
columnMoves.add(ReorderColumnEvent.ColumnMove.last(newColumnNames.get(i)));
+ }
+ }
+
+ return Optional.of(new ReorderColumnEvent(event.tableId(),
columnMoves));
+ }
+
+ private static List<String> longestCommonSubsequence(
+ List<String> oldColumnNames, List<String> newColumnNames) {
+ int[][] lengths = new int[oldColumnNames.size() +
1][newColumnNames.size() + 1];
Review Comment:
**Problem:** This allocates an `(n+1) x (m+1)` `int` matrix per transform
instance per reorder event. For a 10,000-column table that is about 400 MB and
about 100M comparisons; 1,000 columns is about 4 MB per instance.
**Suggestion:** Since the two schemas are permutations of the same column
set, compute the LIS over the old-index to new-index permutation in `O(n log
n)`, or derive moves directly from the event. At minimum, guard the matrix size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]