sd4324530 commented on code in PR #4626:
URL: https://github.com/apache/flink-cdc/pull/4626#discussion_r4602925375


##########
flink-cdc-common/src/main/java/org/apache/flink/cdc/common/event/visitor/SchemaChangeEventVisitor.java:
##########
@@ -45,6 +46,34 @@ public static <T, E extends Throwable> T visit(
             @Nonnull VisitorHandler<TruncateTableEvent, T, E> 
truncateTableEventVisitor,
             @Nonnull VisitorHandler<AlterTableCommentEvent, T, E> 
alterTableCommentEventVisitor)
             throws E {
+        return visit(
+                event,
+                addColumnVisitor,
+                alterColumnTypeEventVisitor,
+                createTableEventVisitor,
+                dropColumnEventVisitor,
+                dropTableEventVisitor,
+                renameColumnEventVisitor,
+                truncateTableEventVisitor,
+                alterTableCommentEventVisitor,
+                reorderColumnEvent -> {

Review Comment:
   **Problem:** The 8-argument overloads now throw 
`UnsupportedSchemaChangeEventException` for reorder, but the only in-tree 
caller, `HudiMetadataApplier` (`HudiMetadataApplier.java:123-160`), catches 
`Exception` and wraps it in a plain `RuntimeException`. 
`SchemaCoordinator.shouldIgnoreException` only tolerates 
`UnsupportedSchemaChangeEventException` in `TRY_EVOLVE`, so Hudi jobs fail 
instead of keeping the sink schema. Hudi is also missing from the new 
documentation list.
   
   **Suggestion:**
   - Update Hudi to use the 10-arg overload with an explicit `reorder -> throw 
new UnsupportedSchemaChangeEventException(event)` handler, and do not wrap 
`SchemaEvolveException`.
   - Add Hudi to the docs.
   - Consider deprecating the 8-arg overloads so third-party sinks migrate.



##########
flink-cdc-common/src/main/java/org/apache/flink/cdc/common/utils/SchemaUtils.java:
##########
@@ -186,6 +189,75 @@ private static Schema applyDropColumnEvent(DropColumnEvent 
event, Schema oldSche
         return oldSchema.copy(columns);
     }
 
+    private static Schema applyReorderColumnEvent(ReorderColumnEvent event, 
Schema oldSchema) {
+        LinkedList<Column> columns = new LinkedList<>(oldSchema.getColumns());
+        for (ReorderColumnEvent.ColumnMove move : event.getColumnMoves()) {
+            moveColumn(columns, move, event);
+        }
+        return oldSchema.copy(columns);
+    }
+
+    private static void moveColumn(
+            LinkedList<Column> columns,
+            ReorderColumnEvent.ColumnMove move,
+            ReorderColumnEvent event) {
+        Column column = removeMovedColumn(columns, move, event);
+        switch (move.getPosition()) {
+            case FIRST:
+                columns.addFirst(column);
+                return;
+            case LAST:
+                columns.addLast(column);
+                return;
+            case BEFORE:
+                insertColumnRelative(columns, column, move, event, false);
+                return;
+            case AFTER:
+                insertColumnRelative(columns, column, move, event, true);
+                return;
+            default:
+                throw new IllegalArgumentException(
+                        String.format(
+                                "Unable to apply ReorderColumnEvent %s: 
unsupported position %s",
+                                event, move.getPosition()));
+        }
+    }
+
+    private static Column removeMovedColumn(
+            LinkedList<Column> columns,
+            ReorderColumnEvent.ColumnMove move,
+            ReorderColumnEvent event) {
+        for (int i = 0; i < columns.size(); i++) {
+            Column candidate = columns.get(i);
+            if (candidate.getName().equals(move.getColumnName())) {
+                columns.remove(i);
+                return candidate;
+            }
+        }
+        throw new IllegalArgumentException(

Review Comment:
   **Problem:** This throws for a moved column (line 253 does the same for a 
missing anchor) that is absent from the evolved schema. That happens for a 
supported configuration: exclude `add.column` (or `rename.column`) but keep 
`reorder.column`. The original schema has the column, the evolved schema does 
not, and applying the reorder fails here. 
`SchemaCoordinator.shouldIgnoreException` does not tolerate 
`IllegalArgumentException`, so even `TRY_EVOLVE` fails. The new E2E 
`testExcludedReorderAndAddColumns` excludes both types, so it does not cover 
this.
   
   **Suggestion:** Rewrite/filter reorder moves against the current evolved 
schema in `SchemaDerivator.normalizeSchemaChangeEvents` (drop moves whose moved 
or reference column is absent, and drop the event if nothing remains), or throw 
`UnsupportedSchemaChangeEventException`. Add a test that excludes only 
`add.column`.



##########
flink-cdc-common/src/main/java/org/apache/flink/cdc/common/event/AddColumnEvent.java:
##########
@@ -94,26 +122,72 @@ public ColumnWithPosition(Column addColumn) {
             existedColumnName = null;
         }
 
+        /**
+         * @deprecated Use {@link #of(Column, 
org.apache.flink.cdc.common.event.ColumnPosition,
+         *     String)} instead.
+         */
+        @Deprecated
         public ColumnWithPosition(
                 Column addColumn, ColumnPosition position, @Nullable String 
existedColumnName) {
             this.addColumn = addColumn;
             this.position = position;
             this.existedColumnName = existedColumnName;
         }
 
+        public static ColumnWithPosition of(
+                Column addColumn,
+                org.apache.flink.cdc.common.event.ColumnPosition position,
+                @Nullable String referenceColumnName) {
+            Preconditions.checkArgument(addColumn != null, "addColumn must not 
be null");
+            Preconditions.checkArgument(position != null, "position must not 
be null");
+            boolean relativePosition =
+                    position == 
org.apache.flink.cdc.common.event.ColumnPosition.BEFORE
+                            || position == 
org.apache.flink.cdc.common.event.ColumnPosition.AFTER;
+            Preconditions.checkArgument(
+                    relativePosition == (referenceColumnName != null),
+                    relativePosition
+                            ? "Column %s with position %s must define 
referenceColumnName"
+                            : "Column %s with position %s must not define 
referenceColumnName",
+                    addColumn.getName(),
+                    position);
+            Preconditions.checkArgument(
+                    !addColumn.getName().equals(referenceColumnName),
+                    "Added column %s cannot reference itself",
+                    addColumn.getName());
+            return new ColumnWithPosition(
+                    addColumn, ColumnPosition.fromColumnPosition(position), 
referenceColumnName);
+        }
+
         public Column getAddColumn() {
             return addColumn;
         }
 
+        /**
+         * @deprecated Use {@link #getColumnPosition()} instead.
+         */
+        @Deprecated
         public ColumnPosition getPosition() {
             return position;
         }
 
+        public org.apache.flink.cdc.common.event.ColumnPosition 
getColumnPosition() {

Review Comment:
   **Problem:** `getColumnPosition()` dereferences `position` without a null 
check, but the still-public deprecated constructor at line 130 accepts a null 
`position`. Code that previously relied on the null-safe 
`AddColumnEvent.ColumnPosition.LAST.equals(e.getPosition())` check in 
`SchemaUtils.transformSchemaChangeEvent` will now throw an NPE.
   
   **Suggestion:** Make this method null-safe (return null, or `checkNotNull` 
with a clear message), or validate `position` in the deprecated constructor. 
Low likelihood, but it is a behavior change.



##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-starrocks/src/main/java/org/apache/flink/cdc/connectors/starrocks/sink/StarRocksMetadataApplier.java:
##########
@@ -356,4 +361,173 @@ private void applyDropTable(DropTableEvent 
dropTableEvent) {
             throw new SchemaEvolveException(dropTableEvent, e.getMessage(), e);
         }
     }
+
+    private void applyReorderColumn(ReorderColumnEvent reorderColumnEvent)

Review Comment:
   **Problem:** MySQL parses `MODIFY COLUMN ... AFTER` into an 
`AlterColumnTypeEvent` followed by this `ReorderColumnEvent` 
(`CustomAlterTableParserListener.java:450-471`). `applyAlterColumnType` (line 
324) calls `StarRocksEnrichedCatalog.alterColumnType`, which submits an 
asynchronous StarRocks schema-change job and returns without waiting 
(`StarRocksEnrichedCatalog.java:121-152`). This method then immediately issues 
the position `ALTER`. StarRocks allows only one schema change per table, so the 
second `ALTER` is rejected with `SCHEMA_CHANGE`, and the job fails 
deterministically. This also happens for a pure position change, because the 
type event is always emitted.
   
   **Suggestion:** Make `alterColumnType` wait for its job like `executeAlter`, 
wait for any in-flight column alter job before the first move, or combine type 
and position into one `ALTER`. Add a test that applies `AlterColumnTypeEvent` 
and then `ReorderColumnEvent`; the current IT only sends the reorder.



##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/main/java/org/apache/flink/cdc/connectors/mysql/source/parser/CustomAlterTableParserListener.java:
##########
@@ -431,6 +451,24 @@ public void 
exitAlterByModifyColumn(MySqlParser.AlterByModifyColumnContext ctx)
                             new AlterColumnTypeEvent(currentTable, 
typeMapping);
                     alterColumnTypeEvent.addColumnComment(columnName, 
column.comment());
                     changes.add(alterColumnTypeEvent);
+                    if (columnName != null && ctx.FIRST() != null) {
+                        changes.add(
+                                new ReorderColumnEvent(
+                                        currentTable,
+                                        Collections.singletonList(
+                                                
ReorderColumnEvent.ColumnMove.first(columnName))));
+                    } else if (columnName != null && ctx.AFTER() != null) {

Review Comment:
   **Problem:** Column identifiers in MySQL are case-insensitive, but 
`parseName` preserves the DDL casing. With `isTableIdCaseInsensitive=false` 
(`lower_case_table_names=0`, the Linux default), `MODIFY COLUMN NAME ... AFTER 
ID`, or a case-only `CHANGE COLUMN name Name ... FIRST`, emits a move for 
`NAME`/`Name` while the tracked schema contains the create-time `name`. 
`SchemaUtils.removeMovedColumn` / `insertColumnRelative` use exact `equals`, so 
the job fails. The new case-normalization test only uses 
`isTableIdCaseInsensitive=true`.
   
   **Suggestion:** Resolve the parsed names against the existing table schema 
case-insensitively and emit the canonical column name. Add a test with 
`isTableIdCaseInsensitive=false` and mixed-case DDL.



##########
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/PostTransformOperator.java:
##########
@@ -282,6 +283,11 @@ private Optional<Event> processSchemaChangeEvent(
 
         if (hasAsteriskMap.getOrDefault(tableId, true)) {
             // See comments in PreTransformOperator#cacheChangeSchema method.
+            if (event instanceof ReorderColumnEvent) {

Review Comment:
   **Problem:** Non-asterisk projections still silently misalign. When 
`hasAsteriskMap` is false, `PreTransformOperator.cacheChangeSchema` calls 
`SchemaUtils.transformSchemaChangeEvent(false, ...)`, which returns 
`Optional.empty()` for `ReorderColumnEvent` (`SchemaUtils.java:379-383`), so 
the event never reaches `PostTransformOperator`. However, 
`cachePreTransformProcessor` (`PreTransformOperator.java:259`) rebuilds the 
pre-transform processor from the reordered source schema, so the rows emitted 
by `PreTransformOperator` change order while this operator pre-schema stays the 
same. Every subsequent record is then decoded positionally with the old schema, 
and values are silently swapped.
   
   **Suggestion:** Handle reorder in the non-asterisk branch as well — for 
example, after rebuilding the processor, derive `oldPreSchema`/`newPreSchema` 
and forward `ReorderColumnEventTransformer.transformReorderColumnEvent(...)`, 
or keep the pre-transform output order stable. Add a test with a non-asterisk 
projection such as `id, age` and a move of `age`; the current test only covers 
`*` (`testReorderWithPreWildcardAndComputedColumn`).



##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-starrocks/src/main/java/org/apache/flink/cdc/connectors/starrocks/sink/StarRocksMetadataApplier.java:
##########
@@ -356,4 +361,173 @@ private void applyDropTable(DropTableEvent 
dropTableEvent) {
             throw new SchemaEvolveException(dropTableEvent, e.getMessage(), e);
         }
     }
+
+    private void applyReorderColumn(ReorderColumnEvent reorderColumnEvent)
+            throws SchemaEvolveException {
+        TableId tableId = reorderColumnEvent.tableId();
+        StarRocksTable currentTable;
+        try {
+            currentTable =
+                    catalog.getTable(tableId.getSchemaName(), 
tableId.getTableName()).orElse(null);
+        } catch (StarRocksCatalogException e) {
+            throw new SchemaEvolveException(reorderColumnEvent, "Failed to 
load table metadata", e);
+        }
+        if (currentTable == null) {
+            throw new SchemaEvolveException(reorderColumnEvent, "Table does 
not exist", null);
+        }
+
+        List<StarRocksColumn> targetColumns =
+                applyColumnMoves(currentTable.getColumns(), 
reorderColumnEvent);
+        validateKeyColumnsRemainFirst(currentTable, targetColumns, 
reorderColumnEvent);
+        if (sameColumnOrder(currentTable, targetColumns)) {
+            return;
+        }
+
+        List<StarRocksColumn> appliedColumns = new 
ArrayList<>(currentTable.getColumns());
+        Set<String> tableKeys =
+                currentTable.getTableKeys().orElseGet(ArrayList::new).stream()
+                        .collect(Collectors.toSet());
+        try {
+            for (int index = 0; index < targetColumns.size(); index++) {

Review Comment:
   **Problem:** This loop turns one column move into up to n-1 separate 
`ALTER`s: moving `a` to the end of `[id,a,b,c,d]` produces `b AFTER id`, `c 
AFTER b`, `d AFTER c` instead of a single `a AFTER d`. Each 
`alterColumnPosition` is a full StarRocks schema change, is waited on 
synchronously with the full `schema-change.timeout`, and opens 3-4 new JDBC 
connections (`StarRocksEnrichedCatalog.java:231-247, 249-281, 303-348`). The 
poll loop also aborts on the first `SQLException` instead of retrying like 
`executeAlter`. For a wide table this can block the metadata path for a very 
long time and leave the table partially reordered.
   
   **Suggestion:**
   - Apply the event moves directly (one `ALTER` per `ColumnMove`), or compute 
a minimal move set from current to target.
   - Reuse one connection for polling, add connect/query timeouts and 
transient-error retries, and use one overall deadline.



##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-starrocks/src/main/java/org/apache/flink/cdc/connectors/starrocks/sink/StarRocksMetadataApplier.java:
##########
@@ -356,4 +361,173 @@ private void applyDropTable(DropTableEvent 
dropTableEvent) {
             throw new SchemaEvolveException(dropTableEvent, e.getMessage(), e);
         }
     }
+
+    private void applyReorderColumn(ReorderColumnEvent reorderColumnEvent)
+            throws SchemaEvolveException {
+        TableId tableId = reorderColumnEvent.tableId();
+        StarRocksTable currentTable;
+        try {
+            currentTable =
+                    catalog.getTable(tableId.getSchemaName(), 
tableId.getTableName()).orElse(null);
+        } catch (StarRocksCatalogException e) {
+            throw new SchemaEvolveException(reorderColumnEvent, "Failed to 
load table metadata", e);
+        }
+        if (currentTable == null) {
+            throw new SchemaEvolveException(reorderColumnEvent, "Table does 
not exist", null);
+        }
+
+        List<StarRocksColumn> targetColumns =
+                applyColumnMoves(currentTable.getColumns(), 
reorderColumnEvent);
+        validateKeyColumnsRemainFirst(currentTable, targetColumns, 
reorderColumnEvent);
+        if (sameColumnOrder(currentTable, targetColumns)) {
+            return;
+        }
+
+        List<StarRocksColumn> appliedColumns = new 
ArrayList<>(currentTable.getColumns());
+        Set<String> tableKeys =
+                currentTable.getTableKeys().orElseGet(ArrayList::new).stream()
+                        .collect(Collectors.toSet());
+        try {
+            for (int index = 0; index < targetColumns.size(); index++) {
+                StarRocksColumn targetColumn = targetColumns.get(index);
+                if (appliedColumns
+                        .get(index)
+                        .getColumnName()
+                        .equals(targetColumn.getColumnName())) {
+                    continue;
+                }
+
+                String precedingColumn =
+                        index == 0 ? null : targetColumns.get(index - 
1).getColumnName();
+                catalog.alterColumnPosition(
+                        tableId.getSchemaName(),
+                        tableId.getTableName(),
+                        targetColumn,
+                        tableKeys.contains(targetColumn.getColumnName()),
+                        precedingColumn,
+                        schemaChangeConfig.getTimeoutSecond());
+                appliedColumns.remove(targetColumn);
+                appliedColumns.add(index, targetColumn);
+            }
+            LOG.info("Successful to apply reorder column, event: {}", 
reorderColumnEvent);
+        } catch (StarRocksCatalogException e) {
+            if (isTargetOrderPresent(tableId, targetColumns)) {
+                LOG.info(
+                        "Successful to apply reorder column after a failed 
request, event: {}",
+                        reorderColumnEvent,
+                        e);
+                return;
+            }
+            throw new SchemaEvolveException(
+                    reorderColumnEvent, "Failed to apply reorder column", e);
+        }
+    }
+
+    private static StarRocksColumn findColumn(List<StarRocksColumn> columns, 
String columnName) {
+        if (columnName == null) {
+            return null;
+        }
+        for (StarRocksColumn column : columns) {
+            if (column.getColumnName().equals(columnName)) {
+                return column;
+            }
+        }
+        return null;
+    }
+
+    private List<StarRocksColumn> applyColumnMoves(
+            List<StarRocksColumn> currentColumns, ReorderColumnEvent 
reorderColumnEvent) {
+        if (reorderColumnEvent.getColumnMoves() == null) {
+            throw new SchemaEvolveException(
+                    reorderColumnEvent, "Column moves must not be null", null);
+        }
+
+        List<StarRocksColumn> targetColumns = new ArrayList<>(currentColumns);
+        for (ReorderColumnEvent.ColumnMove move : 
reorderColumnEvent.getColumnMoves()) {
+            if (move == null || move.getColumnName() == null || 
move.getPosition() == null) {
+                throw new SchemaEvolveException(reorderColumnEvent, "Invalid 
column move", null);
+            }
+            StarRocksColumn movedColumn = findColumn(targetColumns, 
move.getColumnName());
+            if (movedColumn == null) {
+                throw new SchemaEvolveException(
+                        reorderColumnEvent,
+                        "Referenced column does not exist: " + 
move.getColumnName(),
+                        null);
+            }
+            targetColumns.remove(movedColumn);
+
+            int targetIndex;
+            switch (move.getPosition()) {
+                case FIRST:
+                    targetIndex = 0;
+                    break;
+                case LAST:
+                    targetIndex = targetColumns.size();
+                    break;
+                case BEFORE:
+                case AFTER:
+                    StarRocksColumn anchorColumn =
+                            findColumn(targetColumns, 
move.getReferenceColumnName());
+                    if (anchorColumn == null) {
+                        throw new SchemaEvolveException(
+                                reorderColumnEvent,
+                                "Referenced anchor column does not exist: "
+                                        + move.getReferenceColumnName(),
+                                null);
+                    }
+                    targetIndex = targetColumns.indexOf(anchorColumn);
+                    if (move.getPosition() == ColumnPosition.AFTER) {
+                        targetIndex++;
+                    }
+                    break;
+                default:
+                    throw new SchemaEvolveException(
+                            reorderColumnEvent, "Unsupported column position", 
null);
+            }
+            targetColumns.add(targetIndex, movedColumn);
+        }
+        return targetColumns;
+    }
+
+    private void validateKeyColumnsRemainFirst(
+            StarRocksTable table,
+            List<StarRocksColumn> targetColumns,
+            ReorderColumnEvent reorderColumnEvent) {
+        List<String> tableKeys = 
table.getTableKeys().orElseGet(ArrayList::new);
+        for (int index = 0; index < tableKeys.size(); index++) {
+            if 
(!tableKeys.get(index).equals(targetColumns.get(index).getColumnName())) {
+                throw new SchemaEvolveException(

Review Comment:
   **Problem:** Key-order violations (here) and missing columns in 
`applyColumnMoves` (line 438) throw a plain `SchemaEvolveException`. 
`SchemaCoordinator.shouldIgnoreException` only tolerates 
`UnsupportedSchemaChangeEventException` in `TRY_EVOLVE`, so these moves fail 
the job even in tolerant mode. This contradicts the new docs, which say 
`TRY_EVOLVE` keeps the sink schema.
   
   **Suggestion:** Throw `UnsupportedSchemaChangeEventException` (or a 
dedicated tolerated subtype) for unsupported moves. Paimon `checkArgument` 
(`PaimonMetadataApplier.java:561`) and the `SchemaUtils` missing-column path 
(lines 237/253) need the same treatment.



##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-starrocks/src/test/java/org/apache/flink/cdc/connectors/starrocks/sink/StarRocksMetadataApplierTest.java:
##########
@@ -62,13 +72,233 @@ public void setup() {
                                 .put(TABLE_CREATE_NUM_BUCKETS.key(), "10")
                                 
.put("table.create.properties.replication_num", "5")
                                 .build());
-        SchemaChangeConfig schemaChangeConfig = 
SchemaChangeConfig.from(configuration);
-        TableCreateConfig tableCreateConfig = 
TableCreateConfig.from(configuration);
+        this.schemaChangeConfig = SchemaChangeConfig.from(configuration);
+        this.tableCreateConfig = TableCreateConfig.from(configuration);
         this.catalog = new MockStarRocksCatalog();
         this.metadataApplier =
                 new StarRocksMetadataApplier(catalog, tableCreateConfig, 
schemaChangeConfig);
     }
 
+    @Test
+    void testReorderColumnSupportsMovesAndPreservesMetadata() throws Exception 
{

Review Comment:
   **Problem:** This test cannot catch metadata loss: 
`MockStarRocksCatalog.alterColumnPosition` (`MockStarRocksCatalog.java:200`) 
ignores the passed `StarRocksColumn`/`isKey` and just reorders the existing 
column objects, so dropping DEFAULT/nullability/COMMENT in 
`loadColumnDefault`/`buildAlterColumnPositionSql` leaves it green. The IT 
(`StarRocksMetadataApplierITCase.java:737`) also sends only the reorder event 
and has no defaults/comments.
   
   **Suggestion:** Make the mock apply the restated definition (or assert on 
real `information_schema` values), and add an IT with DEFAULT/COMMENT and a 
preceding `AlterColumnTypeEvent`. Also cover defaults like 
`CURRENT_TIMESTAMP(3)`, which `formatDefaultValue` currently quotes as a string.



##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-paimon/src/main/java/org/apache/flink/cdc/connectors/paimon/sink/PaimonMetadataApplier.java:
##########
@@ -502,4 +506,91 @@ private Map<String, String> filterBlobOptions(
     private boolean isSupportedTypeForBlob(DataType dataType) {
         return dataType.isAnyOf(BINARY_STRING, CHARACTER_STRING);
     }
+
+    private void applyReorderColumn(ReorderColumnEvent event) throws 
SchemaEvolveException {
+        try {
+            Identifier identifier =
+                    new Identifier(event.tableId().getSchemaName(), 
event.tableId().getTableName());
+            FileStoreTable table = (FileStoreTable) 
catalog.getTable(identifier);
+            List<String> simulatedColumnNames = new 
ArrayList<>(table.rowType().getFieldNames());
+            List<SchemaChange> tableChangeList = new ArrayList<>();
+            for (ReorderColumnEvent.ColumnMove columnMove : 
event.getColumnMoves()) {
+                SchemaChange.Move move = toPaimonMove(columnMove);
+                if (applyColumnMove(simulatedColumnNames, columnMove)) {
+                    
tableChangeList.add(SchemaChange.updateColumnPosition(move));
+                }
+            }
+            if (!tableChangeList.isEmpty()) {
+                catalog.alterTable(identifier, tableChangeList, true);
+            }
+        } catch (Catalog.TableNotExistException
+                | Catalog.ColumnAlreadyExistException
+                | Catalog.ColumnNotExistException e) {
+            throw new SchemaEvolveException(event, e.getMessage(), e);
+        }
+    }
+
+    private SchemaChange.Move toPaimonMove(ReorderColumnEvent.ColumnMove 
columnMove) {
+        String columnName = columnMove.getColumnName();
+        switch (columnMove.getPosition()) {
+            case FIRST:
+                return SchemaChange.Move.first(columnName);
+            case LAST:
+                return SchemaChange.Move.last(columnName);
+            case BEFORE:
+                return SchemaChange.Move.before(
+                        columnName,
+                        checkNotNull(
+                                columnMove.getReferenceColumnName(),
+                                "Existing column name must be provided for 
BEFORE position"));
+            case AFTER:
+                return SchemaChange.Move.after(
+                        columnName,
+                        checkNotNull(
+                                columnMove.getReferenceColumnName(),
+                                "Existing column name must be provided for 
AFTER position"));
+            default:
+                throw new IllegalArgumentException(
+                        "Unknown column position: " + 
columnMove.getPosition());
+        }
+    }
+
+    private boolean applyColumnMove(
+            List<String> simulatedColumnNames, ReorderColumnEvent.ColumnMove 
columnMove) {
+        List<String> reorderedColumnNames = new 
ArrayList<>(simulatedColumnNames);
+        checkArgument(

Review Comment:
   **Problem:** Move application is now implemented four times with slightly 
different semantics: `SchemaUtils.applyReorderColumnEvent` 
(`SchemaUtils.java:192-259`), this method, 
`StarRocksMetadataApplier.applyColumnMoves` (lines 438-491), and 
`ReorderColumnEventTransformer`. Missing columns here throw 
`IllegalArgumentException`, which `TRY_EVOLVE` does not tolerate.
   
   **Suggestion:** Extract a shared helper (for example 
`applyMoves(List<String>, List<ColumnMove>)` returning the reordered names, or 
a normalized/minimal move list on `ReorderColumnEvent`) so no-op handling and 
missing-column policy stay consistent. Use 
`UnsupportedSchemaChangeEventException` for unsupported moves.



##########
docs/content/docs/core-concept/schema-evolution.md:
##########
@@ -146,6 +147,12 @@ The following YAML configuration is set to include 
`CreateTableEvent` and column
 
 ```yaml
 sink:
-  include.schema.changes: [create.table, column] # This matches CreateTable, 
AddColumn, AlterColumnType, RenameColumn, and DropColumn Events
+  include.schema.changes: [create.table, column] # This matches CreateTable, 
AddColumn, AlterColumnType, RenameColumn, ReorderColumn, and DropColumn Events
   exclude.schema.changes: [drop.column] # This excludes DropColumn Events
 ```
+
+### Column order changes
+
+The MySQL pipeline source emits `ReorderColumnEvent` for `MODIFY COLUMN` and 
`CHANGE COLUMN` statements with `FIRST` or `AFTER` clauses. Renaming, type 
changes, and column moves in a compound DDL are applied in statement order. The 
`reorder` and `column` families both include `reorder.column`.
+
+Kafka updates its serialization schema. Paimon applies physical column moves, 
while StarRocks only accepts moves that keep primary key columns first and in 
their existing order. Doris, Fluss, Iceberg, MaxCompute, and OceanBase reject 
physical column reordering explicitly. In `EVOLVE` mode, an unsupported move 
fails the job rather than silently misaligning field values. In `TRY_EVOLVE` 
mode, the framework retains the sink schema and maps subsequent records to that 
schema. `LENIENT` mode does not reorder the sink columns; record values are 
mapped to the unchanged column order. Explicitly excluding `reorder.column` 
also keeps the sink column order unchanged.

Review Comment:
   **Problem:** There are three documentation gaps:
   1. Hudi is missing from the list of sinks that reject reorder explicitly; 
`HudiMetadataApplier` currently fails `TRY_EVOLVE` as described in the review 
comment on `SchemaChangeEventVisitor`.
   2. The `TRY_EVOLVE` sentence is only true for sinks that throw 
`UnsupportedSchemaChangeEventException`, but StarRocks validation, Paimon 
missing columns and `SchemaUtils` currently throw other exceptions. Either fix 
those or qualify the sentence.
   3. The StarRocks connector page 
(`docs/content/docs/connectors/pipeline-connectors/starrocks.md:257-261`) still 
says only add/drop/rename/alter-type and "new column always last". Document 
reorder support, the key-order constraint, and the one-schema-change-at-a-time 
behavior.
   
   **Suggestion:** Also consider an upgrade note: the default accepted set is 
`ALL` (`SinkDef.java:56-57`), so existing EVOLVE pipelines will now fail or 
physically reorder on the next `MODIFY ... AFTER`.



##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-starrocks/src/main/java/org/apache/flink/cdc/connectors/starrocks/sink/StarRocksMetadataApplier.java:
##########
@@ -356,4 +361,173 @@ private void applyDropTable(DropTableEvent 
dropTableEvent) {
             throw new SchemaEvolveException(dropTableEvent, e.getMessage(), e);
         }
     }
+
+    private void applyReorderColumn(ReorderColumnEvent reorderColumnEvent)
+            throws SchemaEvolveException {
+        TableId tableId = reorderColumnEvent.tableId();
+        StarRocksTable currentTable;
+        try {
+            currentTable =

Review Comment:
   **Problem:** `catalog.getTable()` reads `information_schema.COLUMNS` without 
`ORDER BY ORDINAL_POSITION`. `StarRocksTable` is documented as sorted by 
ordinal position, but the builder does not sort it. The entire logic here 
(`applyColumnMoves`, `sameColumnOrder`, `precedingColumn`) and the `tableKeys` 
order depend on the list being in physical order.
   
   **Suggestion:** Sort the columns by `getOrdinalPosition()` (and derive 
`tableKeys` accordingly) before using them, or add an ordered query/override in 
`StarRocksEnrichedCatalog`.



##########
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/ReorderColumnEventTransformer.java:
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.cdc.runtime.operators.transform;
+
+import org.apache.flink.cdc.common.annotation.Internal;
+import org.apache.flink.cdc.common.event.ReorderColumnEvent;
+import org.apache.flink.cdc.common.schema.Schema;
+import org.apache.flink.cdc.common.utils.Preconditions;
+
+import javax.annotation.CheckReturnValue;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+/** Transforms source reorder events into moves that apply to post-transformed 
schemas. */
+@Internal
+public final class ReorderColumnEventTransformer {
+
+    private ReorderColumnEventTransformer() {}
+
+    @CheckReturnValue
+    public static Optional<ReorderColumnEvent> transformReorderColumnEvent(
+            ReorderColumnEvent event, Schema schemaBeforeChange, Schema 
schemaAfterChange) {
+        List<String> oldColumnNames = schemaBeforeChange.getColumnNames();
+        List<String> newColumnNames = schemaAfterChange.getColumnNames();
+        if (oldColumnNames.equals(newColumnNames)) {
+            return Optional.empty();
+        }
+
+        Preconditions.checkArgument(
+                oldColumnNames.size() == newColumnNames.size()
+                        && new HashSet<>(oldColumnNames).equals(new 
HashSet<>(newColumnNames)),
+                "Unable to transform ReorderColumnEvent %s between schemas 
with different columns: %s and %s",
+                event,
+                oldColumnNames,
+                newColumnNames);
+
+        List<String> stableAnchors = longestCommonSubsequence(oldColumnNames, 
newColumnNames);
+        Map<String, Integer> newColumnPositions = new HashMap<>();
+        for (int i = 0; i < newColumnNames.size(); i++) {
+            newColumnPositions.put(newColumnNames.get(i), i);
+        }
+        List<ReorderColumnEvent.ColumnMove> columnMoves = new ArrayList<>();
+        if (stableAnchors.isEmpty()) {

Review Comment:
   **Problem:** This branch is unreachable: two non-empty permutations of the 
same column set always have an LCS of at least one element, and equal lists 
return at line 45. It has no test coverage.
   
   **Suggestion:** Remove it. The `targetIndex = 0` initialization in the 
following loop already handles the first-anchor case.



##########
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/ReorderColumnEventTransformer.java:
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.cdc.runtime.operators.transform;
+
+import org.apache.flink.cdc.common.annotation.Internal;
+import org.apache.flink.cdc.common.event.ReorderColumnEvent;
+import org.apache.flink.cdc.common.schema.Schema;
+import org.apache.flink.cdc.common.utils.Preconditions;
+
+import javax.annotation.CheckReturnValue;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+/** Transforms source reorder events into moves that apply to post-transformed 
schemas. */
+@Internal
+public final class ReorderColumnEventTransformer {
+
+    private ReorderColumnEventTransformer() {}
+
+    @CheckReturnValue
+    public static Optional<ReorderColumnEvent> transformReorderColumnEvent(
+            ReorderColumnEvent event, Schema schemaBeforeChange, Schema 
schemaAfterChange) {
+        List<String> oldColumnNames = schemaBeforeChange.getColumnNames();
+        List<String> newColumnNames = schemaAfterChange.getColumnNames();
+        if (oldColumnNames.equals(newColumnNames)) {
+            return Optional.empty();
+        }
+
+        Preconditions.checkArgument(
+                oldColumnNames.size() == newColumnNames.size()
+                        && new HashSet<>(oldColumnNames).equals(new 
HashSet<>(newColumnNames)),
+                "Unable to transform ReorderColumnEvent %s between schemas 
with different columns: %s and %s",
+                event,
+                oldColumnNames,
+                newColumnNames);
+
+        List<String> stableAnchors = longestCommonSubsequence(oldColumnNames, 
newColumnNames);
+        Map<String, Integer> newColumnPositions = new HashMap<>();
+        for (int i = 0; i < newColumnNames.size(); i++) {
+            newColumnPositions.put(newColumnNames.get(i), i);
+        }
+        List<ReorderColumnEvent.ColumnMove> columnMoves = new ArrayList<>();
+        if (stableAnchors.isEmpty()) {
+            for (int i = newColumnNames.size() - 1; i >= 0; i--) {
+                
columnMoves.add(ReorderColumnEvent.ColumnMove.first(newColumnNames.get(i)));
+            }
+        } else {
+            int targetIndex = 0;
+            for (int anchorIndex = 0; anchorIndex < stableAnchors.size(); 
anchorIndex++) {
+                String anchor = stableAnchors.get(anchorIndex);
+                int anchorTargetIndex = newColumnPositions.get(anchor);
+                if (anchorIndex == 0) {
+                    for (int i = anchorTargetIndex - 1; i >= targetIndex; i--) 
{
+                        
columnMoves.add(ReorderColumnEvent.ColumnMove.first(newColumnNames.get(i)));
+                    }
+                } else {
+                    for (int i = targetIndex; i < anchorTargetIndex; i++) {
+                        columnMoves.add(
+                                ReorderColumnEvent.ColumnMove.before(
+                                        newColumnNames.get(i), anchor));
+                    }
+                }
+                targetIndex = anchorTargetIndex + 1;
+            }
+            for (int i = targetIndex; i < newColumnNames.size(); i++) {
+                
columnMoves.add(ReorderColumnEvent.ColumnMove.last(newColumnNames.get(i)));
+            }
+        }
+
+        return Optional.of(new ReorderColumnEvent(event.tableId(), 
columnMoves));
+    }
+
+    private static List<String> longestCommonSubsequence(
+            List<String> oldColumnNames, List<String> newColumnNames) {
+        int[][] lengths = new int[oldColumnNames.size() + 
1][newColumnNames.size() + 1];

Review Comment:
   **Problem:** This allocates an `(n+1) x (m+1)` `int` matrix per transform 
instance per reorder event. For a 10,000-column table that is about 400 MB and 
about 100M comparisons; 1,000 columns is about 4 MB per instance.
   
   **Suggestion:** Since the two schemas are permutations of the same column 
set, compute the LIS over the old-index to new-index permutation in `O(n log 
n)`, or derive moves directly from the event. At minimum, guard the matrix size.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to