[
https://issues.apache.org/jira/browse/FLINK-40982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-40982:
-----------------------------------
Labels: pull-request-available (was: )
> Fluss pipeline sink fails to recover when an ADD COLUMN event is replayed
> -------------------------------------------------------------------------
>
> Key: FLINK-40982
> URL: https://issues.apache.org/jira/browse/FLINK-40982
> Project: Flink
> Issue Type: Bug
> Components: Flink CDC
> Reporter: Xiaobing Fang
> Priority: Major
> Labels: pull-request-available
>
> h3. Problem
> Schema changes applied to Fluss are not rolled back when a Flink job restores
> from a checkpoint. If an ADD COLUMN succeeds in Fluss but the job fails
> before the next checkpoint completes, recovery can replay the same
> AddColumnEvent.
> The Fluss sink submits the addition again. Fluss rejects the existing column,
> causing schema evolution to fail and preventing recovery.
> h3. Example scenario
> Consider a CDC pipeline synchronizing an orders table to Fluss, with
> checkpointing and ADD COLUMN schema evolution enabled.
> # Checkpoint C1 completes. Both the source and target tables contain only id
> and amount.
> # The source table adds a nullable remark column, for example: ALTER TABLE
> orders ADD COLUMN remark VARCHAR(255).
> # Flink CDC processes the AddColumnEvent and successfully adds remark to the
> Fluss target.
> # The job fails before a checkpoint containing this schema change completes.
> # The job restores from C1. Its restored schema state does not contain
> remark, but the Fluss target still has the column.
> # When the source re-emits the schema change, the Fluss sink attempts to add
> remark again and receives:
> {noformat}
> org.apache.fluss.exception.InvalidAlterTableException: Column remark already
> exists.
> {noformat}
> Restarting from the same checkpoint can encounter the same failure again. No
> manual modification of the target table is required.
> h3. Expected behavior
> Replaying an ADD COLUMN event should succeed when the requested columns
> already exist with matching Fluss types.
> If only some requested columns exist, add the missing ones. Conflicting
> column definitions and unrelated errors should still fail.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
