Xiaobing Fang created FLINK-40982:
-------------------------------------
Summary: Fluss pipeline sink fails to recover when an ADD COLUMN
event is replayed
Key: FLINK-40982
URL: https://issues.apache.org/jira/browse/FLINK-40982
Project: Flink
Issue Type: Bug
Components: Flink CDC
Reporter: Xiaobing Fang
h3. Problem
Schema changes applied to Fluss are not rolled back when a Flink job restores
from a checkpoint. If an ADD COLUMN succeeds in Fluss but the job fails before
the next checkpoint completes, recovery can replay the same AddColumnEvent.
The Fluss sink submits the addition again. Fluss rejects the existing column,
causing schema evolution to fail and preventing recovery.
h3. Example scenario
Consider a CDC pipeline synchronizing an orders table to Fluss, with
checkpointing and ADD COLUMN schema evolution enabled.
# Checkpoint C1 completes. Both the source and target tables contain only id
and amount.
# The source table adds a nullable remark column, for example: ALTER TABLE
orders ADD COLUMN remark VARCHAR(255).
# Flink CDC processes the AddColumnEvent and successfully adds remark to the
Fluss target.
# The job fails before a checkpoint containing this schema change completes.
# The job restores from C1. Its restored schema state does not contain remark,
but the Fluss target still has the column.
# When the source re-emits the schema change, the Fluss sink attempts to add
remark again and receives:
{noformat}
org.apache.fluss.exception.InvalidAlterTableException: Column remark already
exists.
{noformat}
Restarting from the same checkpoint can encounter the same failure again. No
manual modification of the target table is required.
h3. Expected behavior
Replaying an ADD COLUMN event should succeed when the requested columns already
exist with matching Fluss types.
If only some requested columns exist, add the missing ones. Conflicting column
definitions and unrelated errors should still fail.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
