[
https://issues.apache.org/jira/browse/FLINK-40741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruan Hang updated FLINK-40741:
------------------------------
Description:
The table A had its schema modified last night, after which the original job
started throwing errors. When this table was newly added to the CDC job, no
newly-added (incremental) data was being synchronized for it — the full
snapshot data appeared to sync normally. Looking at the TaskManager logs, all
other tables were syncing normally, but this table indeed had no sink
synchronization, leading to the suspicion that even the full snapshot data had
not been synchronized.
Root Cause:
1. When the user reads the full snapshot data of a newly added table, each
snapshot split reports back once it finishes reading.
2. Once all snapshot splits have finished reading, the enumerator
transitions into the NEWLY_ADDED_ASSIGNING_SNAPSHOT_FINISHED state.
3. The enumerator then periodically checks whether it is in the
NEWLY_ADDED_ASSIGNING_SNAPSHOT_FINISHED state. If it is:
- a. It sends a BinlogSplitUpdateRequest to make the binlog reader pause
reading.
- b. Through a LatestFinishedSplitsNumberEvent, it tells the binlog reader
that there is a newly added table and updates the latest finished-splits number
for that table; only after this can the newly added
table's LatestFinishedSplitsNumber be read.
- d. It transitions into NEWLY_ADDED_ASSIGNING_FINISHED, indicating that
the incremental phase is complete.
- e. The reader then re-reads the binlog of the new table.
However, the job experienced a failover at step 4, when it transitioned into
the NEWLY_ADDED_ASSIGNING_FINISHED state. The job's strategy is region
failover, so the enumerator is not restarted; instead,
SplitEnumerator#addSplitsBack is called. At this point the state remains
NEWLY_ADDED_ASSIGNING_FINISHED, while the reader, after restarting, rolls back
to the state at step 2.
Then, at step 3, the check finds that the state is not
NEWLY_ADDED_ASSIGNING_SNAPSHOT_FINISHED, so the binlog is not updated. As a
result, the incremental data of the new table is never read.
> [Flink cdc]After the full snapshot read of the newly added table completes,
> the newly added table is no longer synchronized, while the existing (old)
> tables continue reading
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-40741
> URL: https://issues.apache.org/jira/browse/FLINK-40741
> Project: Flink
> Issue Type: Bug
> Components: Flink CDC
> Reporter: Ruan Hang
> Priority: Major
>
> The table A had its schema modified last night, after which the original job
> started throwing errors. When this table was newly added to the CDC job, no
> newly-added (incremental) data was being synchronized for it — the full
> snapshot data appeared to sync normally. Looking at the TaskManager logs, all
> other tables were syncing normally, but this table indeed had no sink
> synchronization, leading to the suspicion that even the full snapshot data
> had not been synchronized.
> Root Cause:
> 1. When the user reads the full snapshot data of a newly added table, each
> snapshot split reports back once it finishes reading.
> 2. Once all snapshot splits have finished reading, the enumerator
> transitions into the NEWLY_ADDED_ASSIGNING_SNAPSHOT_FINISHED state.
> 3. The enumerator then periodically checks whether it is in the
> NEWLY_ADDED_ASSIGNING_SNAPSHOT_FINISHED state. If it is:
> - a. It sends a BinlogSplitUpdateRequest to make the binlog reader pause
> reading.
> - b. Through a LatestFinishedSplitsNumberEvent, it tells the binlog
> reader that there is a newly added table and updates the latest
> finished-splits number for that table; only after this can the newly added
> table's LatestFinishedSplitsNumber be read.
> - d. It transitions into NEWLY_ADDED_ASSIGNING_FINISHED, indicating that
> the incremental phase is complete.
> - e. The reader then re-reads the binlog of the new table.
> However, the job experienced a failover at step 4, when it transitioned
> into the NEWLY_ADDED_ASSIGNING_FINISHED state. The job's strategy is region
> failover, so the enumerator is not restarted; instead,
> SplitEnumerator#addSplitsBack is called. At this point the state remains
> NEWLY_ADDED_ASSIGNING_FINISHED, while the reader, after restarting, rolls
> back to the state at step 2.
> Then, at step 3, the check finds that the state is not
> NEWLY_ADDED_ASSIGNING_SNAPSHOT_FINISHED, so the binlog is not updated. As a
> result, the incremental data of the new table is never read.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
