[
https://issues.apache.org/jira/browse/FLINK-38450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18070469#comment-18070469
]
ChaoFang commented on FLINK-38450:
----------------------------------
In fact, the branch we use internally has solved this problem.
However, due to the confusion of code logic (achieving the desired effect), it
has not been submitted to the community.
Maybe when I have time, I will push a PR after optimizing logic with AI AGENT
recently。
[~spoorthibasu] Welcome PR, I will verify it together
> flink-cdc sink to iceberg table record duplication
> --------------------------------------------------
>
> Key: FLINK-38450
> URL: https://issues.apache.org/jira/browse/FLINK-38450
> Project: Flink
> Issue Type: Bug
> Components: Flink CDC
> Affects Versions: 3.0.0
> Reporter: ChaoFang
> Priority: Major
> Labels: Flink-CDC, iceberg
> Fix For: cdc-3.5.0
>
>
> When multiple modifications to the same PrimaryId occur within a single
> checkpoint and are flushed separately, records are written to different file
> batches (data/eqDelete/posDelete files) sharing the same seq_number. During
> reads, the eqDeleteFile fails to correctly filter stale data, resulting in
> duplicate records.
> *Repro Steps* (in 1 checkpoint):
> # Got Rcoed(id=1,name="a") insert event.
> # Got flush event (Triggered from some other change).
> # Rcoed(id=1,name="2") flush to w1-0001.parquet (with it deleteFiles).
> # Got Rcoed(id=1,name="b") update event.
> # New writer.
> # Rcoed(id=1,name="b") flush to w2-0001.parquet (with it deleteFiles).
> # Got checkpoint commit event.
> # commit w1-0001.parquet, w2-0001.parquet (with their deleteFiles) metadata
> to snapshot.
> Since it is a new writer, the distribution of data records cannot be correct
> (for example, the distribution of data records to different files by Hash).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
