huangjc7 opened a new issue, #4290:
URL: https://github.com/apache/amoro/issues/4290
### What happened?
Hi Amoro team,
We observed possible CDC snapshot loss when Flink CDC and Amoro Optimizer
commit to the same Iceberg table at nearly the same time.
We have confirmed the data difference and missing snapshots, but we are not
yet certain about the exact root cause. We would like to provide the evidence
below and ask the Amoro team to help confirm the commit behavior.
### Environment
- Amoro: 0.8.1-incubating
- Flink: 1.20.3
- Flink CDC: 3.6.0
- Flink Iceberg runtime: 1.10.1
- Amoro Optimizer: Spark 3.3.2 / Iceberg 1.6.1
- Catalog: Amoro Iceberg REST Catalog
- Iceberg V2 table with upsert enabled
- Checkpoint interval: 60 seconds
- Table volume: about 4 million rows per day
- Self-optimizing: enabled
The issue is easier to observe on this high-throughput table because CDC and
Optimizer both commit frequently.
### What we observed
For one day:
- MySQL rows: 3,771,955
- Iceberg rows: 3,757,759
- Difference: 14,196 rows
We found five CDC commits whose snapshot IDs are not present in the current
Iceberg snapshot metadata:
| Time | CDC snapshot ID | Added records |
|---|---:|---:|
| 15:47 | 8495459209124852563 | 140 |
| 19:36 | 3149465639900291738 | 849 |
| 21:57 | 5182638233143834397 | 97 |
| 22:01 | 954105368916944994 | 773 |
| 22:10 | 5462846769141107542 | 12,337 |
The added record count of these five commits totals 14,196, which happens to
match the MySQL/Iceberg row-count difference.
### Example
For checkpoint 4693, Flink logged two committables containing 12,337 records:
2026-07-23 22:10:14 IcebergCommitter -
CheckpointId: 4693, AddCount: 6060
2026-07-23 22:10:14 IcebergCommitter -
CheckpointId: 4693, AddCount: 6277
It then logged:
2026-07-23 22:10:16 SnapshotProducer -
Committed snapshot 5462846769141107542
2026-07-23 22:10:16 MergingSnapshotProducer -
Failed to load committed snapshot: omitting sequence number from
notifications
The commit report contained:
attempts=1
addedRecords=12337
At approximately the same time, an Amoro OPTIMIZE snapshot was committed:
committed_at = 2026-07-23 22:10:15.411
snapshot_id = 6115375457875037679
operation = replace
snapshot.producer = OPTIMIZE
The next visible CDC snapshot uses this OPTIMIZE snapshot as its parent.
Snapshot 5462846769141107542 cannot be found in collector$snapshots, and time
travel reports that the snapshot ID does not exist.
Flink still completed the checkpoint, so these records were not replayed
later.
### Possible investigation direction
While reviewing the Amoro 0.8.1 source code, we noticed that
IcebergInternalTableOperations.commit() catches commit exceptions without
rethrowing them:
https://github.com/apache/amoro/blob/v0.8.1-incubating/amoro-ams/src/main/java/org/apache/amoro/server/table/internal/IcebergInternalTableOperations.java#L102-L111
try {
commitTableInternal(tableMetadata, base, metadata,
newMetadataFileLocation);
org.apache.amoro.server.table.TableMetadata updatedMetadata =
doCommit();
checkCommitSuccess(updatedMetadata, newMetadataFileLocation);
} catch (Exception e) {
io.deleteFile(newMetadataFileLocation);
} finally {
this.tableMetadata = null;
}
refresh();
We are not sure whether this code path was triggered in our case because the
original exception is not logged.
However, doCommit() and checkCommitSuccess() may throw CommitFailedException
during concurrent commits. If that exception is swallowed, could the REST
client receive a successful response instead of HTTP 409?
This might explain why Flink reports attempts=1 and considers the checkpoint
successful even though the snapshot is not visible afterward.
### Questions
1. Is this behavior expected when REST Catalog and Optimizer commit
concurrently?
2. Is it possible for a commit conflict to be caught here without being
returned to the REST client?
3. Should CommitFailedException be rethrown so the REST layer can return
HTTP 409?
4. Are there recommended Amoro or Iceberg retry settings for this scenario?
5. Could the Iceberg version difference between Flink and Amoro Optimizer
affect this behavior?
6. Is there an existing fix or newer version we should test?
### Affects Versions
Amoro: 0.8.1-incubating
### What table formats are you seeing the problem on?
Iceberg
### What engines are you seeing the problem on?
Optimizer, Flink, Spark
### How to reproduce
_No response_
### Relevant log output
```shell
```
### Anything else
_No response_
### Are you willing to submit a PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
