johnclara edited a comment on pull request #1739:
URL: https://github.com/apache/iceberg/pull/1739#issuecomment-723561556
@openinx did you ever run into this issue in the flink committer with a
"failed" commit which actually succeeded?
Looking at:
```
// A sorted map to maintain the completed data files for each pending
checkpointId (which have not been committed
// to iceberg table). We need a sorted map here because there's possible
that few checkpoints snapshot failed, for
// example: the 1st checkpoint have 2 data files <1, <file0, file1>>, the
2st checkpoint have 1 data files
// <2, <file3>>. Snapshot for checkpoint#1 interrupted because of
network/disk failure etc, while we don't expect
// any data loss in iceberg table. So we keep the finished files <1,
<file0, file1>> in memory and retry to commit
// iceberg table when the next checkpoint happen.
private final NavigableMap<Long, byte[]> dataFilesPerCheckpoint =
Maps.newTreeMap();
```
Nevermind, it looks like you skip already committed files during initialize
to avoid this. I need to learn more about Two Phase Commit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
