zhongyujiang edited a comment on pull request #3365:
URL: https://github.com/apache/iceberg/pull/3365#issuecomment-1020781584
Hi @openinx, I am still seeing this error in CI sometimes, I think this is
maybe caused by the way `notifyCheckpointComplete` works in
IcebergFilesCommitter:
```
public void notifyCheckpointComplete(long checkpointId) throws Exception {
super.notifyCheckpointComplete(checkpointId);
// It's possible that we have the following events:
// 1. snapshotState(ckpId);
// 2. snapshotState(ckpId+1);
// 3. notifyCheckpointComplete(ckpId+1);
// 4. notifyCheckpointComplete(ckpId);
// For step#4, we don't need to commit iceberg table again because in
step#3 we've committed all the files,
// Besides, we need to maintain the max-committed-checkpoint-id to be
increasing.
if (checkpointId > maxCommittedCheckpointId) {
commitUpToCheckpoint(dataFilesPerCheckpoint, flinkJobId, checkpointId);
this.maxCommittedCheckpointId = checkpointId;
}
}
```
My understanding is that not every ckpt of Flink corresponds to a commit in
Iceberg, sometimes a snapshot may contains data files of several ckpts. And the
checkpoint interval (400 ms) here is rather small, which makes this situation
very likely. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
