elkhand commented on issue #2033:
URL: https://github.com/apache/iceberg/issues/2033#issuecomment-765864038
The root cause of the issue is related to this part of the code in
`IcebergFilesCommitter` class:
```
@Override
public void notifyCheckpointComplete(long checkpointId) throws Exception {
super.notifyCheckpointComplete(checkpointId);
// It's possible that we have the following events:
// 1. snapshotState(ckpId);
// 2. snapshotState(ckpId+1);
// 3. notifyCheckpointComplete(ckpId+1);
// 4. notifyCheckpointComplete(ckpId);
// For step#4, we don't need to commit iceberg table again because in
step#3 we've committed all the files,
// Besides, we need to maintain the max-committed-checkpoint-id to be
increasing.
if (checkpointId > maxCommittedCheckpointId) {
commitUpToCheckpoint(dataFilesPerCheckpoint, flinkJobId, checkpointId);
this.maxCommittedCheckpointId = checkpointId;
}
}
```
When **savepoint** is created, in `metadata.json` file
`flink.max-committed-checkpoint-id` is set to the very big number
`9223372036854775807`:
```
...
{
"snapshot-id" : 1466502797494274716,
"parent-snapshot-id" : 7039191958488319023,
"timestamp-ms" : 1611375333522,
"summary" : {
...
"flink.max-committed-checkpoint-id" : "9223372036854775807",
...
...
```
This condition `if (checkpointId > maxCommittedCheckpointId)` always
evaluates to fase, thus the transaction is not committed., thus Iceberg
specific files are not created.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
