stevenzwu edited a comment on issue #2808: URL: https://github.com/apache/iceberg/issues/2808#issuecomment-893979874
This is an interesting scenario. IcebergFileCommitter [does the check upon restore/restart](https://github.com/apache/iceberg/blob/master/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java#L148-L155). Maybe in this case, job didn't restart? Checkpoint should fail. But checkpoint failure doesn't necessarily cause job restart. Did you set this [Flink config](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/config/#execution-checkpointing-tolerable-failed-checkpoints)? In the no-restart scenario, the next commit will commit a new set of files along with the old set of files (due to CommitStateUnknownException). Maybe when the next checkpoint and commit came, there was no new files. That could explain the scenario. This is one possible scenario that I can think of. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
