stevenzwu commented on issue #2033: URL: https://github.com/apache/iceberg/issues/2033#issuecomment-767156860
I think one of the problem is how `IcebergFilesCommitter` use `maxCommittedCheckpointId` to de-dup committed manifests upon recovery. That is why it checkpoint and restore the checkpointId, which was impacted by the `Long.MAX_VALUE` with `endInput()`. In our internal implementation, we are storing a hash value for each of the committing manifest file path. Upon recovery, we use that metadata in snapshot summary to filter out committed manifests. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
