pvary commented on issue #6630: URL: https://github.com/apache/iceberg/issues/6630#issuecomment-1610804650
The Flink snapshot/checpoint state is kept in 3 places: - Flink internal state - list of temp manifest files - File system - temp manifest files and uncomitted data files - Iceberg table data/metadata These 3 needs to be in sync, and we need to keep the changes since the last sync. So if we do not commit to Iceberg, then the Flink internal state and the file system temp tables are keep growing. To avoid this, we commit from time to time (and write Flink metadata to the Iceberg table in the process), and after this commit we are able to remove old temp files and clean some data from the Flink state. So writing empty commit is intentional/needed, but the failure seems like a bug. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
