thexiay commented on PR #8367: URL: https://github.com/apache/inlong/pull/8367#issuecomment-1612725984
Problem scenario: 1. Assume that the current chk is n, the iceberg snapshot corresponding to n has been submitted, and the max-committed-chk-id is n 2. Manually roll back the task to the point where chk is m (m<n) 3. Before the checkpointId catches up to max-committed-chk-id, the new snapshot will not be submitted, and all uncommited manifests are cached in the state 4. At this time, once a runtime exception occurs, the task restarts and restores the uncommited manifest from the state, but because it will be compared with the max-committed-chk-id, only the max-committed-chk-id will be submitted, and all uncommited manifests will be submitted. is cropped, and the data is lost. For details, please refer to the test case TestRollbackAndRecover#testRecoveryFromSnapshotWithoutCompletedNotification Concept introduction: restore checkpoint id: checkpoint id corresponding to restore max-committed-checkpoint-id: checkpoint-id information in the summary in the most recent snapshot in iceberg snapshot uncommitted manifest: uncommitted manifest, saved in the state Solutions: 1. Try to rollback iceberg table during restore. Keep the source and destination table data consistent. The following is the idea of rollback: 1.1. max-checkpoint-id <= restore checkpoint id: It means that the snapshots that have been submitted have not yet reached the restore checkpoint id, so there is no need for rollback at all, just make up the submission 1.2. max-checkpoint-id > restore checkpoint id: indicates that the checkpoint id corresponding to the submitted snapshot has exceeded the restore checkpoint id, and a rollback is required If the rollback fails (usually the snapshot is expired, resulting in the failure to obtain the manifest). At this time, it is not stupid to submit all the manifests, but to submit them selectively. At this point try to find the earliest uncommitted manifest The following is the algorithm to find the latest uncommitted manifest: 2.1. Sort by chkID, search from back to front, if the manifest file corresponding to chkID is cleaned up, it means it has been submitted. The manifest file is only cleaned up after the commit is complete. Then at this time its last chkID is the earliest uncommitted manifest 2.2 The above-mentioned supplementary situation: Of course, there may be situations: the manifest has been submitted, but the cleaning failed, and the manifest file remains. Only in this case will it lead to duplicate data. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
