zhangshenghang opened a new issue, #13322: URL: https://github.com/apache/seatunnel/issues/13322
Related to #12677 (which tracks retention reaching ~2x max-retained within a single run). This issue tracks two adjacent behaviors not covered there. ### What happened `completedCheckpointIds` is not reset when the coordinator is restored, and ids are recorded before storage succeeds (dev @ 89f91a89af): 1. **Not cleared across restores**: `cleanPendingCheckpoint` (`CheckpointCoordinator.java:1246-1302`) resets `pendingCheckpoints`, `pipelineTaskStatus`, ready-to-close sets, `pendingCounter` and `schemaChanging` — but not `completedCheckpointIds`. After a pipeline restore, ids from the old generation stay in the queue, so the physical storage can hold up to ~2x `max-retained` across the restore boundary, beyond what the user configured. 2. **Enqueued before stored**: `completePendingCheckpoint` appends the id at `CheckpointCoordinator.java:1410` and only then runs `storeCheckPoint` in the try block. On a storage failure the exception propagates (sneakyThrow at 1439-1442) but **the id stays in the queue**; a later rotation (1422-1437) will call `deleteCheckpoint` with an id that was never persisted (LocalFileStorage logs errors for the missing file), and the orphaned file (if partially written) is never tracked for deletion again. ### Impact Storage usage exceeding the configured retention after restores; retention deletions targeting non-existent ids; untracked orphan files after store failures. ### Suggested fix Clear `completedCheckpointIds` in `cleanPendingCheckpoint(CHECKPOINT_COORDINATOR_RESET)` (or otherwise reset it per generation), and only enqueue an id after `storeCheckPoint` succeeds (remove it on failure). ### SeaTunnel Version dev (89f91a89af), Zeta engine -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
