DanielLeens commented on PR #12096: URL: https://github.com/apache/seatunnel/pull/12096#issuecomment-6607597653
Thanks @SEZ9, agreed on the priorities, with F1 first. Here is the shape I would like to settle before writing code, so you can push back early. **1. ID-only enumeration (F5).** `restoreCompletedCheckpointRetentionState` (`CheckpointCoordinator.java:1682-1708` on `aa30f22e`) only uses `PipelineState::getCheckpointId` from what it loads, yet `getCheckpointsByJobIdAndPipelineId` reads and deserializes every file. Both in-repo plugins (local-file and HDFS) already encode the pipeline id and checkpoint id in the file name (`AbstractCheckpointStorage#getCheckPointName`, parsed by `getPipelineIdByFileName` / `getCheckpointIdByFileName`), so the IDs can come from a directory listing without opening any payload. Proposal: a new `default` method on the `CheckpointStorage` SPI that returns the retention-eligible checkpoint IDs for a job/pipeline, overridden in both in-repo plugins via file-name parsing. The `default` returns an empty list, so an out-of-tree plugin that does not implement it keeps today's `dev` behavior (no restart-time pruning, no payload read) instead of breaking compilation or paying for deserialization. **2. Savepoint signal (F2 / F4).** The type only lives inside the serialized `CompletedCheckpoint`, so the listing needs a marker that does not require reading payloads. My preference is a type segment in the file name for every file written from now on, with a fail-safe restart scan: only files positively identified as regular checkpoints are eligible. Savepoints, and any file the scan cannot classify (which includes everything written before the upgrade), are never queued by the restart path. Pre-upgrade checkpoints are then not pruned after a restart, which is exactly `dev` behavior today, so nothing regresses and this path can never delete a user savepoint. The detail I would like your view on is the name format: it must stay parseable by the existing `getCheckpointIdByFileName` (segment 3, split on `.`), so that an older node reading a newer file during a rolling upgrade still resolves the right ID. I would therefore append the marker as a trailing segment after the checkpoin t id and leave every existing segment unchanged. The alternative, a sidecar metadata record, avoids a name change but adds a second write per checkpoint and a consistency problem, so I lean toward the name. **3. Independent of the storage change:** - **F1:** exclude the restore checkpoint ID from the cleanup candidates, with a test that restores an older same-job checkpoint while `max-retained` is already exceeded. - **F3:** per-ID progress with a bounded retry, with a test for a permanently undeletable file followed by deletable ones. - **F7:** catch and log any runtime failure in the constructor-time scan and cleanup (today only `CheckpointStorageException` at `:1688` and `:1713`), so coordinator recreation during failover cannot be aborted by retention. **4. F6 / F8:** en/zh docs for `checkpoint.storage.max-retained` (exactly `max-retained` kept, one `deleteCheckpoint` call per completed checkpoint, restart-time pruning of identifiable checkpoints only), plus a compatibility note that also tells storage-plugin implementers about the new `default` SPI method. I plan to push all of this as one revision once you are comfortable with point 2. Nothing is implemented yet, so the current head stays not ready. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
