SEZ9 commented on PR #12678:
URL: https://github.com/apache/seatunnel/pull/12678#issuecomment-6937367030
@kalayciburak thanks for the direct answers on both points.
**Delete failure during retention pruning**
- The description saying a reported delete failure doesn't fail the
checkpoint, plus
`CheckpointCoordinatorTest#testRetentionDeleteFailureRetriesOnNextCompletion`
with mocked checked and unchecked delete failures, covers the coordinator-side
requeue and retry.
- As you note, `notifyCompleted` is stubbed there and
`CheckpointStorageRetentionPruneTest` only covers successful pruning, so no
test exercises a real storage failure end to end yet. Please push the
storage-backed failure/retry test you mentioned. It should show that the
checkpoint still completes and that the id whose delete failed is removed on a
later completion rather than left behind.
- One part of the original point is still unanswered: pruning is exercised
on every completion instead of once per `max-retained`. Is that still the case
at `71ac656652f0`? If so, please confirm that an id whose delete keeps failing
stays queued for retry and is never dropped.
**Bound after a restart**
- Thanks for confirming the deque still starts empty after a coordinator
restart, so the bound only holds for a fresh job.
- I can't find an earlier agreement in this thread to keep that out of
scope, so I can't confirm it as an existing plan. Could you link the comment
you mean?
- I'm fine with deferring it to the follow-up you linked, on two conditions:
- The documented limitation states the concrete effect: after a pipeline
restart, storage can hold up to 2x `max-retained`, and the pre-restart files
are never pruned.
- The follow-up explicitly covers seeding `completedCheckpointIds` on
restore.
- Please also point me to where the limitation is documented, since the text
isn't quoted in the thread.
**Verification**
Your local results at `71ac656652f0` (18 and 1 passing) are noted, but as
you say they aren't an exact-head Build. Once the storage-backed test is on the
branch, please link a Build run for that head.
<!-- streview-comment:2401 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]