bryancall opened a new pull request, #476:
URL: https://github.com/apache/trafficserver-ci/pull/476
The branch pipelines run `timeout(10) { retry(3) { checkout } }`. The
timeout covers all three tries together, so one hung try uses up the budget and
the retries never run.
[#473](https://github.com/apache/trafficserver-ci/pull/473) raised that shared
budget from 1 minute to 10 but kept the order. This replaces it with a budget
per try:
```groovy
int attempt = 0
retry(3) {
if (attempt > 0) {
sleep(time: 30 * attempt + (int) (Math.random() * 60), unit:
'SECONDS')
}
attempt++
timeout(time: 5, unit: 'MINUTES') {
checkout(...)
}
}
```
| What the clone does | Today | After |
|---|---|---|
| Fails fast | Retried at once, all within 10 min | Retried after a random
30 to 90 s, then 60 to 120 s |
| Hangs | Try 1 uses all 10 min, tries 2 and 3 never run | Try 1 is stopped
at 5 min, tries 2 and 3 run |
| Slow, 5 to 10 min | Passes on try 1 | Stopped at 5 min and retried |
| Worst case before the stage fails | 10 min | about 18.5 min |
The random wait keeps children that failed together, for example during a
mirror restart, from all hitting the mirror again at the same moment. It sits
outside the timeout, so it does not use up a try's 5 minutes. `retry` does
retry after a nested `timeout` fires: the timeout marks its own interruption as
not an actual one
([TimeoutStepExecution.java#L253](https://github.com/jenkinsci/workflow-basic-steps-plugin/blob/1098.v808b_fd7f8cf4/src/main/java/org/jenkinsci/plugins/workflow/steps/TimeoutStepExecution.java#L253))
and `retry` only stops on actual ones
([RetryStepExecution.java#L93](https://github.com/jenkinsci/workflow-basic-steps-plugin/blob/1098.v808b_fd7f8cf4/src/main/java/org/jenkinsci/plugins/workflow/steps/RetryStepExecution.java#L93)).
`Math.random()` is on the script security allowlist, so it needs no approval.
The 5 minutes comes from recent successful branch Clone stages, 44 runs
across 14 jobs on Oct 14: median 43 s, p90 59 s, slowest 117 s (freebsd). That
is about 2.5 times the slowest. Some of those runs predate
[#474](https://github.com/apache/trafficserver-ci/pull/474), so branch-only
clones should be faster.
freebsd, osx and osx-m1 had no timeout or retry on the source checkout at
all, so they get the same block. This does not fix whatever makes their Clone
stage fail, which is still open, but a failed try now gets two more. The
checkout comment also now says that a requested SHA must be on the branch being
built, since #474 fetches only that branch.
Checked all 15 files with the controller's declarative linter
(`pipeline-model-converter/validate`). Not yet exercised in a Jenkins run. The
same change for the GitHub PR pipelines is
[#475](https://github.com/apache/trafficserver-ci/pull/475).
Part of
[apache/trafficserver#13839](https://github.com/apache/trafficserver/issues/13839).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
