bryancall opened a new pull request, #475:
URL: https://github.com/apache/trafficserver-ci/pull/475
The GitHub PR pipelines run the source checkout once, with no timeout and no
retry, so a single failed or hung fetch from the CI mirror fails the check. The
only limit today is git's own 20 minute command timeout from `CloneOption`.
This wraps the checkout in all 13 PR pipelines the same way:
```groovy
int attempt = 0
retry(3) {
if (attempt > 0) {
sleep(time: 30 * attempt + (int) (Math.random() * 60), unit:
'SECONDS')
}
attempt++
timeout(time: 5, unit: 'MINUTES') {
checkout(...)
}
}
```
| Try | Waits before it | Its own timeout |
|---|---|---|
| 1 | none | 5 min |
| 2 | 30 to 90 s, random | 5 min |
| 3 | 60 to 120 s, random | 5 min |
The random wait keeps jobs that failed together, for example during a mirror
restart, from all hitting the mirror again at the same moment. The wait sits
outside the timeout, so it does not use up a try's 5 minutes. Worst case is
about 18.5 minutes before the check fails. `retry` does retry after a nested
`timeout` fires: the timeout marks its own interruption as not an actual one
([TimeoutStepExecution.java#L253](https://github.com/jenkinsci/workflow-basic-steps-plugin/blob/1098.v808b_fd7f8cf4/src/main/java/org/jenkinsci/plugins/workflow/steps/TimeoutStepExecution.java#L253))
and `retry` only stops on actual ones
([RetryStepExecution.java#L93](https://github.com/jenkinsci/workflow-basic-steps-plugin/blob/1098.v808b_fd7f8cf4/src/main/java/org/jenkinsci/plugins/workflow/steps/RetryStepExecution.java#L93)).
`Math.random()` is on the script security allowlist, so it needs no approval.
The 5 minutes comes from the Clone stage times of the last 10 runs of each
of the 13 PR jobs on Oct 14, 129 runs, all of them successful:
| | Clone stage |
|---|---|
| Median | 23 s |
| p90 | 36 s |
| p99 | 61 s |
| Slowest (freebsd) | 66 s |
That is about 5 times the slowest clone seen. The stage also includes a
short `sh` step after the checkout, so the checkout itself is a little faster
than these numbers.
One trade-off: a PR that does not merge cleanly into its target fails in the
`PreBuildMerge` step of the checkout, so it will now be retried too and fail
about 2 to 4 minutes later than it does today.
Checked all 13 files with the controller's declarative linter
(`pipeline-model-converter/validate`), which also rejects a known-bad control
file. Not yet exercised in a Jenkins run.
Part of
[apache/trafficserver#13839](https://github.com/apache/trafficserver/issues/13839).
The same change for the branch pipelines comes separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
