potiuk opened a new pull request, #70524:
URL: https://github.com/apache/airflow/pull/70524

   An integration job dies whenever Docker Hub is briefly unreachable from the 
runner, before a single test executes. Example — [run 
30267566636](https://github.com/apache/airflow/actions/runs/30267566636/job/89987964575),
 "Integration core otel":
   
   ```
   13:24:45  otel-collector Error Get "https://registry-1.docker.io/v2/":
             net/http: request canceled (Client.Timeout exceeded while awaiting 
headers)
   13:25:29  postgres Error Get "https://registry-1.docker.io/v2/": context 
deadline exceeded
   13:25:39  The core integration tests failed for the second time! Giving up
   ```
   
   `docker compose run` had to pull `otel/opentelemetry-collector-contrib` (and 
`postgres` on the retry) and the registry did not answer. Two things made that 
fatal:
   
   * the pull is done implicitly by `docker compose run`, so there is no retry 
at the level where the failure happens — recovery costs a full re-run of the 
suite;
   * `run_integration_tests_with_retry.sh` starts that re-run immediately, so 
both attempts land inside a minute (13:24:45 and 13:25:29). Anything longer 
than a one-minute blip cannot be survived.
   
   This changes both:
   
   * `breeze testing {core,providers}-integration-tests` now pulls the 
third-party images the compose project needs before running, retrying each with 
a 15/30/45/60s backoff. The locally built CI image is excluded, and images 
already present locally are not touched, so a warm runner or a local run pulls 
nothing.
   * the retry script waits (`RETRY_DELAY_SECONDS`, default 60) before the 
second attempt.
   
   Behaviour is unchanged when the registry is healthy. If an image genuinely 
cannot be pulled the run proceeds and fails with docker compose's own error, as 
before.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to