potiuk opened a new pull request, #71171:
URL: https://github.com/apache/airflow/pull/71171
`v3-3-test` builds its `linux/amd64` CI images from scratch, ~20 minutes per
Python version, because the registry has no cache for them:
```
#20 importing cache manifest from
ghcr.io/apache/airflow/v3-3-test/ci/python3.11:cache-linux-amd64
#20 ERROR: failed to configure registry cache importer:
ghcr.io/apache/airflow/v3-3-test/ci/python3.11:cache-linux-amd64: not
found
```
The registry holds `cache-linux-amd64` for Python 3.10 alone on that branch,
while `cache-linux-arm64` exists for all five.
### Why
A branch built on `push` refreshed its cache from inside the CI run, which
cancels in progress on the next push. With no cache the build runs cold, which
widens the window the next push cancels in, which leaves the cache missing. Run
`30973271401` is the loop in one screenshot:
```
success 04:09–04:46 Build CI linux/amd64 image 3.10 … 3.14 (all
five built)
success 04:58–05:17:11 Push CI Early:3.10 image cache ← the
one tag that exists
cancelled 04:58–05:17:44 Push CI Early:3.11 image cache
cancelled 04:58–05:17:43 Push CI Early:3.13 image cache
cancelled 04:58–05:17:41 Push CI Early:3.12 image cache
cancelled 04:57–05:17:44 Push CI Early:3.14 image cache
```
The next push landed at 05:17:26 and killed four of the five, fifteen
seconds after the first finished. Of the last 60 AMD runs on that branch, 45
were cancelled and 14 failed — so the Regular cache push in
`finalize-tests.yml`, which sits behind the whole test matrix, never runs there
either.
A job cannot opt out of its run being cancelled, so the refresh has to be a
workflow of its own.
### What this does
**`refresh-image-registry-cache.yml`** — refreshes the CI cache for both
platforms, on `workflow_dispatch` and on `push` to release-prep / providers
branches. Push runs are never cancelled (a newer one queues behind); manual
runs are grouped separately, so a merge cannot cancel one, and re-firing it
replaces the run in flight.
**Every Python version, both paths.** Selective checks narrows
`python-versions` to the default one on a push where only text files changed
(`_should_run_all_tests_and_versions`), which is a second, independent way a
branch ends up with cache for one version. The workflow forces the `all
versions` label; a new test in `test_selective_checks.py` pins that.
**The early cache push is removed from the regular build.** It is what this
workflow replaces, so `additional-ci-image-checks.yml` loses that job and the
seven `workflow_call` inputs only it consumed (removed from both wrappers
identically, so `check-ci-workflows-in-sync` stays green). `main` is
unaffected: it is not built on push, and its Push Image Cache job at the end of
the scheduled canary runs to completion.
**`refresh-constraints.yml`** — the manual constraints refresh, split out of
`update-constraints-on-push.yml` so it is not filtered by that workflow's
`paths: uv.lock` and appears under its own name. It calls that workflow rather
than copying it, and refreshes all three flavours — source-providers,
no-providers and PyPI — since refreshing only PyPI leaves CI and Breeze on the
older pins. `release-constraints.yml` is untouched.
**Docs** — both procedures in
`dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md` now lead with the
workflows; the local buildx / qemu / `refresh_images.sh` route is kept but
demoted to a fallback. The automated-CI section no longer describes the early
cache push this PR deletes.
Also fixed in `dev/MANUALLY_BUILDING_IMAGES.md`: the merge step was
documented as `docker release-management merge-prod-images` in both the regular
and slim variants — it is a `breeze` subcommand, so the documented command
fails as written.
### Follow-ups this does not do
- `v3-3-test` needs the workflow backported before its cache can be
refreshed there — `workflow_dispatch` runs the file from the branch you select.
- `dev/MANUALLY_BUILDING_IMAGES.md` contradicts itself on the buildx version
(a warning says "at least 0.13", the prose says 0.23 is needed for
`docker-container`). Left alone rather than guessed at.
- The buildx setup instructions now exist in both dev docs and could be
collapsed to one copy.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Opus 5)
Generated-by: Claude Code (Opus 5) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
