potiuk opened a new pull request, #70618:
URL: https://github.com/apache/airflow/pull/70618
Addresses Question 1 of #70587, and takes Question 2's parallel option.
A provider docs publish built **two** CI images from the same sources — the
docs job inline at `publish-docs-to-s3.yml:301-318` (python3.10) and the
registry's own `build-ci-image` job (python3.12) — and only started the
registry once the docs were already live on S3. Per #70587 the registry
sub-workflow medians 23m, of which ~13m is that second image build, and it is
100% additive because it is strictly serial after the publish.
### What changes
* **One image per run.** A `build-ci-image` job in `publish-docs-to-s3.yml`
builds from `inputs.ref` and stashes it; `build-docs` restores it instead of
building inline, and `update-registry` passes `ci-image-already-built: true` so
`registry-build.yml` skips its own build. A standalone registry dispatch has no
such image and still builds one.
* **Both on one Python.** `default-python-version` moves 3.10 → 3.12, which
is what the registry has always extracted on. This removes the second image
that differed only by interpreter. Note `dev/registry/extract_metadata.py:47`
and `extract_versions.py:49` fall back to `tomli` below 3.11 — that path is `#
pragma: no cover`, so 3.12 is the better-exercised side.
* **Docs and registry run side by side.** `update-registry` moves from
`needs: [publish-docs-to-s3, build-info]` to `needs: [build-info,
build-ci-image]`. The registry reads nothing the docs publish produces.
* **The no-cache fallback is preserved**, moved into the build step it
belongs to. `ci-image-build.yml` gains a `retry-without-cache` input (**default
`false`**, so every other caller is unaffected); when the cached build fails it
retries once with `DOCKER_CACHE=disabled`. This keeps the behaviour of the
inline `breeze ci-image build || docker buildx build --pull` it replaces, for
old refs whose `pyproject.toml` has drifted from main.
* **Mount cache retained.** Both image builds now set
`upload-mount-cache-artifact: "true"`, so the next publish on the branch
restores it rather than re-downloading the dependency set.
* **Cache source unchanged** — `docker-cache: registry` against `branch:
main`, the cache the regular Test workflow keeps warm.
Resulting graph:
```
build-info -> build-ci-image -> { build-docs, update-registry }
```
### Consequences worth reviewing
1. **The failed-docs gate is gone.** `needs: [publish-docs-to-s3]`
previously *gated* the registry, so a failed docs build skipped it. Running in
parallel removes that: the registry can publish `/stable/` links to docs that
never landed. #70587 measures the exposure window at median ~3 min, up to ~21
min, negative on 5 of 11 sampled runs. If that is not acceptable, the gate
should move onto the S3 sync step rather than back onto the whole job.
2. **The registry publish job re-asserts the committer allowlist.** It
previously inherited it by being `needs:` a job the allowlist skipped; now that
job is also skipped on the shared-image path, so the check is stated explicitly
on `build-and-publish-registry`.
3. **3.10 → 3.12 is exercised by the registry but new for the docs build.**
Nothing in the docs path looked version-specific, but this is the change least
verifiable without a real release run.
Not done here, and worth a separate look: restoring the canary image instead
of building at all. It does not work through the stash — the Tests (AMD) run
stashes only the default Python (`ci-image-save-v3-linux_amd64-3.10-…`), the
stash name is branch-suffixed with no input to select a branch (current +
base/default only), and artifacts expire in ~2 days. The cross-run-viable route
would be pulling `ghcr.io/apache/airflow/<branch>/ci/python<ver>:latest`, which
`push-image-cache.yml` publishes for every canary Python.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Opus 5)
Generated-by: Claude Code (Opus 5) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
