potiuk opened a new pull request, #70650: URL: https://github.com/apache/airflow/pull/70650
Docs publishes rebuild the CI image from scratch whenever the ref being documented was cut before the last `Dockerfile.ci` change on main. The registry cache the build reads is main's, so #70285 (Node.js/pnpm baked into the image) breaks the cache chain at layer 3 of 26 and the ~20 minute Python build runs again. Measured on run 30398809294: 1 of 26 layers `CACHED`, `install_os_dependencies.sh` 1173.7s, build step 25m24s. A canary main build, whose sources match its cache, is 6m42s. The image the previous publish of that same ref stashed matches those layers exactly, so this seeds the build from it. - `ci-image-build.yml` gains `seed-cache-from-stashed-image` (default off). When on, it restores the stashed image, republishes it to a `registry:2` on localhost and passes it to the build as an extra `--cache-from`. - The registry hop is required, not incidental. BuildKit imports the cache manifest of a `docker load`ed image and then discards every record whose layers it cannot pull from a registry. Verified locally with the same image, same builder and same args: through `docker load` every layer re-ran, through `localhost:5000` every layer came back `CACHED`. - Images are now built with `BUILDKIT_INLINE_CACHE=1` so they carry those records at all. Also verified that this arg does not participate in layer cache keys, so it does not invalidate the existing registry cache when it lands. - `image-stash-suffix` keeps a per-ref copy of the image stash, retained 6 days because the same ref is rebuilt at least three days apart. Seeding prefers that copy and falls back to the shared per-branch stash, which Tests AMD keeps warm for main. Only `publish-docs-to-s3.yml` opts in. Tests AMD is unchanged apart from the inline cache record in the images it produces. Expected effect: re-publishing the same ref should build in a couple of minutes instead of 25; a different ref that shares the image-defining files should skip the 20 minute layer; a genuine miss costs about 6 minutes of seeding once, and that run then writes its own per-ref slot. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5) Generated-by: Claude Code (Opus 5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
