potiuk opened a new pull request, #77110:
URL: https://github.com/apache/airflow/pull/77110

   The registry cache of the CI image on `main` is refreshed only by the 
canary, twice a day, under one moving tag per platform. After a dependency 
upgrade merges, every local `breeze ci-image build` reinstalls all dependencies 
from scratch for up to half a day; and a checkout of any other commit, branch 
or worktree whose dependency inputs differ from the latest canary never hits 
the cache at all. This keeps the cache usable right after dependency changes, 
and for past ones.
   
   **Keyed cache.** Every cache push now also writes 
`cache-linux-<arch>-<key>`, where the key hashes the inputs of the image's 
dependency layers: `uv.lock`, every `pyproject.toml`, `Dockerfile.ci` and the 
scripts it runs. `breeze ci-image build` passes `--cache-from` for its 
checkout's keyed tag first and the shared tag second. The key only names the 
cache to look up: BuildKit still checks every layer, so a wrong key costs a 
cache miss, never a wrong layer.
   
   **Pushed before it is needed.**
   - `refresh-image-cache-on-dependency-change.yml` refreshes the shared and 
keyed cache right after a push to `main` that changes those inputs (60 commits 
in the last 4 weeks; bursts coalesce into one queued run).
   - A pull request from an `apache/airflow` branch that changes them (e.g. the 
automated CI environment upgrade) pushes its keyed cache for every supported 
Python version and both platforms, built without upgrading dependencies, as 
`main` will after the merge. A key already in the registry is skipped; the job 
never moves the shared cache or `latest`; fork pull requests never run it. New 
selective-checks outputs `ci-image-dependencies-changed` and 
`ci-image-cache-python-versions` drive it.
   
   **buildx check.** Writing two caches in one build needs BuildKit 0.11 
([moby/buildkit#3024](https://github.com/moby/buildkit/pull/3024)), so breeze 
now stops before building an image with docker buildx older than 0.10.
   
   **Cleanup.**
   - `cleanup-keyed-image-cache.yml` (daily) deletes keyed tags 30 days after 
they were first pushed.
   - `cleanup-old-image-versions.yml` (every 6 hours) slowly deletes the 
untagged versions every push leaves behind (`airflow/main/ci/python3.11` holds 
~43k versions), oldest first, 300 per run, one package per run, so it stays far 
inside the workflow token's rate limit. A version is deleted only when no tag 
references it: what every tag lists is read anonymously from the registry, and 
versions pushed within 5 minutes of a tagged one are kept too. Manual runs 
default to dry run.
   
   Tested:
   - breeze unit tests (2798 passed), including new tests for the cache key, 
keyed `--cache-from`/`--cache-to`, the keyed-only push, the buildx check and 
the selective-checks outputs; scripts tests for both cleanups (32 passed).
   - Read-only dry runs against ghcr.io: the keyed cleanup scans the 76 CI 
packages in ~4 minutes; the old-versions cleanup read the 18,098 tags of 
`airflow/main/ci/python3.10` (referencing 25,519 manifests, ~40 minutes) and 
selected 30 unreferenced versions from 2022.
   - Not yet exercised: a real push of two caches to a registry (the 
keyed-cache PR job only runs on pull requests that change dependency inputs, 
which this one does not), and the cleanups with the workflow token instead of a 
personal one; the first dispatched runs of the cleanups default to dry run.
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes — Claude Code (Claude Opus 5)
   
   Generated-by: Claude Code (Claude Opus 5) following [the 
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   
   https://claude.ai/code/session_01NyQq9U7i5HhjDgrQ9qd3gq
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to