potiuk opened a new pull request, #76023: URL: https://github.com/apache/airflow/pull/76023
### CI reproducibility CI and PROD images were built on the moving tags of our ghcr.io mirror of the Docker Hardened Images. Docker rebuilds those tags in place, so every mirror refresh changed the image every build started from. Nothing changed in the repository, and the full CI had not run on the new image. A CI run could pass or fail depending only on when it ran, and rebuilding the same commit later produced a different image. Today's breakage of "Test examples of PROD image building" is an example ([run](https://github.com/apache/airflow/actions/runs/40968123434/job/123973461638)). The mirrored base has `libbz2-1.0 1.0.8-6+dhi2`, while the dhi.io apt repository already moved to a `bzip2` that needs `1.0.8-6+deb13u0+dhi2`, so `build-essential` cannot be installed (docker-hardened-images/catalog#689). ### What changes - **Builds use a digest, not a tag.** `BASE_IMAGE` is now `ghcr.io/apache/airflow/base/python:<tag>@sha256:<digest>` for breeze-built CI and PROD images and for the `Dockerfile` / `Dockerfile.ci` defaults. The digests live in `dev/breeze/src/airflow_breeze/files/hardened_python_image_digests.json`, covering Python 3.11–3.14 on Debian 13 and 12. - **Only the upgrade check moves the pins.** `breeze ci upgrade` (`upgrade_important_versions.py`) reads the current digests from the mirror, rewrites the pins file and every reference that already carries a digest, and opens the upgrade PR. A new base image therefore reaches the builds only after the full CI has passed on it. - **The mirror refreshes every six hours instead of weekly.** When a run moves a tag past its pin, it triggers the main upgrade check, so the pins follow Docker's rebuilds within hours. This matters because the dhi.io apt repository moves together with the images, and a pin left behind for long stops matching what apt installs. The scheduled Mon/Wed/Fri upgrade check stays as the backstop, for example when that trigger is missed. - **Pinned digests stay tagged.** The mirror tags every mirrored and every pinned digest as `<tag>-<digest12>`, so the image a pin or an open upgrade PR refers to never becomes untagged. - **Local rebuilds follow the pins.** The pins file is in `FILES_FOR_REBUILD_CHECK`, so a changed pin also changes the CI image sources hash and makes breeze rebuild. - **apt prefers Docker's packages.** The hardened images keep stock Debian's sources beside dhi.io, and apt took whichever offered the higher version, mixing Debian builds into an image whose dependencies Docker built (docker-hardened-images/catalog#610). `/etc/apt/preferences.d/airflow-prefer-dhi` ranks dhi.io at 900, so apt takes Docker's build whenever dhi.io publishes the package and still never downgrades an installed package. The file stays in the image, so images extending it get the same behaviour. - **The docs examples keep a readable tag.** The `.sh` examples use `${BASE_IMAGE:-ghcr.io/apache/airflow/base/python:3.13.16-debian13-dev}`, so a pinned image can be passed in from outside. CI builds the example Dockerfiles on the CI-built PROD image, which already carries the pin. The initial pins are the digests the mirror serves today, so this PR does not change the base image the builds use. Picking up Docker's fixed images is the first job of the new flow: a mirror run, then the upgrade PR it triggers. ### Not covered Only the base image is pinned. Packages installed with apt during the build still come from the live repositories. Neither Docker nor the third-party repositories we use offer snapshots we could pin to, and pinning individual package versions would fail as soon as a repository drops a superseded version. ### Checks run - `uv run --project dev/breeze pytest dev/breeze/tests/test_build_prod_params.py dev/breeze/tests/test_mirror_base_images.py`: passed. - `uv run --project scripts pytest scripts/tests/ci/prek/test_upgrade_important_versions.py`: passed. - `prek run --from-ref main --stage pre-commit`: passed, except `lychee-docker`, `lint-dockerfile` and `shellcheck`, which were skipped because they need a local Docker daemon. They run in CI. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: Claude Code (Opus 5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
