potiuk opened a new pull request, #73038: URL: https://github.com/apache/airflow/pull/73038
Both image builds now consume a [Docker Hardened Image](https://dhi.io) for Python instead of compiling CPython from source on top of `debian:bookworm-slim`. `install_python()` and `install_cosign()` are gone, along with the tarball download, the sigstore/GPG verification and `scripts/docker/keys/python-3.10.asc`. 15 apt packages that existed only to build CPython (`libbz2-dev`, `tk-dev`, `libgdbm*-dev`, `uuid-dev`, `lzma*`, …) go with them. The PROD `main` stage no longer copies `/usr/python` from the build stage, because both stages get the same Python from the base. Python lives in `/opt/python` in the hardened images; `/usr/python` is now a symlink to it, so every path that documentation, volume mounts and user customizations refer to keeps working unchanged. `PYTHON_LTO` is removed — it only disabled LTO while compiling Python for FIPS environments, and there is no compilation left. FIPS builds now point `BASE_IMAGE` at a FIPS variant of the hardened image instead. ##### Base image distribution Pulling `dhi.io` requires a Docker Hub login. Rather than force credentials on every contributor and CI job, the tags are mirrored to `ghcr.io/apache/airflow/base/python` and `BASE_IMAGE` defaults there, so building needs no registry credentials at all. The images are Apache-2.0 licensed, so redistribution is fine. `breeze release-management mirror-base-images` plus a weekly workflow keep the mirror current; the mirror publishes both the pinned (`3.10.21-debian12-dev`) and floating (`3.10-debian12-dev`) tags. ##### Gaps the hardened base has versus `debian:bookworm-slim` The hardened images ship a deliberately minimal `/etc` and toolset. Four things had to be restored, all in one `restore_debian_base_files` function plus `link_python`: | Missing | Symptom | Handling | | --- | --- | --- | | `base-passwd` accounts | `sasl2-bin`: `install: invalid group 'sasl'` | install `base-passwd`, run `update-passwd` | | `/etc/shells` | `tmux`: `add-shell` aborts | create it | | `libpam-runtime` | `adduser --gecos` → `chfn: PAM: Critical error` | install it; it generates the `/etc/pam.d/common-*` files the shipped PAM configs `@include` | | `/usr/local`, `gzip` | `ln: /usr/local/bin/pip3: No such file`; `tar: gzip: Cannot exec` | `mkdir -p` in `link_python`; `gzip` in dev deps | `lsb-release` is dropped entirely — `install_mysql/mssql/postgres.sh` now read `/etc/os-release` through `common::debian_codename` / `common::debian_release`. That package pulled in `usrmerge`, whose postinst removed `/lib64` and broke the build; it was the blocker in #60123. ##### Drive-by fixes * `libxmlsec1-openssl` added to the runtime deps. `libxmlsec1` ships no crypto engine of its own, so `import xmlsec` (via `python3-saml`) failed with `libxmlsec1-openssl.so.1: cannot open shared object file`. Pre-existing — reproduced on `debian:bookworm-slim` with the same package set. * `scripts/ci/prek/upgrade_important_versions.py` kept its own hard-coded copy of the Python version list that stopped at 3.13, so the pinned 3.14 patchlevel was never bumped after 3.14 was added in #63520 — it had been stale since March. It now reads the list from `global_constants.py`, and the 3.14 pin moves to 3.14.7. ##### Verification Both images built and exercised locally on Python 3.10: * CI image: all native-extension imports (`psycopg2`, `MySQLdb`, `pyodbc`, `xmlsec`, `cryptography`, `python-ldap`, `lxml`), `airflow version`, and the gcc/helm/node/go/java/docker toolchain. * PROD image: 28/30 of `docker-tests/tests/docker_tests/test_prod_image.py`. The two failures are environmental — a local dev build installs every provider rather than the `prod_image_installed_providers.txt` set that only `prod-image-build.yml` passes, and Docker Desktop refused to bind-mount `/tmp`. * The OpenShift random-UID path works (`--user 123456:0` → `airflow version`), which `update-passwd` could plausibly have broken. * Mirror verified to be a faithful copy: the digest resolved from `ghcr.io` is identical to the one from `dhi.io`, and it resolves anonymously. ##### Size and build time Like-for-like PROD builds (same flags, same UI assets, all providers), `docker inspect .Size`: | | bytes | | --- | ---: | | `main` | 1,835,854,673 | | this branch | 1,842,348,729 | | delta | +6,494,056 (+0.35%) | The build-stage OS dependency layer went from 210.5s to 105.3s, the difference being the CPython build. ##### Follow-ups deliberately not in this PR * The pinned patchlevel's source of truth is now what Docker publishes, while the version bumper still scrapes `python.org`. They are in lockstep today, but a python.org release landing before Docker builds the matching hardened image would pin a tag that does not exist. * A version-bump PR pins a tag the mirror does not hold yet, so the mirror needs refreshing before that PR's CI can pass. * The shell-less `runtime` variant for the PROD `main` stage is not attempted here — it would mean rewriting the entrypoint away from bash and dropping the `ADDITIONAL_RUNTIME_APT_DEPS` customization contract, and it is coupled to #65384. closes: #59625 --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5) Generated-by: Claude Code (Opus 5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01DcRW9x8n7Jor7ftsKFxQHD -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
