+1 for both the concept and the naming! I'm a bit concerned though about how it converges into the release schedule along with dropping 3.10 + the CoC conference. I'll help with reviews and w/e necessary, but please coordinate with Rahul.
Shahar On Mon, Oct 5, 2026, 12:28 Jarek Potiuk <[email protected]> wrote: > Hello everyone, > > I would like to propose a change to how we build our PROD (and CI) Docker > images, starting with Airflow 3.4.0. The groundwork is in > https://github.com/apache/airflow/pull/73038. > > TL;DR > > * 3.4.0: the default PROD images are based on Docker Hardened Images > (Python prebuilt by Docker) on Debian 13 "trixie". > * 3.4.x: we still publish the current-style images (bookworm, Python > compiled from source by us), but only under a new "-legacy" tag suffix, > with a deprecation warning printed by the entrypoint. > * 3.5.0: we drop the bookworm and legacy (non-hardened) images. > > 1. What are Docker Hardened Images > > Docker publishes "Docker Hardened Images" (DHI): minimal, continuously > rebuilt base images with a small attack surface, published SBOMs and > provenance. Among them are Python images, available as Debian-based > "-dev" variants (with a shell and apt), which is what we need to build > on. > > The images are free and Apache-2.0 licensed. The only friction is that > pulling from dhi.io requires logging in with a Docker account. We do not > want every contributor, CI job and user rebuilding our images to need > Docker credentials. So we mirror the tags we use to > > ghcr.io/apache/airflow/base/python > > and our Dockerfiles default to that mirror. Anyone can pull it anonymously. > > This is fully legal: the Apache 2.0 licence explicitly allows > redistribution, and the mirror is a byte-for-byte copy (the manifest > digest on ghcr.io is the same as on dhi.io). A breeze command > ("breeze release-management mirror-base-images") and a weekly workflow > will keep the mirror current. The mirror also copies Docker's newest patch > release before we pin it, so a version bump never points at a missing tag. > > 2. FIPS - a path we can offer without promising it > > FIPS compliance comes up regularly from users in regulated environments. > It is not something an open-source project like ours can promise or > certify. Docker sells FIPS-validated variants of the same hardened > images as part of its paid offering. > > Because our images are now just "our layers on top of a hardened Python > base", a user who needs FIPS can rebuild the image with a different base > image, e.g.: > > docker build . \ > --build-arg BASE_IMAGE="dhi.io/python:<version>-debian13-fips-dev" \ > --tag my-company/airflow:3.4.0-fips > > That requires a paid Docker subscription, and the commercial and > compliance relationship is between the user and Docker. I think this is > a very good split: we keep building the open-source image, and users who > need certified crypto get a supported path with one build argument > instead of a fork. > > 3. Not everything will "just work" - expect some adaptations > > Hardened images are deliberately minimal and Docker patches/configures > Python differently from the python.org / docker-library builds. Users > who extend our image (FROM apache/airflow + apt-get install / pip > install) or rely on details of the OS will likely need some changes. > This is what we ran into while switching: > > * Patched CPython: Docker ships a fix for CVE-2026-3479, so > pkgutil.get_data() rejects ".." in resource paths. moto < 5.2.2 loads > its EC2 data via "../resources/...", and all EC2-based tests broke. > Fixed by requiring moto >= 5.2.2. > * 1 MiB thread stack: Python is built with THREAD_STACK_SIZE=1 MiB > (previously the 8 MiB glibc default). On Python 3.12/3.13 deeply > nested data parsed in a thread crashed the interpreter (segfault in > a test parsing deeply nested JSON). The image now restores 8 MiB via > a .pth file. > * No profile-guided optimisation: Python is built with LTO but without > PGO (--enable-optimizations), so CPU-bound pure-Python code is > somewhat slower. > * Modified OS files: /etc/debian_version is modified, so upgrading > base-files stopped at dpkg's interactive conffile prompt and failed > the build. apt keeps the image's own config files instead of prompting > how to update (but this is not really an issue - for containers where > you apply your own configuration changes in customisation layers) > * Minimal OS: no "which", no lsb-release, no /etc/shells, no > base-passwd accounts, no libpam-runtime, no gzip, no /usr/local. > Examples: "install: invalid group 'sasl'" when installing sasl2-bin, > tmux failing on add-shell, "chfn: PAM: Critical error" from > adduser, "tar: gzip: Cannot exec". We restore what we need in the > image, but users installing additional OS packages may hit similar > issues. > * Python lives in /opt/python (we keep /usr/python as a symlink, so > existing paths still work). > > We will document all of these (see "Properties of the hardened base > image" in the docker-stack docs in the PR), but users should expect to > test their customised images. > > 4. Bookworm -> trixie > > Debian 12 "bookworm" will keep getting (LTS) security support for a few > more years, but Debian 13 "trixie" is the current stable release and the > one we should target long-term. Rather than doing the bookworm -> trixie > and the source-built -> hardened transitions separately, I propose to do > them together. The new default images are hardened AND trixie-based, > and bookworm goes out together with the legacy images. > > The Dockerfiles and build workflow will get a bit more complex anyway > as they will have to support both variants, but at least we will avoid > the necessity of tessting the full matrix *bookworm/trixie x > source/hardened" > > 5. Proposal - same pattern as our past transitions > > For 3.4.x we publish two flavours of PROD images: > > * Default (primary): Docker Hardened Python base, Debian 13 trixie. > These get the standard tags (apache/airflow:3.4.0, > apache/airflow:3.4.0-python3.12, ...). > * Legacy: built as today - Python compiled from source by us on > Debian 12 bookworm - for maximum compatibility. They use a new > "-legacy" suffix (e.g. apache/airflow:3.4.0-python3.12-legacy), so > that it is obvious from the tag that they are going away. Their > entrypoint prints a deprecation warning saying the image is > deprecated and that users should switch to the hardened images. > > 3.5.0 drops the legacy images and bookworm entirely. > > Testing of the legacy images: > > * They are NOT built/tested in canary "main" builds or in PRs. > * A weekly "checkpoint" job on main builds and smoke-tests them, so they do > not rot. > * They ARE fully built and tested on the v3-4-test branch, and before > every 3.4.x release. > > 6. Why this is worth it - also for CI > > Apart from security, the hardened base makes our CI cheaper and > smoother. Today, whenever a new Python patch release comes out or we > change system dependencies, the image cache is invalidated and every > image build has to compile Python again. In practice there are a few > days after such a change when image builds in PRs are slow, until > everyone rebases onto the latest main. With a prebuilt Python base there > is nothing to compile. In our measurements, the OS-dependencies layer of > the build stage went from ~210s to ~105s, and Python patch upgrades > become just a base-image tag bump. The image size is practically > unchanged (+0.35%). > > Let me know what you think - especially about the "-legacy" naming, the > testing split, and whether 3.5.0 is the right time to drop bookworm. > > If there are no strong objections, I will follow up with a [LAZY > CONSENSUS] thread. > > J. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
