> I'm a bit concerned though about how it converges into the release schedule along with dropping 3.10 + the CoC conference. I'll help with reviews and w/e necessary, but please coordinate with Rahul.
Agreed. I am quite confident it will be on time, I have it **almost** passing all the tests, and with agents I can iterate and add "conditional" building and changing the workflows to build and test things in CI in "almost no time" - my goal is to have it ready for b1 - so that the tests for beta can already use the new image. We can revert it quickly if we find it problematic - simply revert the naming, dropping "-legacy" and adding "-bleeding-edge" or something similar. It will be literally a few lines of change we can implement any time. If anything I consider b1 and 3.4.0 a great opportunity to conduct more comprehensive tests if the hardened images introduce unexpected quirks—we already know everything else. J. On Mon, Oct 5, 2026 at 12:07 PM Shahar Epstein <[email protected]> wrote: > > +1 for both the concept and the naming! > > I'm a bit concerned though about how it converges into the release schedule > along with dropping 3.10 + the CoC conference. I'll help with reviews and > w/e necessary, but please coordinate with Rahul. > > > Shahar > > > On Mon, Oct 5, 2026, 12:28 Jarek Potiuk <[email protected]> wrote: > > > Hello everyone, > > > > I would like to propose a change to how we build our PROD (and CI) Docker > > images, starting with Airflow 3.4.0. The groundwork is in > > https://github.com/apache/airflow/pull/73038. > > > > TL;DR > > > > * 3.4.0: the default PROD images are based on Docker Hardened Images > > (Python prebuilt by Docker) on Debian 13 "trixie". > > * 3.4.x: we still publish the current-style images (bookworm, Python > > compiled from source by us), but only under a new "-legacy" tag suffix, > > with a deprecation warning printed by the entrypoint. > > * 3.5.0: we drop the bookworm and legacy (non-hardened) images. > > > > 1. What are Docker Hardened Images > > > > Docker publishes "Docker Hardened Images" (DHI): minimal, continuously > > rebuilt base images with a small attack surface, published SBOMs and > > provenance. Among them are Python images, available as Debian-based > > "-dev" variants (with a shell and apt), which is what we need to build > > on. > > > > The images are free and Apache-2.0 licensed. The only friction is that > > pulling from dhi.io requires logging in with a Docker account. We do not > > want every contributor, CI job and user rebuilding our images to need > > Docker credentials. So we mirror the tags we use to > > > > ghcr.io/apache/airflow/base/python > > > > and our Dockerfiles default to that mirror. Anyone can pull it anonymously. > > > > This is fully legal: the Apache 2.0 licence explicitly allows > > redistribution, and the mirror is a byte-for-byte copy (the manifest > > digest on ghcr.io is the same as on dhi.io). A breeze command > > ("breeze release-management mirror-base-images") and a weekly workflow > > will keep the mirror current. The mirror also copies Docker's newest patch > > release before we pin it, so a version bump never points at a missing tag. > > > > 2. FIPS - a path we can offer without promising it > > > > FIPS compliance comes up regularly from users in regulated environments. > > It is not something an open-source project like ours can promise or > > certify. Docker sells FIPS-validated variants of the same hardened > > images as part of its paid offering. > > > > Because our images are now just "our layers on top of a hardened Python > > base", a user who needs FIPS can rebuild the image with a different base > > image, e.g.: > > > > docker build . \ > > --build-arg BASE_IMAGE="dhi.io/python:<version>-debian13-fips-dev" \ > > --tag my-company/airflow:3.4.0-fips > > > > That requires a paid Docker subscription, and the commercial and > > compliance relationship is between the user and Docker. I think this is > > a very good split: we keep building the open-source image, and users who > > need certified crypto get a supported path with one build argument > > instead of a fork. > > > > 3. Not everything will "just work" - expect some adaptations > > > > Hardened images are deliberately minimal and Docker patches/configures > > Python differently from the python.org / docker-library builds. Users > > who extend our image (FROM apache/airflow + apt-get install / pip > > install) or rely on details of the OS will likely need some changes. > > This is what we ran into while switching: > > > > * Patched CPython: Docker ships a fix for CVE-2026-3479, so > > pkgutil.get_data() rejects ".." in resource paths. moto < 5.2.2 loads > > its EC2 data via "../resources/...", and all EC2-based tests broke. > > Fixed by requiring moto >= 5.2.2. > > * 1 MiB thread stack: Python is built with THREAD_STACK_SIZE=1 MiB > > (previously the 8 MiB glibc default). On Python 3.12/3.13 deeply > > nested data parsed in a thread crashed the interpreter (segfault in > > a test parsing deeply nested JSON). The image now restores 8 MiB via > > a .pth file. > > * No profile-guided optimisation: Python is built with LTO but without > > PGO (--enable-optimizations), so CPU-bound pure-Python code is > > somewhat slower. > > * Modified OS files: /etc/debian_version is modified, so upgrading > > base-files stopped at dpkg's interactive conffile prompt and failed > > the build. apt keeps the image's own config files instead of prompting > > how to update (but this is not really an issue - for containers where > > you apply your own configuration changes in customisation layers) > > * Minimal OS: no "which", no lsb-release, no /etc/shells, no > > base-passwd accounts, no libpam-runtime, no gzip, no /usr/local. > > Examples: "install: invalid group 'sasl'" when installing sasl2-bin, > > tmux failing on add-shell, "chfn: PAM: Critical error" from > > adduser, "tar: gzip: Cannot exec". We restore what we need in the > > image, but users installing additional OS packages may hit similar > > issues. > > * Python lives in /opt/python (we keep /usr/python as a symlink, so > > existing paths still work). > > > > We will document all of these (see "Properties of the hardened base > > image" in the docker-stack docs in the PR), but users should expect to > > test their customised images. > > > > 4. Bookworm -> trixie > > > > Debian 12 "bookworm" will keep getting (LTS) security support for a few > > more years, but Debian 13 "trixie" is the current stable release and the > > one we should target long-term. Rather than doing the bookworm -> trixie > > and the source-built -> hardened transitions separately, I propose to do > > them together. The new default images are hardened AND trixie-based, > > and bookworm goes out together with the legacy images. > > > > The Dockerfiles and build workflow will get a bit more complex anyway > > as they will have to support both variants, but at least we will avoid > > the necessity of tessting the full matrix *bookworm/trixie x > > source/hardened" > > > > 5. Proposal - same pattern as our past transitions > > > > For 3.4.x we publish two flavours of PROD images: > > > > * Default (primary): Docker Hardened Python base, Debian 13 trixie. > > These get the standard tags (apache/airflow:3.4.0, > > apache/airflow:3.4.0-python3.12, ...). > > * Legacy: built as today - Python compiled from source by us on > > Debian 12 bookworm - for maximum compatibility. They use a new > > "-legacy" suffix (e.g. apache/airflow:3.4.0-python3.12-legacy), so > > that it is obvious from the tag that they are going away. Their > > entrypoint prints a deprecation warning saying the image is > > deprecated and that users should switch to the hardened images. > > > > 3.5.0 drops the legacy images and bookworm entirely. > > > > Testing of the legacy images: > > > > * They are NOT built/tested in canary "main" builds or in PRs. > > * A weekly "checkpoint" job on main builds and smoke-tests them, so they do > > not rot. > > * They ARE fully built and tested on the v3-4-test branch, and before > > every 3.4.x release. > > > > 6. Why this is worth it - also for CI > > > > Apart from security, the hardened base makes our CI cheaper and > > smoother. Today, whenever a new Python patch release comes out or we > > change system dependencies, the image cache is invalidated and every > > image build has to compile Python again. In practice there are a few > > days after such a change when image builds in PRs are slow, until > > everyone rebases onto the latest main. With a prebuilt Python base there > > is nothing to compile. In our measurements, the OS-dependencies layer of > > the build stage went from ~210s to ~105s, and Python patch upgrades > > become just a base-image tag bump. The image size is practically > > unchanged (+0.35%). > > > > Let me know what you think - especially about the "-legacy" naming, the > > testing split, and whether 3.5.0 is the right time to drop bookworm. > > > > If there are no strong objections, I will follow up with a [LAZY > > CONSENSUS] thread. > > > > J. > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
