Hello everyone,
I would like to propose a change to how we build our PROD (and CI) Docker
images, starting with Airflow 3.4.0. The groundwork is in
https://github.com/apache/airflow/pull/73038.
TL;DR
* 3.4.0: the default PROD images are based on Docker Hardened Images
(Python prebuilt by Docker) on Debian 13 "trixie".
* 3.4.x: we still publish the current-style images (bookworm, Python
compiled from source by us), but only under a new "-legacy" tag suffix,
with a deprecation warning printed by the entrypoint.
* 3.5.0: we drop the bookworm and legacy (non-hardened) images.
1. What are Docker Hardened Images
Docker publishes "Docker Hardened Images" (DHI): minimal, continuously
rebuilt base images with a small attack surface, published SBOMs and
provenance. Among them are Python images, available as Debian-based
"-dev" variants (with a shell and apt), which is what we need to build
on.
The images are free and Apache-2.0 licensed. The only friction is that
pulling from dhi.io requires logging in with a Docker account. We do not
want every contributor, CI job and user rebuilding our images to need
Docker credentials. So we mirror the tags we use to
ghcr.io/apache/airflow/base/python
and our Dockerfiles default to that mirror. Anyone can pull it anonymously.
This is fully legal: the Apache 2.0 licence explicitly allows
redistribution, and the mirror is a byte-for-byte copy (the manifest
digest on ghcr.io is the same as on dhi.io). A breeze command
("breeze release-management mirror-base-images") and a weekly workflow
will keep the mirror current. The mirror also copies Docker's newest patch
release before we pin it, so a version bump never points at a missing tag.
2. FIPS - a path we can offer without promising it
FIPS compliance comes up regularly from users in regulated environments.
It is not something an open-source project like ours can promise or
certify. Docker sells FIPS-validated variants of the same hardened
images as part of its paid offering.
Because our images are now just "our layers on top of a hardened Python
base", a user who needs FIPS can rebuild the image with a different base
image, e.g.:
docker build . \
--build-arg BASE_IMAGE="dhi.io/python:<version>-debian13-fips-dev" \
--tag my-company/airflow:3.4.0-fips
That requires a paid Docker subscription, and the commercial and
compliance relationship is between the user and Docker. I think this is
a very good split: we keep building the open-source image, and users who
need certified crypto get a supported path with one build argument
instead of a fork.
3. Not everything will "just work" - expect some adaptations
Hardened images are deliberately minimal and Docker patches/configures
Python differently from the python.org / docker-library builds. Users
who extend our image (FROM apache/airflow + apt-get install / pip
install) or rely on details of the OS will likely need some changes.
This is what we ran into while switching:
* Patched CPython: Docker ships a fix for CVE-2026-3479, so
pkgutil.get_data() rejects ".." in resource paths. moto < 5.2.2 loads
its EC2 data via "../resources/...", and all EC2-based tests broke.
Fixed by requiring moto >= 5.2.2.
* 1 MiB thread stack: Python is built with THREAD_STACK_SIZE=1 MiB
(previously the 8 MiB glibc default). On Python 3.12/3.13 deeply
nested data parsed in a thread crashed the interpreter (segfault in
a test parsing deeply nested JSON). The image now restores 8 MiB via
a .pth file.
* No profile-guided optimisation: Python is built with LTO but without
PGO (--enable-optimizations), so CPU-bound pure-Python code is
somewhat slower.
* Modified OS files: /etc/debian_version is modified, so upgrading
base-files stopped at dpkg's interactive conffile prompt and failed
the build. apt keeps the image's own config files instead of prompting
how to update (but this is not really an issue - for containers where
you apply your own configuration changes in customisation layers)
* Minimal OS: no "which", no lsb-release, no /etc/shells, no
base-passwd accounts, no libpam-runtime, no gzip, no /usr/local.
Examples: "install: invalid group 'sasl'" when installing sasl2-bin,
tmux failing on add-shell, "chfn: PAM: Critical error" from
adduser, "tar: gzip: Cannot exec". We restore what we need in the
image, but users installing additional OS packages may hit similar
issues.
* Python lives in /opt/python (we keep /usr/python as a symlink, so
existing paths still work).
We will document all of these (see "Properties of the hardened base
image" in the docker-stack docs in the PR), but users should expect to
test their customised images.
4. Bookworm -> trixie
Debian 12 "bookworm" will keep getting (LTS) security support for a few
more years, but Debian 13 "trixie" is the current stable release and the
one we should target long-term. Rather than doing the bookworm -> trixie
and the source-built -> hardened transitions separately, I propose to do
them together. The new default images are hardened AND trixie-based,
and bookworm goes out together with the legacy images.
The Dockerfiles and build workflow will get a bit more complex anyway
as they will have to support both variants, but at least we will avoid
the necessity of tessting the full matrix *bookworm/trixie x source/hardened"
5. Proposal - same pattern as our past transitions
For 3.4.x we publish two flavours of PROD images:
* Default (primary): Docker Hardened Python base, Debian 13 trixie.
These get the standard tags (apache/airflow:3.4.0,
apache/airflow:3.4.0-python3.12, ...).
* Legacy: built as today - Python compiled from source by us on
Debian 12 bookworm - for maximum compatibility. They use a new
"-legacy" suffix (e.g. apache/airflow:3.4.0-python3.12-legacy), so
that it is obvious from the tag that they are going away. Their
entrypoint prints a deprecation warning saying the image is
deprecated and that users should switch to the hardened images.
3.5.0 drops the legacy images and bookworm entirely.
Testing of the legacy images:
* They are NOT built/tested in canary "main" builds or in PRs.
* A weekly "checkpoint" job on main builds and smoke-tests them, so they do
not rot.
* They ARE fully built and tested on the v3-4-test branch, and before
every 3.4.x release.
6. Why this is worth it - also for CI
Apart from security, the hardened base makes our CI cheaper and
smoother. Today, whenever a new Python patch release comes out or we
change system dependencies, the image cache is invalidated and every
image build has to compile Python again. In practice there are a few
days after such a change when image builds in PRs are slow, until
everyone rebases onto the latest main. With a prebuilt Python base there
is nothing to compile. In our measurements, the OS-dependencies layer of
the build stage went from ~210s to ~105s, and Python patch upgrades
become just a base-image tag bump. The image size is practically
unchanged (+0.35%).
Let me know what you think - especially about the "-legacy" naming, the
testing split, and whether 3.5.0 is the right time to drop bookworm.
If there are no strong objections, I will follow up with a [LAZY
CONSENSUS] thread.
J.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
