potiuk opened a new pull request, #75219: URL: https://github.com/apache/airflow/pull/75219
* Base CI and PROD images on hardened Python images Squashed from 19 commits (backup branch: backup/hardened-python-base-images-pre-rebase) before rebasing onto main. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Base images on trixie and keep a legacy bookworm PROD image flavor The 3.4.x line keeps publishing the previous images - Python compiled from sources on bookworm - as a deprecated legacy flavor, so the PROD image has to be buildable both ways while the CI image only needs the hardened trixie one. A weekly canary builds and checks the legacy images so they do not rot between releases. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Support the trixie hardened images' Debian-packaged Python and test legacy images like regular ones The Debian 13 hardened images install Python as Debian packages under /usr instead of /opt/python, so the image has to find the base Python's prefix and the system-Python guard must not mistake those packages for Debian's own Python. The legacy canary now runs the regular PROD image tests, not just a smoke check, so the legacy images are covered as well as the default ones. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Drop CI image packages that Debian trixie no longer has The CI image is trixie-only now: libgcc-11-dev was a bookworm-era pin (build-essential brings the matching libgcc-14-dev) and software-properties-common was removed in Debian 13; nothing uses its add-apt-repository. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Trust Microsoft's newer signing key for the trixie MSSQL repository Microsoft signs its Debian 13 package repository with its 2025 general signing key rather than the 2015 release key, so apt on trixie rejected the repository as unsigned. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Restore the PAM configs the trixie hardened images strip The Debian 13 hardened images ship libpam-runtime and passwd already installed but without their /etc/pam.d files, so installing them restored nothing and adding the CI image's airflow user failed with pam_start() errors. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Document the anonymous Debian package repository of the hardened images The trixie hardened images add Docker's own package repository, whose rebuilt packages take precedence over Debian's - users extending the image should know where their packages come from, and that no credentials are needed for it. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Keep a second hardened Python out of the trixie images On trixie, libenchant-2-dev's build dependencies need python3, which Docker's package repository satisfies with its own python-3.13 - a second interpreter in the 3.12 image that broke building C extensions. pyenchant only needs the runtime library, which it finds through the ldconfig cache, and the guard now fails early on any hardened Python other than the image's own. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Install the Python build dependencies for the legacy image flavor Dropping the packages that only existed to compile Python left the legacy flavor compiling a Python without sqlite3, lzma or readline - CPython builds those modules only when their headers are present, and the image could not even run airflow. The build now also fails when one of them is missing. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Refresh the shared library cache after apt runs in hardened images The hardened images ship libc-bin without its dpkg trigger, so libraries installed by apt stayed invisible to ctypes.util.find_library() - the CI image's docs build failed with "The 'enchant' C library was not found". A dpkg post-invoke hook runs ldconfig after every apt run, also in images extending ours. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Warn about apt pulling a second Python into hardened images Extending a hardened image with apt packages that depend on python3 silently installs another Python next to the image's own, and C extensions then build against the wrong one - users need to know to check before adding such packages. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Add trixie and sigstore to the docs spelling wordlist Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Count only host keys when waiting for sshd in the CI image OpenSSH 10 in trixie prints a '# host:port' banner comment per connection to ssh-keyscan's stdout, so the line count never matched the three host keys and every test job waited until it timed out. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Invalidate the CI image mount cache for the trixie base The uv cache restored into CI image builds held wheels compiled against bookworm's libraries - an xmlsec wheel built against bookworm's MD5-enabled libxmlsec1 failed to import on trixie, which dropped MD5. The cache key does not encode the base OS, so it has to be bumped when that changes. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Install Java that exists on trixie in the Java examples Debian trixie ships no Java 17. The Java SDK e2e image keeps Java 17 for Spark 3.5 by taking it from the Temurin repository, and the docs example uses trixie's Java 21. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Let old C extensions build in the trixie CI image GCC 14 turns incompatible pointer types, integer conversions and implicit declarations into errors, and the hardened Python passes -Werror=implicit-function-declaration on to extensions - the lowest-dependency tests' old sdists (xmlsec 1.3.14, ibm-db 3.2.0) only built because those were warnings. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Document the stricter C extension builds of the hardened images Users extending the image with old source distributions hit the same GCC 14 errors as CI, and need to know how to relax them. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Export KinD logs when the lang-SDK K8S test setup fails A failing lang-SDK helm upgrade raised past the log export that every other step of the complete K8S test run does, so the cluster was deleted with nothing saved and the job's KinD logs artifact stayed empty. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Keep apt from replacing the hardened images' Python Debian 13's python3 packages conflict with the hardened images' python-3.X packages, so installing NodeSource's nodejs (which depends on python3) into the lang-SDK runtime image made apt remove the image's Python 3.11, and the migration pods crash-looped. The image now holds its Python packages, so such an install fails instead, and the runtime image copies the node binary from the official node image rather than installing the deb. Generated-by: Claude Opus 5 * Keep perl-base in step with the perl apt can install The hardened apt repository publishes git, which depends on perl, but not perl itself, while the hardened images ship their own build of perl-base. Debian's perl depends on exactly its own perl-base version, so apt refused to install git and the hardened CI image build failed. The image now moves perl-base to the version of the installable perl before installing its packages; when the hardened repository publishes a matching perl, the versions agree and nothing changes. Generated-by: Claude Opus 5 * Mirror the hardened base images only from main The mirror is the default BASE_IMAGE of every Airflow build, and the job holds packages:write plus the Docker Hub credentials. Scheduled runs are always on main, but anyone with write access could dispatch the workflow from another branch and replace the mirrored images. Generated-by: Claude Opus 5 * Check the Adoptium signing key fingerprint in the Java e2e image The key is downloaded over HTTPS at build time; pinning its fingerprint means a replaced key on the server fails the build instead of being trusted for every package from that repository. Generated-by: Claude Opus 5 * Allow mirroring the hardened base images from v3-*-test branches The release branches build images from the same mirror, so maintainers need to refresh it from there too. (cherry picked from commit 359d361de93f8feac751788baf6934faa65a0c5c) Generated-by: Claude Opus 5 <!-- SPDX-License-Identifier: Apache-2.0 https://www.apache.org/licenses/LICENSE-2.0 --> <!-- Thank you for contributing! Please provide above a brief description of the changes made in this pull request. Write a good git commit message following this guide: https://chris.beams.io/posts/git-commit/ Please make sure that your code changes are covered with tests. And in case of new features or big changes remember to adjust the documentation. For user-facing UI changes, please attach before/after screenshots (or a short screen recording) so reviewers can assess the visual impact. Feel free to ping (in general) for the review if you do not see reaction for a few days (72 Hours is the minimum reaction time you can expect from volunteers) - we sometimes miss notifications. In case of an existing issue, reference it using one of the following: * closes: #ISSUE * related: #ISSUE --> --- ##### Was generative AI tooling used to co-author this PR? <!-- If generative AI tooling has been used in the process of authoring this PR, please change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by". --> - [ ] Yes (please specify the tool below) <!-- Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --> --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
