It looks like building from sources on our CI using base debian image (with
enabled optimizations) "just works" and takes 1m36s. -> and taking into
account that we already have pretty sophisticated CI-level remote caching
that will allow to rebuild it only when either base image is released or
python is released - building sounds like a very good idea.

J.

On Thu, Jun 26, 2025 at 7:02 PM Damian Shaw <ds...@striketechnologies.com>
wrote:

> Going from source is certainly going to give the most control, I
> personally choose python-build-standalone for my own images rather than
> building from source to outsource all the build choices, such as what
> optimization flags to enable, what compiler tool chains to use, etc.
>
> Damian
>
>
>
> -----Original Message-----
> From: Jarek Potiuk <ja...@potiuk.com>
> Sent: Thursday, June 26, 2025 10:15 AM
> To: dev@airflow.apache.org
> Subject: Re: [DISCUSS] Switching base Python container images to
> python-standalone ?
>
> Thanks Damian - very insightful. And yes - I did not really want to
> "diminish" the value of community images and work of the maintainers, it
> was really more on the "we base our image security on the assumptions that
> it's coming from "official" sources and surely there are some guardrails" -
> which turned out to be not really true.
>
> But.... We might not have to even use python standalone.
>
> Aritra just tested installation of Python straight from sources ->
> https://github.com/apache/airflow/pull/52265 and I was actually pretty
> surprised how non-problematic and fast it was - and it seems it passes all
> our CI tests.  We just need to add a little pre-commit magic to get
> notified when we should update patchlevel version of Python when a new one
> is released, and we should be able to try it out in CI image - once it gets
> battle-tested with CI/breeze etc. we can transfer this to PROD image as
> well. We already have an idea how to do it - our PROD images are optimized
> for size and do not contain "build essentials" - but I think we should be
> able to build Python in the "build" segment and simply copy resulting
> binaries to the "main" segment - since in both cases we use the same base
> image, such 1-1 copy should **just work** - we already do the same with
> installed python packages - we install them (including building when
> needed) in build segment and we copy-over the installed .venv to the main
> segment.
>
> So ... we might even not need a discussion - installing from Python
> sources is THE BEST
>
> J.
>
>
> On Wed, Jun 25, 2025 at 6:25 PM Damian Shaw <ds...@striketechnologies.com>
> wrote:
>
> > First, I would like to thank the community members who have been
> > maintaining the Python docker images, it's one of those thankless
> > opensource infrastructure volunteer roles that they've been doing for
> > a long time. Unfortunately Docker assigns the title "Official Image"
> > for various community run images, which creates a misconception on the
> > guarantees being provided, and if I were a suspicious person I would
> > say Docker creates this misconception on purpose to both get free work
> > from community members and make Docker seem more supported by third
> > party organizations than it actually is.
> >
> > On the topic of python-build-standalone, I've been using it in
> > production for several months now and I'm fairly happy with it.
> >
> > However, one minor reproducibility issue I have when installing
> > python-build-standalone via uv, is that uv does not have an ability to
> > pin to a specific build between uv versions, as uv hard codes a
> > mapping of Python version and platform to a specific build and then
> > updates that mapping between releases. So, updating the version of uv
> > between runs, or having two users run different versions of uv to
> > initialize the environment can change the results.
> >
> > While normally such build changes are trivial, if you look at the uv
> > 0.7.8 and 0.7.9 changelogs you will see that sometimes they can have
> > significant
> > impact: https://github.com/astral-sh/uv/releases/tag/0.7.8. Also, this
> > email finally prompted make an issue on this topic:
> > https://github.com/astral-sh/uv/issues/14263.
> >
> > There are other ways to source python-build-standalone, such as
> > pbs-installer or writing your own script, but I've not yet spent any
> > time investigating them, so I can't comment on them.
> >
> > Damian
> >
> > -----Original Message-----
> > From: Jarek Potiuk <ja...@potiuk.com>
> > Sent: Wednesday, June 25, 2025 10:44 AM
> > To: dev@airflow.apache.org
> > Subject: [DISCUSS] Switching base Python container images to
> > python-standalone ?
> >
> > Hello here,
> >
> > Together with Aritra we are looking into adding a few more things to
> > our images (golang tool chain for CI image), also Shahar is
> > experimenting with Rust tool chain and I also recently realized (by
> > some of the issues we had) that 'Docker Official Python Image' that is
> > part of 'Official Program' [1] is not as 'Official' as I thought so we
> > discuss about changing the base of our images (first CI and then when
> > we see it works fine - PROD)
> >
> > Currently we are using the 'Official' image - but after some issues
> > and discussions with people at PyCon and FOSS Backstage (I had a
> > chance to talk to Python maintainers and even had a few beers with
> > them) - it turned out that the official Python Image' is maintained by
> > 'a community's which really is a few pretty random people - and that
> > explains for example why we have sometimes unpacked security
> > vulnerabilities in setuptools etc. - because they made some
> > compatibility choices and decisions that do not allow them to upgrade
> > easily, also they had some delays in releasing updated Python
> > versions. And Docker does not **really** do much vetting there.
> >
> > So I think it would be good to switch how we build the base for our
> images.
> > And following the experience of `uv python` [2] - it seems that maybe
> > using "python-standalone" [3] project is a good alternative. It's
> > managed by Astral now (so yes - another dependency on them), but what
> > you have with it you have practically 100% complete Python interpreter
> > installed in seconds.
> > We could continue using debian-slim as a "base, base image" - and
> > install python using "python-standalone". There are a few
> > incompatibilities [4] of the distributions of Python, but there are
> > very few and mostly related to some obscure systems (compatibilities
> > with terminal in REPL, and gtk / UI integration that is anyhow not
> > really working in "standard" Python distributions).
> >
> > I would love to hear what you think - happy to get any feedback/
> > insights, suggestions and answer additional questions, provide some
> > links to past "troubles" we had with Python "Official" images etc.
> >
> > J.
> >
> >
> > [1] Official Python Images - https://hub.docker.com/_/python [2] UV
> > Python installation - https://docs.astral.sh/uv/guides/install-python/
> > [3] Python Standalone project -
> > https://github.com/astral-sh/python-build-standalone
> > [4] Python Standalone incompatibilities -
> > https://gregoryszorc.com/docs/python-build-standalone/main/quirks.html
> > ________________________________
> >  Strike Technologies, LLC (“Strike”) is part of the GTS family of
> > companies. Strike is a technology solutions provider, and is not a
> > broker or dealer and does not transact any securities related business
> > directly whatsoever. This communication is the property of Strike and
> > its affiliates, and does not constitute an offer to sell or the
> > solicitation of an offer to buy any security in any jurisdiction. It
> > is intended only for the person to whom it is addressed and may
> > contain information that is privileged, confidential, or otherwise
> protected from disclosure.
> > Distribution or copying of this communication, or the information
> > contained herein, by anyone other than the intended recipient is
> > prohibited. If you have received this communication in error, please
> > immediately notify Strike at i...@striketechnologies.com, and delete
> and destroy any copies hereof.
> > ________________________________
> >
> > CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any
> > attachments are intended solely for the addressee. This transmission
> > is covered by the Electronic Communications Privacy Act, 18 U.S.C
> > ''2510-2521. The information contained in this transmission is
> > confidential in nature and protected from further use or disclosure
> > under U.S. Pub. L. 106-102, 113 U.S. Stat. 1338 (1999), and may be
> > subject to attorney-client or other legal privilege. Your use or
> > disclosure of this information for any purpose other than that
> > intended by its transmittal is strictly prohibited, and may subject
> > you to fines and/or penalties under federal and state law. If you are
> > not the intended recipient of this transmission, please DESTROY ALL
> > COPIES RECEIVED and confirm destruction to the sender via return
> transmittal.
> >
> ________________________________
>  Strike Technologies, LLC (“Strike”) is part of the GTS family of
> companies. Strike is a technology solutions provider, and is not a broker
> or dealer and does not transact any securities related business directly
> whatsoever. This communication is the property of Strike and its
> affiliates, and does not constitute an offer to sell or the solicitation of
> an offer to buy any security in any jurisdiction. It is intended only for
> the person to whom it is addressed and may contain information that is
> privileged, confidential, or otherwise protected from disclosure.
> Distribution or copying of this communication, or the information contained
> herein, by anyone other than the intended recipient is prohibited. If you
> have received this communication in error, please immediately notify Strike
> at i...@striketechnologies.com, and delete and destroy any copies hereof.
> ________________________________
>
> CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any attachments
> are intended solely for the addressee. This transmission is covered by the
> Electronic Communications Privacy Act, 18 U.S.C ''2510-2521. The
> information contained in this transmission is confidential in nature and
> protected from further use or disclosure under U.S. Pub. L. 106-102, 113
> U.S. Stat. 1338 (1999), and may be subject to attorney-client or other
> legal privilege. Your use or disclosure of this information for any purpose
> other than that intended by its transmittal is strictly prohibited, and may
> subject you to fines and/or penalties under federal and state law. If you
> are not the intended recipient of this transmission, please DESTROY ALL
> COPIES RECEIVED and confirm destruction to the sender via return
> transmittal.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
> For additional commands, e-mail: dev-h...@airflow.apache.org
>

Reply via email to