Yep. And merging the instalki script ci/prod :) pt., 25 lip 2025, 15:13 użytkownik Aritra Basu <aritrabasu1...@gmail.com> napisał:
> Hey Jarek, > > I haven't seen any suspicious side effects either. I can go ahead and start > working on a pr later today, if nothing breaks I should be able to raise it > by tonight. I think it should be a fairly small change (🤞) just moving the > python install directory within the docker file hopefully. > > -- > Regards, > Aritra Basu > > On Fri, 25 Jul 2025, 1:39 pm Jarek Potiuk, <ja...@potiuk.com> wrote: > > > Did anyone experience any suspicious problems with the breeze CI image > > prepared this way ? > > > > I did not - python works as usual, all airflow features are working, I > have > > not seen any "negative" side effect of changing python to source-built > one > > - except slightly longer build time occasionally. > > > > If ther are no other observations or issues - Aritra, I tink it might be > > good time to look at trasplanting it to PROD image, Happy to either just > > help and review or do it and have you review it - up to you :) > > > > J. > > > > On Fri, Jul 11, 2025 at 6:54 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > And now I have to rebase my Python 3.13 PR to see if 3.13 also compiles > > > well :D > > > > > > On Fri, Jul 11, 2025 at 6:53 PM Aritra Basu <aritrabasu1...@gmail.com> > > > wrote: > > > > > >> Yay! 🙌 > > >> -- > > >> Regards, > > >> Aritra Basu > > >> > > >> On Fri, 11 Jul 2025, 9:48 pm Jarek Potiuk, <ja...@potiuk.com> wrote: > > >> > > >> > Ok. We got it merged again ! > > >> https://github.com/apache/airflow/pull/53150 > > >> > is merged. Once you build breeze images after rebase, python there > > will > > >> be > > >> > built directly from Python official sources (latest released > > patchlevel > > >> for > > >> > each version - and we have automation to upgrade them soon after new > > >> > patchlevel are released). > > >> > > > >> > We are going to try it for a week or so and see if it all looks > good, > > we > > >> > will proceed to apply it to PROD images. > > >> > > > >> > J. > > >> > > > >> > > > >> > On Sat, Jul 5, 2025 at 10:24 AM Jarek Potiuk <ja...@potiuk.com> > > wrote: > > >> > > > >> > > I merget it too early - reverting here > > >> > > https://github.com/apache/airflow/pull/52900 as I found an issue > > with > > >> > > other python versions that I overlooked > > >> > > > > >> > > On Sat, Jul 5, 2025 at 10:06 AM Jarek Potiuk <ja...@potiuk.com> > > >> wrote: > > >> > > > > >> > >> FYI - we just merged Aritra's change to build Python in CI image > > from > > >> > >> sources https://github.com/apache/airflow/pull/52265 -> we will > > run > > >> it > > >> > >> for a while in CI, i also want to test some self-upgrade > scenarios. > > >> > >> > > >> > >> Then we will look at applying it to the PROD image if we find no > > >> > >> surprises. > > >> > >> > > >> > >> It looks really good and we might have slightly more secure > images > > >> > >> produced (i.e. getting read of some libraries CVEs faster than > the > > >> > >> "official" images. > > >> > >> > > >> > >> Good job Aritra! > > >> > >> J. > > >> > >> > > >> > >> > > >> > >> On Thu, Jun 26, 2025 at 9:16 PM Jarek Potiuk <ja...@potiuk.com> > > >> wrote: > > >> > >> > > >> > >>> It looks like building from sources on our CI using base debian > > >> image > > >> > >>> (with enabled optimizations) "just works" and takes 1m36s. -> > and > > >> > taking > > >> > >>> into account that we already have pretty sophisticated CI-level > > >> remote > > >> > >>> caching that will allow to rebuild it only when either base > image > > is > > >> > >>> released or python is released - building sounds like a very > good > > >> idea. > > >> > >>> > > >> > >>> J. > > >> > >>> > > >> > >>> On Thu, Jun 26, 2025 at 7:02 PM Damian Shaw < > > >> > >>> ds...@striketechnologies.com> wrote: > > >> > >>> > > >> > >>>> Going from source is certainly going to give the most control, > I > > >> > >>>> personally choose python-build-standalone for my own images > > rather > > >> > than > > >> > >>>> building from source to outsource all the build choices, such > as > > >> what > > >> > >>>> optimization flags to enable, what compiler tool chains to use, > > >> etc. > > >> > >>>> > > >> > >>>> Damian > > >> > >>>> > > >> > >>>> > > >> > >>>> > > >> > >>>> -----Original Message----- > > >> > >>>> From: Jarek Potiuk <ja...@potiuk.com> > > >> > >>>> Sent: Thursday, June 26, 2025 10:15 AM > > >> > >>>> To: dev@airflow.apache.org > > >> > >>>> Subject: Re: [DISCUSS] Switching base Python container images > to > > >> > >>>> python-standalone ? > > >> > >>>> > > >> > >>>> Thanks Damian - very insightful. And yes - I did not really > want > > to > > >> > >>>> "diminish" the value of community images and work of the > > >> maintainers, > > >> > it > > >> > >>>> was really more on the "we base our image security on the > > >> assumptions > > >> > that > > >> > >>>> it's coming from "official" sources and surely there are some > > >> > guardrails" - > > >> > >>>> which turned out to be not really true. > > >> > >>>> > > >> > >>>> But.... We might not have to even use python standalone. > > >> > >>>> > > >> > >>>> Aritra just tested installation of Python straight from sources > > -> > > >> > >>>> https://github.com/apache/airflow/pull/52265 and I was > actually > > >> > pretty > > >> > >>>> surprised how non-problematic and fast it was - and it seems it > > >> > passes all > > >> > >>>> our CI tests. We just need to add a little pre-commit magic to > > get > > >> > >>>> notified when we should update patchlevel version of Python > when > > a > > >> > new one > > >> > >>>> is released, and we should be able to try it out in CI image - > > once > > >> > it gets > > >> > >>>> battle-tested with CI/breeze etc. we can transfer this to PROD > > >> image > > >> > as > > >> > >>>> well. We already have an idea how to do it - our PROD images > are > > >> > optimized > > >> > >>>> for size and do not contain "build essentials" - but I think we > > >> > should be > > >> > >>>> able to build Python in the "build" segment and simply copy > > >> resulting > > >> > >>>> binaries to the "main" segment - since in both cases we use the > > >> same > > >> > base > > >> > >>>> image, such 1-1 copy should **just work** - we already do the > > same > > >> > with > > >> > >>>> installed python packages - we install them (including building > > >> when > > >> > >>>> needed) in build segment and we copy-over the installed .venv > to > > >> the > > >> > >>>> main segment. > > >> > >>>> > > >> > >>>> So ... we might even not need a discussion - installing from > > Python > > >> > >>>> sources is THE BEST > > >> > >>>> > > >> > >>>> J. > > >> > >>>> > > >> > >>>> > > >> > >>>> On Wed, Jun 25, 2025 at 6:25 PM Damian Shaw < > > >> > >>>> ds...@striketechnologies.com> > > >> > >>>> wrote: > > >> > >>>> > > >> > >>>> > First, I would like to thank the community members who have > > been > > >> > >>>> > maintaining the Python docker images, it's one of those > > thankless > > >> > >>>> > opensource infrastructure volunteer roles that they've been > > doing > > >> > for > > >> > >>>> > a long time. Unfortunately Docker assigns the title "Official > > >> Image" > > >> > >>>> > for various community run images, which creates a > misconception > > >> on > > >> > the > > >> > >>>> > guarantees being provided, and if I were a suspicious person > I > > >> would > > >> > >>>> > say Docker creates this misconception on purpose to both get > > free > > >> > work > > >> > >>>> > from community members and make Docker seem more supported by > > >> third > > >> > >>>> > party organizations than it actually is. > > >> > >>>> > > > >> > >>>> > On the topic of python-build-standalone, I've been using it > in > > >> > >>>> > production for several months now and I'm fairly happy with > it. > > >> > >>>> > > > >> > >>>> > However, one minor reproducibility issue I have when > installing > > >> > >>>> > python-build-standalone via uv, is that uv does not have an > > >> ability > > >> > to > > >> > >>>> > pin to a specific build between uv versions, as uv hard > codes a > > >> > >>>> > mapping of Python version and platform to a specific build > and > > >> then > > >> > >>>> > updates that mapping between releases. So, updating the > version > > >> of > > >> > uv > > >> > >>>> > between runs, or having two users run different versions of > uv > > to > > >> > >>>> > initialize the environment can change the results. > > >> > >>>> > > > >> > >>>> > While normally such build changes are trivial, if you look at > > >> the uv > > >> > >>>> > 0.7.8 and 0.7.9 changelogs you will see that sometimes they > can > > >> have > > >> > >>>> > significant > > >> > >>>> > impact: https://github.com/astral-sh/uv/releases/tag/0.7.8. > > >> Also, > > >> > >>>> this > > >> > >>>> > email finally prompted make an issue on this topic: > > >> > >>>> > https://github.com/astral-sh/uv/issues/14263. > > >> > >>>> > > > >> > >>>> > There are other ways to source python-build-standalone, such > as > > >> > >>>> > pbs-installer or writing your own script, but I've not yet > > spent > > >> any > > >> > >>>> > time investigating them, so I can't comment on them. > > >> > >>>> > > > >> > >>>> > Damian > > >> > >>>> > > > >> > >>>> > -----Original Message----- > > >> > >>>> > From: Jarek Potiuk <ja...@potiuk.com> > > >> > >>>> > Sent: Wednesday, June 25, 2025 10:44 AM > > >> > >>>> > To: dev@airflow.apache.org > > >> > >>>> > Subject: [DISCUSS] Switching base Python container images to > > >> > >>>> > python-standalone ? > > >> > >>>> > > > >> > >>>> > Hello here, > > >> > >>>> > > > >> > >>>> > Together with Aritra we are looking into adding a few more > > >> things to > > >> > >>>> > our images (golang tool chain for CI image), also Shahar is > > >> > >>>> > experimenting with Rust tool chain and I also recently > realized > > >> (by > > >> > >>>> > some of the issues we had) that 'Docker Official Python > Image' > > >> that > > >> > is > > >> > >>>> > part of 'Official Program' [1] is not as 'Official' as I > > thought > > >> so > > >> > we > > >> > >>>> > discuss about changing the base of our images (first CI and > > then > > >> > when > > >> > >>>> > we see it works fine - PROD) > > >> > >>>> > > > >> > >>>> > Currently we are using the 'Official' image - but after some > > >> issues > > >> > >>>> > and discussions with people at PyCon and FOSS Backstage (I > had > > a > > >> > >>>> > chance to talk to Python maintainers and even had a few beers > > >> with > > >> > >>>> > them) - it turned out that the official Python Image' is > > >> maintained > > >> > by > > >> > >>>> > 'a community's which really is a few pretty random people - > and > > >> that > > >> > >>>> > explains for example why we have sometimes unpacked security > > >> > >>>> > vulnerabilities in setuptools etc. - because they made some > > >> > >>>> > compatibility choices and decisions that do not allow them to > > >> > upgrade > > >> > >>>> > easily, also they had some delays in releasing updated Python > > >> > >>>> > versions. And Docker does not **really** do much vetting > there. > > >> > >>>> > > > >> > >>>> > So I think it would be good to switch how we build the base > for > > >> our > > >> > >>>> images. > > >> > >>>> > And following the experience of `uv python` [2] - it seems > that > > >> > maybe > > >> > >>>> > using "python-standalone" [3] project is a good alternative. > > It's > > >> > >>>> > managed by Astral now (so yes - another dependency on them), > > but > > >> > what > > >> > >>>> > you have with it you have practically 100% complete Python > > >> > interpreter > > >> > >>>> > installed in seconds. > > >> > >>>> > We could continue using debian-slim as a "base, base image" - > > and > > >> > >>>> > install python using "python-standalone". There are a few > > >> > >>>> > incompatibilities [4] of the distributions of Python, but > there > > >> are > > >> > >>>> > very few and mostly related to some obscure systems > > >> (compatibilities > > >> > >>>> > with terminal in REPL, and gtk / UI integration that is > anyhow > > >> not > > >> > >>>> > really working in "standard" Python distributions). > > >> > >>>> > > > >> > >>>> > I would love to hear what you think - happy to get any > > feedback/ > > >> > >>>> > insights, suggestions and answer additional questions, > provide > > >> some > > >> > >>>> > links to past "troubles" we had with Python "Official" images > > >> etc. > > >> > >>>> > > > >> > >>>> > J. > > >> > >>>> > > > >> > >>>> > > > >> > >>>> > [1] Official Python Images - https://hub.docker.com/_/python > > >> [2] UV > > >> > >>>> > Python installation - > > >> > >>>> https://docs.astral.sh/uv/guides/install-python/ > > >> > >>>> > [3] Python Standalone project - > > >> > >>>> > https://github.com/astral-sh/python-build-standalone > > >> > >>>> > [4] Python Standalone incompatibilities - > > >> > >>>> > > > >> > >>>> > > >> > > > https://gregoryszorc.com/docs/python-build-standalone/main/quirks.html > > >> > >>>> > ________________________________ > > >> > >>>> > Strike Technologies, LLC (“Strike”) is part of the GTS > family > > of > > >> > >>>> > companies. Strike is a technology solutions provider, and is > > not > > >> a > > >> > >>>> > broker or dealer and does not transact any securities related > > >> > business > > >> > >>>> > directly whatsoever. This communication is the property of > > Strike > > >> > and > > >> > >>>> > its affiliates, and does not constitute an offer to sell or > the > > >> > >>>> > solicitation of an offer to buy any security in any > > >> jurisdiction. It > > >> > >>>> > is intended only for the person to whom it is addressed and > may > > >> > >>>> > contain information that is privileged, confidential, or > > >> otherwise > > >> > >>>> protected from disclosure. > > >> > >>>> > Distribution or copying of this communication, or the > > information > > >> > >>>> > contained herein, by anyone other than the intended recipient > > is > > >> > >>>> > prohibited. If you have received this communication in error, > > >> please > > >> > >>>> > immediately notify Strike at i...@striketechnologies.com, > and > > >> > delete > > >> > >>>> and destroy any copies hereof. > > >> > >>>> > ________________________________ > > >> > >>>> > > > >> > >>>> > CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any > > >> > >>>> > attachments are intended solely for the addressee. This > > >> transmission > > >> > >>>> > is covered by the Electronic Communications Privacy Act, 18 > > U.S.C > > >> > >>>> > ''2510-2521. The information contained in this transmission > is > > >> > >>>> > confidential in nature and protected from further use or > > >> disclosure > > >> > >>>> > under U.S. Pub. L. 106-102, 113 U.S. Stat. 1338 (1999), and > may > > >> be > > >> > >>>> > subject to attorney-client or other legal privilege. Your use > > or > > >> > >>>> > disclosure of this information for any purpose other than > that > > >> > >>>> > intended by its transmittal is strictly prohibited, and may > > >> subject > > >> > >>>> > you to fines and/or penalties under federal and state law. If > > you > > >> > are > > >> > >>>> > not the intended recipient of this transmission, please > DESTROY > > >> ALL > > >> > >>>> > COPIES RECEIVED and confirm destruction to the sender via > > return > > >> > >>>> transmittal. > > >> > >>>> > > > >> > >>>> ________________________________ > > >> > >>>> Strike Technologies, LLC (“Strike”) is part of the GTS family > of > > >> > >>>> companies. Strike is a technology solutions provider, and is > not > > a > > >> > broker > > >> > >>>> or dealer and does not transact any securities related business > > >> > directly > > >> > >>>> whatsoever. This communication is the property of Strike and > its > > >> > >>>> affiliates, and does not constitute an offer to sell or the > > >> > solicitation of > > >> > >>>> an offer to buy any security in any jurisdiction. It is > intended > > >> only > > >> > for > > >> > >>>> the person to whom it is addressed and may contain information > > >> that is > > >> > >>>> privileged, confidential, or otherwise protected from > disclosure. > > >> > >>>> Distribution or copying of this communication, or the > information > > >> > contained > > >> > >>>> herein, by anyone other than the intended recipient is > > prohibited. > > >> If > > >> > you > > >> > >>>> have received this communication in error, please immediately > > >> notify > > >> > Strike > > >> > >>>> at i...@striketechnologies.com, and delete and destroy any > > copies > > >> > >>>> hereof. > > >> > >>>> ________________________________ > > >> > >>>> > > >> > >>>> CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any > > >> > >>>> attachments are intended solely for the addressee. This > > >> transmission > > >> > is > > >> > >>>> covered by the Electronic Communications Privacy Act, 18 U.S.C > > >> > ''2510-2521. > > >> > >>>> The information contained in this transmission is confidential > in > > >> > nature > > >> > >>>> and protected from further use or disclosure under U.S. Pub. L. > > >> > 106-102, > > >> > >>>> 113 U.S. Stat. 1338 (1999), and may be subject to > attorney-client > > >> or > > >> > other > > >> > >>>> legal privilege. Your use or disclosure of this information for > > any > > >> > purpose > > >> > >>>> other than that intended by its transmittal is strictly > > prohibited, > > >> > and may > > >> > >>>> subject you to fines and/or penalties under federal and state > > law. > > >> If > > >> > you > > >> > >>>> are not the intended recipient of this transmission, please > > DESTROY > > >> > ALL > > >> > >>>> COPIES RECEIVED and confirm destruction to the sender via > return > > >> > >>>> transmittal. > > >> > >>>> > > >> > >>>> > > >> --------------------------------------------------------------------- > > >> > >>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > >> > >>>> For additional commands, e-mail: dev-h...@airflow.apache.org > > >> > >>>> > > >> > >>> > > >> > > > >> > > > > > >
