Personally for me it is controversial change and tradeoff between Stability vs Performance
Since Airflow + Providers have 400+ dependencies, using the lowest version of python provides better stability and the reason for this is pretty simple - time spent for maintainers of packages to make it more stable on older Python versions rather than new. That is funny because it is difficult to name 3.11 new one because it was released more than one year ago. However some of core dependencies of Airflow do not updated for a long time and have "formal" support of Python 3.11 Known Issue in Airflow and Python 3.11 is dill + pendulum, see: https://github.com/apache/airflow/issues/35307 . It not affect all users, so maybe we could resolve it by a create a "Known Incompatibilities" page in Airflow documentation However, using tags without an explicit python version is a very nice way for users to shoot in the foot or cast wormhole (depending on your preference), especially if we talk about apache/airflow:latest. So in this case I would rather say it doesn't matter which Python version we would use in this case, maybe better even chouse "golden mean" and use Python 3.10 but this also controversial because it required to decide when we need to shift this selection in this case select between lowest or highest version it is more straight forward rather than select "golden mean" Performance and environment part is also important however we need to take in account that Airflow is an application which uses DB backend very intensively and this is a point where total performance advantages of changing the Python version are dramatically reduced. As outcome I do not have any objection to this potential changes because there is no any difference between each of strategy of "default" python selection at all of them have pros and cons due to my personal opinion that use Airflow Image without pin python version make more problem rather give any advantages and you might have a nice time to debug at the moment when "default" version changed and it would change in any cases ---- Best Wishes *Andrey Anshin* On Fri, 17 Nov 2023 at 10:10, Wei Lee <weilee...@gmail.com> wrote: > Agreed, as long as users can still use different versions through tags, > there are no drawbacks or incompatibilities with this great idea! > > Best, > Wei > > > On Nov 17, 2023, at 1:39 PM, Aritra Basu <aritrabasu1...@gmail.com> > wrote: > > > > Agreed, moving to latest by default sounds like a fine idea. I don't see > > any drawbacks to it and seems like a good enough time as any to make the > > switch with 2.8.0. > > > > -- > > Regards, > > Aritra Basu > > > > On Fri, Nov 17, 2023, 12:33 AM Vincent Beck <vincb...@apache.org> wrote: > > > >> I agree, by default we should use the latest python version. Like any > >> package manager, if the user does not explicitly specify a version, the > >> latest should be used. If the user wants to use a lower version, he can > >> always pin it. > >> > >> On 2023/11/16 12:06:17 Jarek Potiuk wrote: > >>> Hello everyone, > >>> > >>> Since we are close to the Airflow 2.8.0 release, I would like to > propose > >> a > >>> change in the approach for our "default" images. > >>> > >>> Currently there are few images that are considered as "default", for > >>> example: > >>> > >>> apache/airflow:latest > >>> apache/airflow:2.7.4 > >>> > >>> Currently (according to our process [1] and user documentation [2]) > those > >>> point to the "oldest" python version we support (currently they point > to > >>> Python 3.8). > >>> > >>> There is no particular reason why it is like that, and with Airflow > 2.8.0 > >>> we have an opportunity to change it and point the default images to > >> "latest > >>> supported" (and keep this version as default for the whole MINOR line > of > >>> releases. > >>> > >>> In the case of Airflow 2.8.* - that would be "Python 3.11" being > default > >>> for the whole 2.8.* line unless we manage to get Python 3.12 support in > >> our > >>> CI before we release Airflow 2.8.0, then it would be Python 3.12 > >>> > >>> We do not have any SemVer promises about that. Users can still choose > to > >>> use the "2.8.0-python3.8" tag if they want. > >>> > >>> Generally going to 2.8 should always be a deliberate action, so we have > >>> chance to explain in the release notes that if they want to stick to > the > >>> 2.8 release. So they are not "losing" anything, they can have 100% > >>> compatibility by just choosing a different image in their deployment. > >> This > >>> **might** cause a little hassle when they migrate if they find some > >>> incompatibilities, but generally speaking it's a very straightforward > and > >>> simple change - just adding "-python3.8" to your TAG - whatever > >> deployment > >>> option you have. And our users will have to go through it anyway every > >> time > >>> we drop the old Python version (and this change might be even more > costly > >>> as they have no choice then) - so it changes very little, just shifts > the > >>> time where they will have to do it. > >>> > >>> There are benefits of doing it - for both our users and well, > environment > >>> as well (and I really mean a positive impact on the "world environment" > >> to > >>> be honest. Maybe a little impact - but with Airflow's popularity, it > >> might > >>> make a (small) difference. Python 3.11 is generally 30% faster than > >>> previous versions and using it by default means that 30% less CPU is > >> being > >>> wasted. Also it will mean actual money savings for our users. Also > Python > >>> 3.12 comes with even more performance improvements and keeping up with > >>> those being the "default" is a pretty good idea. > >>> > >>> I cannot think of any other drawbacks of this change. > >>> > >>> WDYT? > >>> > >>> [1] Documented versioning approach: > >>> > >> > https://github.com/apache/airflow#base-os-support-for-reference-airflow-images > >>> [2] User documentation > >>> https://airflow.apache.org/docs/docker-stack/index.html > >>> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > >> For additional commands, e-mail: dev-h...@airflow.apache.org > >> > >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
