> I'd agree to drop support for core earlier, but for providers I think it is reasonable to further support the installed base until EOL
Ah. That also reminds me ..... that in fact we can't do it. At least currently all our packages we have in `main` have to share the same Python requirements. This is the limitation of `uv workspace` implementation (and it's actually reasonable). So the only way to make difference in supported versions for Airflow and Providers - will be to separate them to separate repositories (or have a separate branch in airflow repo for providers). But this is an entirely different discussion, and I think separating the python version support should not be a sole reason to make such a split, nor it should not be a strong argument for it. J,. On Sat, Feb 8, 2025 at 12:50 PM Jens Scheffler <j_scheff...@gmx.de.invalid> wrote: > Reading the log email of Jarek... I did not consider all the providers > we have as well. > > I'd agree to drop support for core earlier, but for providers I think it > is reasonable to further support the installed base until EOL. > > On 08.02.25 12:42, Jarek Potiuk wrote: > > I thought a bit that I think if we agree on some "earlier than end of > life" > > policy, that might be a good thing. There are a few packages that are > > dropping Python support generally 3-6 months before EOL. Notably a number > > of "development tools" that sometimes hold us back from upgrading our > > CI/dev env (can't remember exactly which ones, but I definitely recall a > > few times where we could not upgrade to the latest version of some of > > those). > > > > I think what really matters is that we have a "wide-enough" range of > > support. In a number of corporate environments, there are some old > > libraries and services that will have several years lags before they are > > supported in a newer version of Python. However, also I think (but this > is > > more of an anecdotal evidence, but something that Jen's comment also > hints > > about) - the rate of adoption of new versions of Python had significantly > > increased. There are multiple reasons for that - almost 100% backwards > > compatibility of 3.8 - 3.12 compared to relatively significant breaking > > changes in 3.5 - 3.8 played a major role. I can't remember for the last 3 > > years that we had a breaking change because of different Python versions > > (in the python language itself) - and if they were any, it was because of > > some obscure library feature that was usually very quickly fixed when > > found. > > > > If you look at the state of Python > > https://blog.jetbrains.com/pycharm/2024/12/the-state-of-python/ from > > December 2024, Python 3.10+ was already at > 75% back then. > > When you look at pypistats of Airflow > > https://pypistats.org/packages/apache-airflow it's quite a bit less: > Python > > 3.10+ is ~60% ... But.... > > > > I think supporting EOL python is very important for low level libraries > > that might be used by many in different contexts. For example I know > `pip` > > is not only looking at EOL but also at the usage stats. But In our case, > I > > think we can be more of a "driver" of a change. If Airflow 3 supports > only > > Python 3.10+, that might actually make people consider changing. I can't > > imagine many of those old libraries I mention to support 3.9 but not > > support 3.10. The backwards-compatibility problems mentioned in 3.5 - 3.8 > > could have held those libraries back to support 3.9, but if you have > > something supporting 3.9 - after initial ~6 months of catching up on all > > the libraries, moving to 3.10 should be pretty painless. > > > > Maybe...... Just maybe.... We could introduce a rule that we drop support > > for Python version 6 months before it reaches EOL ? We could also do "12 > > months" if we are a bit "aggressive". > > > > That would generally shift our dropping Python to ~ March time from ~ > > October time and that would nicely coincide with our plans to release > > Airflow 3 ? Yes. It's arbitrary, but for me 6 months is quite "enough" to > > see that "the end is coming soon" and also 6 months Is the fastest time > of > > any upgrade I can imagine in any corporate environment that has a problem > > with such an upgrade. The "6 months" and "12 months" are both a bit of a > > magical number but we already use them in some places: > > > >> The version of the base OS image is the stable version of Debian. > Airflow > > supports using all currently active stable versions - as soon as all > > Airflow dependencies support building, and we set up the CI pipeline for > > building and testing the OS version. Approximately 6 months before the > > end-of-regular support of a previous stable version of the OS, Airflow > > switches the images released to use the latest supported version of the > OS. > > > > (that will be end of of 2027 BTW) > > > >> One of the important limitations of the Providers released by the > > community is that we introduce the limit > > of a minimum supported version of Airflow. The minimum version of Airflow > > is the ``MINOR`` version (2.4, 2.5 etc.) > > indicating that the providers might use features that appeared in this > > release. The default support timespan > > for the minimum version of Airflow (there could be justified exceptions) > is > > that we increase the minimum > > Airflow version to the next MINOR release, when 12 months passed since > the > > first release for the > > MINOR version of Airflow. > > > > (April 2025 is when we drop Airflow 2.9 support in providers) > > > > 6 months is also my "rule of thumb "for when I **think** we can start > > supporting a new version of Python as well after its releases, because > that > > gives time to the vast majority of our libraries to support it. I was > about > > to create a PR with Python 3.13 support and we can also see how we fare > > there. > > > > If we could introduce such policy, I'd be warmly supporting either a 6 > > months or 12 months shift from EOL. > > J. > > > > > > > > On Fri, Feb 7, 2025 at 9:22 PM Jens Scheffler <j_scheff...@gmx.de.invalid > > > > wrote: > > > >> I am also +1 for dropping 3.9 support early. That can save a lot of > >> boilerplate for __future__ for type hints as well. > >> > >> As well as Ash sais we would most probably anyway only support 3.0 and > >> 3.1 with 3.9 > >> > >> And I can say, migrating Python from 3.8 straight into 3.12 was only > >> generating one bug in our environment, else all was un-noticed. > >> > >> On 07.02.25 13:51, Ash Berlin-Taylor wrote: > >>> https://endoflife.date/python for a visual of the versions and their > >> support status for anyone following along. > >>> We do have a published get-out-of-jail card: > >>> > >>>> This policy is best-effort which means there may be situations where > we > >> might terminate support earlier if circumstances require it. > >>> But yes, it shouldn’t be willy-nilly. The only sort of reason/rule I > can > >> think of would be “At major version releases, we will support any Python > >> with more than 2 years left of security support” but that feels very > >> arbitrary. > >>> The module I want to use is https://docs.cadwyn.dev/ (which I’ve used > >> on other non OSS projects), but thinking about it a bit more, we are > likely > >> only going to have one or two versions of the Task Execution API (one > with > >> 3.0, and one with 3.1), maybe a couple of more in some patch releases > but > >> that is unlikely and I can manually emulate Cadwyn for now and swap > over to > >> it around the 3.2/3.3 timeframe. > >>> > >>> > >>> > >>>> On 7 Feb 2025, at 12:30, Jarek Potiuk <ja...@potiuk.com> wrote: > >>>> > >>>> I would be for it - but this should be accompanied with a clear > >> proposal of > >>>> the policy we are going to use forward for Airflow 3. We cannot make > >> such > >>>> "ad-hoc" decisions based on "I want to use that package". We need to > >> have > >>>> solid reasoning and clear indication for our users what kind of > support > >>>> they can expect for different python versions. They cannot be > surprised. > >>>> So far our policy was clear - we drop support when Python EOL is > >> reached. > >>>> That was easy to explain and justify. I don't have a concrete > proposal, > >> but > >>>> maybe someone can come up with a rule that we codify and use from now > >> on. > >>>> J. > >>>> > >>>> On Fri, Feb 7, 2025 at 1:25 PM Ash Berlin-Taylor <a...@apache.org> > >> wrote: > >>>>> Hi all, > >>>>> > >>>>> I have a proposal that we increase the minimum required python > version > >> to > >>>>> 3.10, in a slight departure from our published python version req > >>>>> > >> > https://github.com/apache/airflow?tab=readme-ov-file#support-for-python-and-kubernetes-versions > >>>>> As a reminder, Python 3.9 is already in security only fixes, and is > >>>>> supported only until October 2025, or for roughly 7 months after the > >>>>> release of Airflow 3.0. We’d be bringing forward the requirement on > >> Python > >>>>> 3.10 from (likely) Airflow 3.2 to 3.0. > >>>>> > >>>>> We discussed this briefly last July on the Airflow 3 dev calls and > the > >>>>> conclusion then was to follow the our policy, but I would like to > make > >> use > >>>>> of a python module in the Task Execution API server that only > supports > >>>>> Python 3.10+. > >>>>> > >>>>> Pros of this: We’re more up to date, and I can use this module, > rather > >>>>> than having to write something myself to handle API versioning > >>>>> Cons of this: it might make it harder for some users to update to 3.0 > >> as > >>>>> some users would also have to update the Python version before they > can > >>>>> update the Airflow version. > >>>>> > >>>>> Looking at the analytics data we had previously collected (which only > >> gets > >>>>> data from 2.10.x so is imperfect to be sure), I can see that Python > >> 3.10+ > >>>>> accounts for about 77% of the data. So it’s certainly not nothing. > (5% > >> are > >>>>> still on 3.8, and 17% are on 3.9) > >>>>> > >>>>> I’m really not sold one way or another on this, so I thought I’d > >> discuss > >>>>> it with the wider community. > >>>>> > >>>>> What do you all think? > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > >> For additional commands, e-mail: dev-h...@airflow.apache.org > >> > >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
