> I agree, has there been a new release of Python that has worked with Airflow without at least some fixing?
Generally yes. Most of the past bumps were mostly test fixing. I can't recall any "serious" changes in the code. Maybe in some single providers. Generally Python 3.7 -> 3.11 were extremely backwards compatible, there were really maybe 2 cases I remember where we had compatibility issues in airflow "core" and they were because of security fixes (some edge cases of url parsing changed Is one that I remember) - and that was even in patchlevel releases - not even in the minor release. > I understand that Airflow is both a library and an application and I do agree with not putting upper bounds on libraries unless required, but it seems like allowing an arbitrary upper bound of Python is never going to be practical. If users really need to break this support version and install an older version of Airflow with a newer version of Python it isn't difficult to build Airflow locally with patches, and they will likely need to add more patches to get Airflow to work anyway, so their cost is already built in. To be honest, It's been practical for now, this is the first time it has happened for quite a while. But it seems to be changing now. Also I think (by listening to some podcast) - Python community starts to be overwhelmed with keeping all the backwards-compatibility and they are changing their approach for future releases to be more aggressive in removals, separate out certain "core" features to outside-libraries and removing them out of the core Python (which is actually quite an inspiration for us in what we do by moving stuff - like executors, hopefully FAB in the future and maybe few others from core to providers), so we can expect more of such breakages in the moment. Having explicit upper-binding will be a good idea. I think we can do a hybrid solution. We can upper-bind for 2.8 but when we release 2.7.3 we can implement raising error if someone wants to use Python 3.12. I think that will be the best approach. J. On Mon, Oct 23, 2023 at 6:33 PM Damian Shaw <ds...@striketechnologies.com> wrote: > I agree, has there been a new release of Python that has worked with > Airflow without at least some fixing? > > I understand that Airflow is both a library and an application and I do > agree with not putting upper bounds on libraries unless required, but it > seems like allows an arbitrary upper bound of Python is never going to be > practical. If users really need to break this support version and install > an older version of Airflow with a newer version of Python it isn't > difficult to build Airflow locally with patches, and they will likely need > to add more patches to get Airflow to work anyway, so their cost is already > built in. > > Even if just limiting to 3.11 or lower for 2.7.3 or higher I think the > benefits are there, Python 3.12 might be supported soon or there might be > more hidden issues that mean it isn't supported for 9+ months. > > Damian > > -----Original Message----- > From: Pierre Jeambrun <pierrejb...@gmail.com> > Sent: Monday, October 23, 2023 12:13 PM > To: dev@airflow.apache.org > Subject: Re: Limiting (or errorring out) Airflow for Python 3.12 until our > dependencies/we catch up > > I think that limiting to <3.12 makes sense. 2.7.2 is already out so I'm > not sure we can do anything for users trying to install 2.7.2 on python > 3.12. > > I believe there is no such thing as a python minor that is out of the box > working well for airflow. It seems that we always need extra efforts to > bring in a new version due to small (or big) breaking change. Maybe > limiting python version should be by default (always), until we explicitly > move that limit when we integrate a new version. Airflow would not be > installable on newest version that were not explicitly tested against. > > Le lun. 23 oct. 2023 à 14:19, Aritra Basu <aritrabasu1...@gmail.com> a > écrit : > > > I think I'm on the side of giving an error message saying 3.12 is not > > yet supported in 2.7.3, I would assume anyone seeing that would > > understand the implication that neither does 2.7.2 and thus they > > wouldn't try installing it. > > > > Though I would also think they would have the same understanding that > > if > > 2.7.3 doesn't > > list 3.12 as supported neither would 2.7.2. But I'd rather be > > explicitly told 2.7.3 and earlier don't work with 3.12. > > > > Thanks and Regards, > > Aritra Basu > > > > > > On Mon, Oct 23, 2023 at 4:54 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > Hey everyone, > > > > > > I've opened a PR https://github.com/apache/airflow/pull/35123 to > > > limit Airflow to Python < 3.12 though I am not sure if this is the > > > best idea > > so I > > > seek devlist wisdom to decide whether we should do this, or maybe > > something > > > else like allowing airflow to be installed but produce a clean error > > > indicating 3.12 is not (yet) supported. > > > > > > Currently Airflow does not work for Python 3.12 - mainly because of > > > "distutils" removal https://peps.python.org/pep-0632/ and the way > > > how > > some > > > of the core dependencies - Pendulum for on - and likely some of our > > > providers have other breaking dependencies. While Andrey works on > > Pendulum > > > upgrade as they released version 3 that is compatible, this might > > > take > > some > > > time and we might find out there are other problems as we are not > > > running the full test suite for 3.12 due to dependency issues. > > > > > > It looks like that Python 3.12 is a bit different than 3.7 - 3.11 > > releases > > > were - this is the first release that is more aggressive when it > > > comes to backwards compatibility. There are also some unit test > > > methods removal > > and > > > other change that might make our 3.12 migration a bit more painful > > > (more details here for example) > > > https://towardsdatascience.com/python3-12-98245ecd6a97 > > > > > > Currently people attempting to install Airflow with Python 3.12 will > > > get unobvious errors (like build errors or runtime errors that are > cryptic). > > > This is already happening and users are reporting issues that > > > "airflow > > does > > > not work" which boils down to installing Airflow (or breeze) for > > > Python 3.12. > > > > > > I initially thought (hence the PR) that we can limit Airflow to < > > > 3.12 > > and > > > release it in 2.7.3 (and remove the limitation once we get official > > > 3.12 support). > > > > > > So maybe a better solution will be to check the Python version at > > > startup time and fail Airflow with clear error. Or maybe we should > > > not worry > > about > > > it at all? > > > > > > The problem is that if we limit 2.7.3, users installing airflow on > > > 3.12 will install 2.7.2 because it has no limit for Python - and > > > will end up with the very same experience they have now. > > > > > > WDYT? > > > > > > J. > > > > > > ________________________________ > Strike Technologies, LLC (“Strike”) is part of the GTS family of > companies. Strike is a technology solutions provider, and is not a broker > or dealer and does not transact any securities related business directly > whatsoever. This communication is the property of Strike and its > affiliates, and does not constitute an offer to sell or the solicitation of > an offer to buy any security in any jurisdiction. It is intended only for > the person to whom it is addressed and may contain information that is > privileged, confidential, or otherwise protected from disclosure. > Distribution or copying of this communication, or the information contained > herein, by anyone other than the intended recipient is prohibited. If you > have received this communication in error, please immediately notify Strike > at i...@striketechnologies.com, and delete and destroy any copies hereof. > ________________________________ > > CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any attachments > are intended solely for the addressee. This transmission is covered by the > Electronic Communications Privacy Act, 18 U.S.C ''2510-2521. The > information contained in this transmission is confidential in nature and > protected from further use or disclosure under U.S. Pub. L. 106-102, 113 > U.S. Stat. 1338 (1999), and may be subject to attorney-client or other > legal privilege. Your use or disclosure of this information for any purpose > other than that intended by its transmittal is strictly prohibited, and may > subject you to fines and/or penalties under federal and state law. If you > are not the intended recipient of this transmission, please DESTROY ALL > COPIES RECEIVED and confirm destruction to the sender via return > transmittal. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org >
