Hi Jarek, Thanks for this initiative. It is always a good idea to let the end users know that "oh the software does not work with your current set of dependencies, do ---- to handle it"
I like the idea of limiting the versions to < 3.12 and am for it. While we do this, we should not spend a lot of time and effort on this, because this might be a temporary workaround. Is there anything that we can do about 2.7.2 which is already released? Might be too late in the game. We might have to clearly document this so that users don't have this experience at all. Thanks, Amogh Desai On Wed, Oct 25, 2023 at 8:17 PM Damian Shaw <ds...@striketechnologies.com> wrote: > > But it seems to be changing now. Also I think (by listening to some > podcast) - Python community starts to be overwhelmed with keeping all the > backwards-compatibility and they are changing their approach for future > releases to be more aggressive in removals, separate out certain "core" > features to outside-libraries and removing them out of the core Python > > I appreciate this is a little off topic, but I follow core dev of CPython > closely and would just like to add I don't think this take is quite > accurate. IMO Python core dev are focused on Python backwards compatibility > in general but say a few things happened: > > 1. Functions that have been marked as deprecated for many releases of > Python are finally being cleaned up > 2. A group of modules which are completely unmaintained got marked as > deprecated in PEP 594 (the Steering Council made clear no similar PEPs > would be accepted and each module will be treated on a case-by-case basis) > 3. Several large projects are working on the underlying implementation of > CPython and they are more willing to propose larger changes to the C-API > than has happened in the past > > There has been discussion implementing an official "soft depreciation", > which will avoid the eventual removal of a function but no longer support > any development: > https://discuss.python.org/t/formalize-the-concept-of-soft-deprecation-dont-schedule-removal-in-pep-387-backwards-compatibility-policy/27957 > > Damian > > -----Original Message----- > From: Jarek Potiuk <ja...@potiuk.com> > Sent: Wednesday, October 25, 2023 6:15 AM > To: dev@airflow.apache.org > Subject: Re: Limiting (or errorring out) Airflow for Python 3.12 until our > dependencies/we catch up > > > I agree, has there been a new release of Python that has worked with > Airflow without at least some fixing? > > Generally yes. Most of the past bumps were mostly test fixing. I can't > recall any "serious" changes in the code. Maybe in some single providers. > Generally Python 3.7 -> 3.11 were extremely backwards compatible, there > were really maybe 2 cases I remember where we had compatibility issues in > airflow "core" and they were because of security fixes (some edge cases of > url parsing changed Is one that I remember) - and that was even in > patchlevel releases - not even in the minor release. > > > I understand that Airflow is both a library and an application and I > > do > agree with not putting upper bounds on libraries unless required, but it > seems like allowing an arbitrary upper bound of Python is never going to be > practical. If users really need to break this support version and install > an older version of Airflow with a newer version of Python it isn't > difficult to build Airflow locally with patches, and they will likely need > to add more patches to get Airflow to work anyway, so their cost is already > built in. > > To be honest, It's been practical for now, this is the first time it has > happened for quite a while. But it seems to be changing now. Also I think > (by listening to some podcast) - Python community starts to be overwhelmed > with keeping all the backwards-compatibility and they are changing their > approach for future releases to be more aggressive in removals, separate > out certain "core" features to outside-libraries and removing them out of > the core Python (which is actually quite an inspiration for us in what we > do by moving stuff - like executors, hopefully FAB in the future and maybe > few others from core to providers), so we can expect more of such breakages > in the moment. Having explicit upper-binding will be a good idea. > > I think we can do a hybrid solution. We can upper-bind for 2.8 but when we > release 2.7.3 we can implement raising error if someone wants to use Python > 3.12. I think that will be the best approach. > > J. > > > > On Mon, Oct 23, 2023 at 6:33 PM Damian Shaw <ds...@striketechnologies.com> > wrote: > > > I agree, has there been a new release of Python that has worked with > > Airflow without at least some fixing? > > > > I understand that Airflow is both a library and an application and I > > do agree with not putting upper bounds on libraries unless required, > > but it seems like allows an arbitrary upper bound of Python is never > > going to be practical. If users really need to break this support > > version and install an older version of Airflow with a newer version > > of Python it isn't difficult to build Airflow locally with patches, > > and they will likely need to add more patches to get Airflow to work > > anyway, so their cost is already built in. > > > > Even if just limiting to 3.11 or lower for 2.7.3 or higher I think the > > benefits are there, Python 3.12 might be supported soon or there might > > be more hidden issues that mean it isn't supported for 9+ months. > > > > Damian > > > > -----Original Message----- > > From: Pierre Jeambrun <pierrejb...@gmail.com> > > Sent: Monday, October 23, 2023 12:13 PM > > To: dev@airflow.apache.org > > Subject: Re: Limiting (or errorring out) Airflow for Python 3.12 until > > our dependencies/we catch up > > > > I think that limiting to <3.12 makes sense. 2.7.2 is already out so > > I'm not sure we can do anything for users trying to install 2.7.2 on > > python 3.12. > > > > I believe there is no such thing as a python minor that is out of the > > box working well for airflow. It seems that we always need extra > > efforts to bring in a new version due to small (or big) breaking > > change. Maybe limiting python version should be by default (always), > > until we explicitly move that limit when we integrate a new version. > > Airflow would not be installable on newest version that were not > explicitly tested against. > > > > Le lun. 23 oct. 2023 à 14:19, Aritra Basu <aritrabasu1...@gmail.com> a > > écrit : > > > > > I think I'm on the side of giving an error message saying 3.12 is > > > not yet supported in 2.7.3, I would assume anyone seeing that would > > > understand the implication that neither does 2.7.2 and thus they > > > wouldn't try installing it. > > > > > > Though I would also think they would have the same understanding > > > that if > > > 2.7.3 doesn't > > > list 3.12 as supported neither would 2.7.2. But I'd rather be > > > explicitly told 2.7.3 and earlier don't work with 3.12. > > > > > > Thanks and Regards, > > > Aritra Basu > > > > > > > > > On Mon, Oct 23, 2023 at 4:54 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > > > Hey everyone, > > > > > > > > I've opened a PR https://github.com/apache/airflow/pull/35123 to > > > > limit Airflow to Python < 3.12 though I am not sure if this is the > > > > best idea > > > so I > > > > seek devlist wisdom to decide whether we should do this, or maybe > > > something > > > > else like allowing airflow to be installed but produce a clean > > > > error indicating 3.12 is not (yet) supported. > > > > > > > > Currently Airflow does not work for Python 3.12 - mainly because > > > > of "distutils" removal https://peps.python.org/pep-0632/ and the > > > > way how > > > some > > > > of the core dependencies - Pendulum for on - and likely some of > > > > our providers have other breaking dependencies. While Andrey works > > > > on > > > Pendulum > > > > upgrade as they released version 3 that is compatible, this might > > > > take > > > some > > > > time and we might find out there are other problems as we are not > > > > running the full test suite for 3.12 due to dependency issues. > > > > > > > > It looks like that Python 3.12 is a bit different than 3.7 - 3.11 > > > releases > > > > were - this is the first release that is more aggressive when it > > > > comes to backwards compatibility. There are also some unit test > > > > methods removal > > > and > > > > other change that might make our 3.12 migration a bit more > > > > painful (more details here for example) > > > > https://towardsdatascience.com/python3-12-98245ecd6a97 > > > > > > > > Currently people attempting to install Airflow with Python 3.12 > > > > will get unobvious errors (like build errors or runtime errors > > > > that are > > cryptic). > > > > This is already happening and users are reporting issues that > > > > "airflow > > > does > > > > not work" which boils down to installing Airflow (or breeze) for > > > > Python 3.12. > > > > > > > > I initially thought (hence the PR) that we can limit Airflow to < > > > > 3.12 > > > and > > > > release it in 2.7.3 (and remove the limitation once we get > > > > official > > > > 3.12 support). > > > > > > > > So maybe a better solution will be to check the Python version at > > > > startup time and fail Airflow with clear error. Or maybe we should > > > > not worry > > > about > > > > it at all? > > > > > > > > The problem is that if we limit 2.7.3, users installing airflow on > > > > 3.12 will install 2.7.2 because it has no limit for Python - and > > > > will end up with the very same experience they have now. > > > > > > > > WDYT? > > > > > > > > J. > > > > > > > > > ________________________________ > > Strike Technologies, LLC (“Strike”) is part of the GTS family of > > companies. Strike is a technology solutions provider, and is not a > > broker or dealer and does not transact any securities related business > > directly whatsoever. This communication is the property of Strike and > > its affiliates, and does not constitute an offer to sell or the > > solicitation of an offer to buy any security in any jurisdiction. It > > is intended only for the person to whom it is addressed and may > > contain information that is privileged, confidential, or otherwise > protected from disclosure. > > Distribution or copying of this communication, or the information > > contained herein, by anyone other than the intended recipient is > > prohibited. If you have received this communication in error, please > > immediately notify Strike at i...@striketechnologies.com, and delete > and destroy any copies hereof. > > ________________________________ > > > > CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any > > attachments are intended solely for the addressee. This transmission > > is covered by the Electronic Communications Privacy Act, 18 U.S.C > > ''2510-2521. The information contained in this transmission is > > confidential in nature and protected from further use or disclosure > > under U.S. Pub. L. 106-102, 113 U.S. Stat. 1338 (1999), and may be > > subject to attorney-client or other legal privilege. Your use or > > disclosure of this information for any purpose other than that > > intended by its transmittal is strictly prohibited, and may subject > > you to fines and/or penalties under federal and state law. If you are > > not the intended recipient of this transmission, please DESTROY ALL > > COPIES RECEIVED and confirm destruction to the sender via return > transmittal. > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > ________________________________ > Strike Technologies, LLC (“Strike”) is part of the GTS family of > companies. Strike is a technology solutions provider, and is not a broker > or dealer and does not transact any securities related business directly > whatsoever. This communication is the property of Strike and its > affiliates, and does not constitute an offer to sell or the solicitation of > an offer to buy any security in any jurisdiction. It is intended only for > the person to whom it is addressed and may contain information that is > privileged, confidential, or otherwise protected from disclosure. > Distribution or copying of this communication, or the information contained > herein, by anyone other than the intended recipient is prohibited. If you > have received this communication in error, please immediately notify Strike > at i...@striketechnologies.com, and delete and destroy any copies hereof. > ________________________________ > > CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any attachments > are intended solely for the addressee. This transmission is covered by the > Electronic Communications Privacy Act, 18 U.S.C ''2510-2521. The > information contained in this transmission is confidential in nature and > protected from further use or disclosure under U.S. Pub. L. 106-102, 113 > U.S. Stat. 1338 (1999), and may be subject to attorney-client or other > legal privilege. Your use or disclosure of this information for any purpose > other than that intended by its transmittal is strictly prohibited, and may > subject you to fines and/or penalties under federal and state law. If you > are not the intended recipient of this transmission, please DESTROY ALL > COPIES RECEIVED and confirm destruction to the sender via return > transmittal. >
