One more finding which I miss.. * Pendulum - the elephant in the room (I like that phrase, yes).
I saw that Ash had already been involved in discussions about Python 3.13 support for pendulum in the PR that my friend from DLTHub started - https://github.com/python-pendulum/pendulum/pull/871 - but regardless from that we also need to have Python 3.13 wheels https://github.com/python-pendulum/pendulum/issues/879#issuecomment-2671428096 . For now in my PR I had to install cargo (i.e. rust build system) in order to compile it - and I had to install a very new one - because the one that comes by default in Debian Bookworm, is too old to compile Pendulum. And unfortunately - as we know from the past Pendulum is not really well "maintained" - it took a long time for them to support 3.12 - and there does not seem a lot of "responsibiity/responsiveness" of doing it from the maintainers. On my personal (for now) list of "Airflow Beach Cleaning" projects, Pendulum is quite firmly between "yellow" and "red" ("watch" vs. "forego"). Being very blunt - should reconsider dropping Pendulum for Airflow 3 ? Or maybe just make it optional and only used in Python < 3.13 - and introduce potentially breaking changes by replacing Pendulum with non-naive datetime everywhere ? I think TP and Ash were the ones who knew most and discussed it before and there was even a proposal from TP to remove the pendulum (as far as I remember). Those changes would be potentially breaking, but because of Pendulum' drop-in replacement for datetime - very little breaking. WDYT? I will add it to the agenda of today's dev call -similarly as the other "cleaning" topics I started recently. J. On Wed, Feb 19, 2025 at 7:26 PM Jarek Potiuk <ja...@potiuk.com> wrote: > This is the time of the year when we attempt to see how far we are from > supporting minor python version released in October (i.e. Python 3.13 now). > Usually it takes a couple of months before we can even try - because of the > number of dependencies - and it was no different this time around. > > For now (it took me a better part of the day today) I managed to build the > CI image with all dependencies and run `uv sync --all-extras --python 3.13 > --reinstall` successfully (but with some cheating, so don't hold your > breath yet). Also I bet there will be a number of test errors to fix, but > for me the goal for now is to see how we stand with dependencies. > > For those who would like to see the changes - the PR is here > https://github.com/apache/airflow/pull/46891 > > And let me summarize the errors/issues/inconveniences I had to workaround: > > *The big one:* > > 1) Combination of Pandas, FAB and Sqlalchemy is such that we cannot have > all of them installed in Python 3.12. We can have either Pandas + > Sqlalchemy 2 (no Fab) or Fab and sqlalchemy 1 (no pandas). > > We know we need to upgrade to sqlalchemy 2 anyway - whatever the road we > choose with Fab - either vendoring in and bumping Sqlalchemy 2 or hoping > for Fab 5 release. For now - I simply removed Fab provider while the PR is > in draft, to see if there are no more problems, and we will have to fix it > before beta I am afraid anyway. > > 2) as explained in [1] I had to remove google-re and I used this > opportunity to make an inventory of needed changes - I will summarize them > in the google-re2 thread > > 3) Apache Beam is as usual a problem - this time there are two > incompatibilities - pyarrow and pandas - but we can simply exclude beam > from 3.13 (as usual) - usually they catch-up with the new version of python > when the next is released (around october). They also have a lot of deps > and more "holding" than ours. > > 4) databricks requires bumping of databricks-connector to 4.0.0 - we might > need to hold on with that one depending on resulting of tests (this is also > possible with bumping pandas + sqlalchemy and was not possible before, I do > not expect major problems here) > > 5) Bumped grpcio-tools - they were held back previously - but removal of > beam, upgrade of pandas allows it to be bumped to a newer version that > supports 3.13 - that is helpful for a number of providers (waeviate, > qdrant) to be 3.13 compatible.. > > 6) Few small things removed: plyvel, ydb, possibly also yandex - depends > on test results > > Currently there is a `uv` workspace bug that requires us to comment out > rather than exclude python versions for dependencies, but I hope that - as > usual - they will fix it in days rather than weeks. The issue in uv repo > https://github.com/astral-sh/uv/issues/11629 > > Those are all (non-trivial) issues so far, I think we are getting really, > really close to having to decide on what to do with FAB and how to unleash > sqlalchemy 2. > > I will talk to Daniel to get more informed about his 5.0.0 plans (he > merged my security fix today in FAB and gearing for a new release > apparently, but I bet it's going to be 4.5.4). > > > Any comments? > > [1] https://lists.apache.org/thread/xtzdp4dx8s6dds4xdp9kdpohns9lvpst >
