This is the time of the year when we attempt to see how far we are from supporting minor python version released in October (i.e. Python 3.13 now). Usually it takes a couple of months before we can even try - because of the number of dependencies - and it was no different this time around.
For now (it took me a better part of the day today) I managed to build the CI image with all dependencies and run `uv sync --all-extras --python 3.13 --reinstall` successfully (but with some cheating, so don't hold your breath yet). Also I bet there will be a number of test errors to fix, but for me the goal for now is to see how we stand with dependencies. For those who would like to see the changes - the PR is here https://github.com/apache/airflow/pull/46891 And let me summarize the errors/issues/inconveniences I had to workaround: *The big one:* 1) Combination of Pandas, FAB and Sqlalchemy is such that we cannot have all of them installed in Python 3.12. We can have either Pandas + Sqlalchemy 2 (no Fab) or Fab and sqlalchemy 1 (no pandas). We know we need to upgrade to sqlalchemy 2 anyway - whatever the road we choose with Fab - either vendoring in and bumping Sqlalchemy 2 or hoping for Fab 5 release. For now - I simply removed Fab provider while the PR is in draft, to see if there are no more problems, and we will have to fix it before beta I am afraid anyway. 2) as explained in [1] I had to remove google-re and I used this opportunity to make an inventory of needed changes - I will summarize them in the google-re2 thread 3) Apache Beam is as usual a problem - this time there are two incompatibilities - pyarrow and pandas - but we can simply exclude beam from 3.13 (as usual) - usually they catch-up with the new version of python when the next is released (around october). They also have a lot of deps and more "holding" than ours. 4) databricks requires bumping of databricks-connector to 4.0.0 - we might need to hold on with that one depending on resulting of tests (this is also possible with bumping pandas + sqlalchemy and was not possible before, I do not expect major problems here) 5) Bumped grpcio-tools - they were held back previously - but removal of beam, upgrade of pandas allows it to be bumped to a newer version that supports 3.13 - that is helpful for a number of providers (waeviate, qdrant) to be 3.13 compatible.. 6) Few small things removed: plyvel, ydb, possibly also yandex - depends on test results Currently there is a `uv` workspace bug that requires us to comment out rather than exclude python versions for dependencies, but I hope that - as usual - they will fix it in days rather than weeks. The issue in uv repo https://github.com/astral-sh/uv/issues/11629 Those are all (non-trivial) issues so far, I think we are getting really, really close to having to decide on what to do with FAB and how to unleash sqlalchemy 2. I will talk to Daniel to get more informed about his 5.0.0 plans (he merged my security fix today in FAB and gearing for a new release apparently, but I bet it's going to be 4.5.4). Any comments? [1] https://lists.apache.org/thread/xtzdp4dx8s6dds4xdp9kdpohns9lvpst
