Dear Community, I’d like to start a discussion regarding a proposal to *migrate Airflow to SQLAlchemy 2.0* for the upcoming *v3.2* feature release. The Problem: Maintenance Overhead and Locked Ecosystem
As many of you know, there is a massive ongoing effort by community members to improve and upgrade type hints across the entire codebase. However, because our CI currently runs on SQLAlchemy 1.4, *Mypy does not "protect" us* against introducing new violations that would be caught in a 2.0 environment. This creates a significant maintenance burden where contributors and reviewers must manually account for type safety that the CI is unable to enforce. Moving to 2.0 provides native PEP 484 support, which will drastically improve ORM class coverage and allow us to catch bugs during development rather than at runtime. Staying on 1.4 has created a "dependency ceiling." Migrating to SQLAlchemy 2.0 will finally allow us to upgrade several key libraries that the community has been asking for, including: - *pandas* >= 2.2 (released on Jan 2024) - *numpy* >= 2 (released on Jun 2024) - *flask-sqlalchemy* >= 3.1 (released on Sep 2023) - *sqlmodel* >= 0.12 (released on Nov 2023) Current Progress We already have a functional path forward. There is an outstanding PR that implements this transition: - *PR:* https://github.com/apache/airflow/pull/59218 *Note:* This PR is currently *green* (passing all CI checks), demonstrating that the core functional changes are stable and do not break our existing test suite. Seeking Community Feedback We want to ensure this transition is as smooth as possible for all stakeholders. We are particularly interested in your views on: 1. *Custom Providers/Plugins:* Are there concerns regarding plugins that rely heavily on SQLAlchemy 1.4 internals? 2. *Database Backends:* Does anyone foresee specific issues with our supported versions of Postgres, MySQL, or SQLite during this jump? 3. *General Timeline:* Does the community feel Airflow 3.2 is the right target for this breaking internal change? Should we aim for official dual support for a period of time? Please share your thoughts, concerns, or support for this move. If there is a general consensus, we plan to proceed with the review and merge of the PR in the near future. Best regards, Dev-iL
