Hi all, this has been long overdue - thanks for taking on this important work Evan! I agree many connectors are now 2.0 only on the latest stable version. Also, db connectors are arguably one of the most critical components in Superset, so I feel we really can’t kick the can on this anymore. Any engine not currently supporting 2.0 can IMO be considered abandoned, so I feel it’s ok to move on. I suggest waiting on this getting merged before cutting 7.0.
Ville > On Aug 3, 2026, at 3:08 PM, Evan Rusackas <[email protected]> wrote: > > Hi all, > > Sorry for the long email, but here’s an update on bumping to SQLAlchemy 2.0. > > TL;DR (consensus needed): it’ll cost us compatibility with a few smaller > databases, and I want an OK on that. > > Here’s the full story: > > > We've been chipping away at a SQLAlchemy 2.0 migration for a while now (see > Discussion #40273 [1], started by @hy144328). Steps 1-5 of that plan are done > and merged, hooray! > > The current PR [2] is to prep the actual core `sqlalchemy` version bump is > open, which brings every SQLAlchemy-dependent driver package we ship up to > the newest version that still works under both 1.4 and 2.0, so each driver > bump can be reviewed and tested independently before we flip the core version. > > > That prep work surfaced something we haven't had a real conversation about > yet, thus this email. A handful of databases don't have a SQLAlchemy > 2.0-compatible driver available at all, and a few more can only move in > lockstep with the core bump itself, with zero room for a staged rollout. > Before we commit to step 6, I want this list in front of the list, not just > buried in a PR description. > > > Why SQLA 2.0 now: This isn't purely a "should modernize" push. Two pains are > driving it: > > * Dremio's SQLAlchemy dialect (and probably others now, if not soon) now > requires 2.0. > * We're effectively pinned to Python 3.11 (nearing EOL) as a downstream > consequence of the SQLAlchemy 1.4 pin so we seemingly must bump it to keep up > with Python itself. > > SQLAlchemy 1.4 itself has no official EOL date but is "virtually there," per > upstream, and 2.1 is coming, which will make 1.4 fully obsolete. > > So… what I want consensus on: > This would break databases with no SQLAlchemy 2.0-compatible driver anywhere > upstream. > > These would need to be dropped when we move to 2.0-only drivers, unless > someone (us?) does the upstream work first: > > - Azure Data Explorer / Kusto (sqlalchemy-kusto) — upstream is actively > maintained (releases into this year) but hard-pinned to `sqlalchemy==1.4.*` > even in its latest release. > - Aurora, Data-API mode specifically (sqlalchemy-aurora-data-api). Our own > fork (`preset-io/sqlalchemy-aurora-data-api`) has been dormant since 2021. A > more active community fork exists but has an open, unresolved SQLAlchemy 2.0 > issue. Note this is *only* the serverless HTTP Data API access mode, and > regular Aurora MySQL/Postgres-protocol access uses our standard > mysql/postgres engine specs and is unaffected. > - Apache Solr (sqlalchemy-solr) — effectively dormant upstream, > dependabot-only bumps since 2024. > - Cloudflare D1 (sqlalchemy-d1) — very young project (first real commits Nov > 2025), no signal either way yet. > > I don't have usage telemetry on how many real deployments run these four. If > anyone does, that data point would help a lot here — my working guess is > these are low-traffic engines, but I'd rather not assume. > > Less risky: Databases that aren't lost, but require a synchronized cutover. > These drivers jumped straight from 1.4-only to 2.0-only with no > dual-compatible bridge release, so the driver bump has to land in the *same* > release as the core bump, not before and not after: > > - RisingWave. The first 2.0-only release only landed this week, so this is > freshly resolved, not long-settled. > - Exasol. Its Dec 2025 release requires 2.0+. > - Firebird. Worth noting this driver has “required" SQLAlchemy 2.0 on any > Python 3.8+ for three years already so Firebird is probably broken now unless > anyone can validate otherwise. > - Redshift. Clean cutover, ~3-year gap with zero intermediate releases > between the 1.4-only and 2.0-only versions. > - Dremio. This is messier: the currently-pinned version has an open-ended > version constraint, so pip metadata alone wouldn't block installing it under > 2.0, but whether it actually *works* under 2.0 isn't documented anywhere I > could find, so it needs an independent smoke test, but id does have a 2.0 > targeted release available. > > Unrelated to the above but worth fixing regardless: > Teradata's recommended SQLAlchemy dialect package (teradatasqlalchemy) isn't > pinned anywhere in our dependency files at all, but there's already a > SQLAlchemy 2.0-safe release on PyPI, so Teradata itself isn't at risk. We’ve > Just been shipping an unmanaged dependency here and should pin it. > > Heads up: Flask-SQLAlchemy bump needed > We’ll need to rekindle something like > https://github.com/apache/superset/pull/35117 > > Version 3.0+ changes session scoping from per-thread to per-app-context, and > that broke our Celery task-boundary handling in real CI runs when tested (a > reproducible `AttributeError`s across dashboards/security/ tasks tests, plus > MySQL lock-wait timeouts). > > Flask-SQLAlchemy 3.1+ requires SQLAlchemy 2.0, so we can't dodge this forever > if we do the core bump, and it needs its own investigation. I think this is a > bigger open risk to how long remaining steps actually take than any single > database driver above, and didn't want to bury it under “driver > compatibility." > > What I'd like to hear on this thread > > Does anyone have usage data or objections on the four "would be dropped” > engines (Kusto, Aurora Data API, Solr, D1)? That's the single biggest input > into how controversial this actually is. > > Any volunteers to smoke-test Dremio and sanity-check current Firebird support? > > Given the Python-EOL pressure described above, does the timeline urgency > outweigh the disruption to these ~9 engines? And getting this into the > “breaking change window” 7.0 offers us? > > Per lazy consensus, if there are no objections raised in 3 days, I’ll take > that as license to move forward with attempting the upgrade, adjust docs and > UPDATING.md with compatibility notes, and starting to try to contribute > upstream to restore compatibility. > > Thanks, > Evan > > > - [1] Battleplan discussion: > https://github.com/apache/superset/discussions/40273 > - [2] Driver-compat prep PR: https://github.com/apache/superset/pull/42542 > - [3] Original migration attempt (bumping w/flask-sqlalchemy): > https://github.com/apache/superset/pull/35117 > > Evan Rusackas > Preset | preset.io
