I agree. Thanks for the investigation and write up, Evan! Sam
On Mon, Aug 3, 2026, at 6:42 PM, Ville Brofeldt wrote: > Hi all, > > this has been long overdue - thanks for taking on this important work Evan! I > agree many connectors are now 2.0 only on the latest stable version. Also, db > connectors are arguably one of the most critical components in Superset, so I > feel we really can’t kick the can on this anymore. Any engine not currently > supporting 2.0 can IMO be considered abandoned, so I feel it’s ok to move on. > I suggest waiting on this getting merged before cutting 7.0. > > Ville > > > On Aug 3, 2026, at 3:08 PM, Evan Rusackas <[email protected]> wrote: > > > > Hi all, > > > > Sorry for the long email, but here’s an update on bumping to SQLAlchemy 2.0. > > > > TL;DR (consensus needed): it’ll cost us compatibility with a few smaller > > databases, and I want an OK on that. > > > > Here’s the full story: > > > > > > We've been chipping away at a SQLAlchemy 2.0 migration for a while now (see > > Discussion #40273 [1], started by @hy144328). Steps 1-5 of that plan are > > done and merged, hooray! > > > > The current PR [2] is to prep the actual core `sqlalchemy` version bump is > > open, which brings every SQLAlchemy-dependent driver package we ship up to > > the newest version that still works under both 1.4 and 2.0, so each driver > > bump can be reviewed and tested independently before we flip the core > > version. > > > > > > That prep work surfaced something we haven't had a real conversation about > > yet, thus this email. A handful of databases don't have a SQLAlchemy > > 2.0-compatible driver available at all, and a few more can only move in > > lockstep with the core bump itself, with zero room for a staged rollout. > > Before we commit to step 6, I want this list in front of the list, not just > > buried in a PR description. > > > > > > Why SQLA 2.0 now: This isn't purely a "should modernize" push. Two pains > > are driving it: > > > > * Dremio's SQLAlchemy dialect (and probably others now, if not soon) now > > requires 2.0. > > * We're effectively pinned to Python 3.11 (nearing EOL) as a downstream > > consequence of the SQLAlchemy 1.4 pin so we seemingly must bump it to keep > > up with Python itself. > > > > SQLAlchemy 1.4 itself has no official EOL date but is "virtually there," > > per upstream, and 2.1 is coming, which will make 1.4 fully obsolete. > > > > So… what I want consensus on: > > This would break databases with no SQLAlchemy 2.0-compatible driver > > anywhere upstream. > > > > These would need to be dropped when we move to 2.0-only drivers, unless > > someone (us?) does the upstream work first: > > > > - Azure Data Explorer / Kusto (sqlalchemy-kusto) — upstream is actively > > maintained (releases into this year) but hard-pinned to `sqlalchemy==1.4.*` > > even in its latest release. > > - Aurora, Data-API mode specifically (sqlalchemy-aurora-data-api). Our own > > fork (`preset-io/sqlalchemy-aurora-data-api`) has been dormant since 2021. > > A more active community fork exists but has an open, unresolved SQLAlchemy > > 2.0 issue. Note this is *only* the serverless HTTP Data API access mode, > > and regular Aurora MySQL/Postgres-protocol access uses our standard > > mysql/postgres engine specs and is unaffected. > > - Apache Solr (sqlalchemy-solr) — effectively dormant upstream, > > dependabot-only bumps since 2024. > > - Cloudflare D1 (sqlalchemy-d1) — very young project (first real commits > > Nov 2025), no signal either way yet. > > > > I don't have usage telemetry on how many real deployments run these four. > > If anyone does, that data point would help a lot here — my working guess is > > these are low-traffic engines, but I'd rather not assume. > > > > Less risky: Databases that aren't lost, but require a synchronized cutover. > > These drivers jumped straight from 1.4-only to 2.0-only with no > > dual-compatible bridge release, so the driver bump has to land in the > > *same* release as the core bump, not before and not after: > > > > - RisingWave. The first 2.0-only release only landed this week, so this is > > freshly resolved, not long-settled. > > - Exasol. Its Dec 2025 release requires 2.0+. > > - Firebird. Worth noting this driver has “required" SQLAlchemy 2.0 on any > > Python 3.8+ for three years already so Firebird is probably broken now > > unless anyone can validate otherwise. > > - Redshift. Clean cutover, ~3-year gap with zero intermediate releases > > between the 1.4-only and 2.0-only versions. > > - Dremio. This is messier: the currently-pinned version has an open-ended > > version constraint, so pip metadata alone wouldn't block installing it > > under 2.0, but whether it actually *works* under 2.0 isn't documented > > anywhere I could find, so it needs an independent smoke test, but id does > > have a 2.0 targeted release available. > > > > Unrelated to the above but worth fixing regardless: > > Teradata's recommended SQLAlchemy dialect package (teradatasqlalchemy) > > isn't pinned anywhere in our dependency files at all, but there's already a > > SQLAlchemy 2.0-safe release on PyPI, so Teradata itself isn't at risk. > > We’ve Just been shipping an unmanaged dependency here and should pin it. > > > > Heads up: Flask-SQLAlchemy bump needed > > We’ll need to rekindle something like > > https://github.com/apache/superset/pull/35117 > > > > Version 3.0+ changes session scoping from per-thread to per-app-context, > > and that broke our Celery task-boundary handling in real CI runs when > > tested (a reproducible `AttributeError`s across dashboards/security/ tasks > > tests, plus MySQL lock-wait timeouts). > > > > Flask-SQLAlchemy 3.1+ requires SQLAlchemy 2.0, so we can't dodge this > > forever if we do the core bump, and it needs its own investigation. I think > > this is a bigger open risk to how long remaining steps actually take than > > any single database driver above, and didn't want to bury it under “driver > > compatibility." > > > > What I'd like to hear on this thread > > > > Does anyone have usage data or objections on the four "would be dropped” > > engines (Kusto, Aurora Data API, Solr, D1)? That's the single biggest input > > into how controversial this actually is. > > > > Any volunteers to smoke-test Dremio and sanity-check current Firebird > > support? > > > > Given the Python-EOL pressure described above, does the timeline urgency > > outweigh the disruption to these ~9 engines? And getting this into the > > “breaking change window” 7.0 offers us? > > > > Per lazy consensus, if there are no objections raised in 3 days, I’ll take > > that as license to move forward with attempting the upgrade, adjust docs > > and UPDATING.md with compatibility notes, and starting to try to contribute > > upstream to restore compatibility. > > > > Thanks, > > Evan > > > > > > - [1] Battleplan discussion: > > https://github.com/apache/superset/discussions/40273 > > - [2] Driver-compat prep PR: https://github.com/apache/superset/pull/42542 > > - [3] Original migration attempt (bumping w/flask-sqlalchemy): > > https://github.com/apache/superset/pull/35117 > > > > Evan Rusackas > > Preset | preset.io > >
