rusackas opened a new pull request, #42542: URL: https://github.com/apache/superset/pull/42542
### SUMMARY See #40273 (SQLAlchemy 2.0 migration battleplan). Steps 1-5 of the battleplan are done as of this week (#42277, #42365 merged, `__allow_unmapped__` landed in #41179). This PR is prep for step 6, the actual core `sqlalchemy` bump: it bumps every SQLAlchemy-dependent driver/framework package Superset ships (default install + every optional dialect extra) to the newest version that supports **both** SQLAlchemy 1.4 and 2.0, so the driver work can be merged and tested independently under today's SQLAlchemy 1.4 pin, before the core bump happens as its own small, final PR. **The one bump that actually matters:** `flask-sqlalchemy` was purely transitive (via `flask-appbuilder`, resolving to `2.5.1`, 1.4-only) — now pinned explicitly at `>=3.0.5,<3.1`, the last release before 3.1.0 drops SQLAlchemy 1.4 support. `flask-appbuilder` (5.2.2) and `marshmallow-sqlalchemy` (1.5.0) are already dual-compatible as currently pinned, `alembic` and `flask-migrate` are both unbounded on SQLAlchemy — no changes needed there. Superset's own code already has forward-compat shims in place for this bump (`superset/queries/filters.py`, `superset/extensions/__init__.py`), which is presumably why it went this smoothly. **Small, safe bumps:** - `sqlalchemy-bigquery` 1.17.0 → 1.17.1 (patch; `googleapis/python-bigquery-sqlalchemy` was archived 2026-05-16, likely its final release) - `sqlalchemy-cratedb` 0.41.0 → 0.43.1 - `sqlalchemy-drill`: dropped an artificial `<2` SQLAlchemy-version ceiling upstream never had (1.1.10 already ships `import_dbapi`) - `firebolt-sqlalchemy` 1.0.0 → 1.1.2 - `sqlalchemy-ocient` 1.0.0 → 3.0.0 (lower confidence — closed-source vendor package, no public changelog to verify against) - `snowflake-sqlalchemy`'s floor was already independently bumped by dependabot (#42418) to the same target this research landed on — nothing to do there. **Five extras skipped for now — no dual-compat version exists:** `sqlalchemy-risingwave`, `sqlalchemy-exasol`, `sqlalchemy-firebird`, `sqlalchemy-redshift`, and `sqlalchemy-dremio`'s newest release all cut hard from SQLAlchemy-1.4-only straight to 2.0-only with zero overlapping version. Each is pinned to its last 1.4-only release with a comment explaining why, and the SQLAlchemy-2.0 target version to bump to **once the core sqlalchemy bump lands, not before**. **Four extras flagged as currently unsupported under SQLAlchemy 2.0:** `sqlalchemy-kusto`, `sqlalchemy-aurora-data-api` (including our own dormant `preset-io` fork), `sqlalchemy-solr`, and `sqlalchemy-d1` have no SQLAlchemy 2.0 support anywhere in their ecosystem today. Rather than just noting this in the PR description, added a new `known_incompatibilities` field to `DBEngineSpecMetadata` (`superset/db_engine_specs/base.py`) that flows through to the auto-generated database docs, so this is visible to anyone looking at those DBs' docs pages, not just buried in a merged PR. Checked each project's repo (not just PyPI metadata) for a path forward: - **kusto**: actively maintained upstream (commits within the last week), but `setup.py` on `main` hard-pins `sqlalchemy==1.4.*` — worth filing an issue asking their plans. - **aurora-data-api**: our own fork (`preset-io/sqlalchemy-aurora-data-api`) is dead since 2021. The more active community fork (`cloud-utils/sqlalchemy-aurora-data-api`) has an open, unresolved SQLAlchemy 2.0 issue ([#43](https://github.com/cloud-utils/sqlalchemy-aurora-data-api/issues/43)) — since we already own a fork here, this is the one case where contributing a patch is realistically in our own hands rather than waiting on a stranger. - **solr**: effectively dormant (only dependabot bumps since 2024), no engagement to build on. - **d1**: very young (first real commits Nov 2025), no signal either way yet — probably just needs watching. **One correction worth flagging:** `sqlalchemy-vertica-python` was initially flagged as a fifth unsupported case based on PyPI metadata alone, but turned out to already be dual-compatible — the actual merged upstream PR ([bluelabsio/sqlalchemy-vertica-python#18](https://github.com/bluelabsio/sqlalchemy-vertica-python/pull/18)) landed in the current pin (0.6.3) and adds `import_dbapi` while keeping the 1.4-style `dbapi` alias. No change needed there; removed from the flagged list once verified against the actual source. ### TESTING INSTRUCTIONS - `requirements/base.txt` and `requirements/development.txt` regenerated via `./scripts/uv-pip-compile.sh` (the official Docker-based compile process) — full dependency resolution succeeds with no conflicts. - Locally verified in a clean venv: app boots and `db.create_all()` succeeds cleanly under SQLAlchemy 1.4.54 + Flask-SQLAlchemy 3.0.5. - Ran `tests/unit_tests/models/`, `tests/unit_tests/db_engine_specs/test_postgres.py`, `test_mysql.py`, `test_aurora.py`, `test_kusto.py`: 483/487 passed. The 4 `test_mysql.py` failures are a pre-existing local `mysqlclient` linkage issue on this machine (unrelated `dlopen` symbol error), not a regression from this change. - Verified the new `known_incompatibilities` metadata flows through `generate_yaml_docs()` end-to-end for all four flagged engines (Solr, D1, Kusto, and the Aurora entries nested under MySQL/Postgres `compatible_databases`). - Local `pre-commit` (mypy, ruff, pylint, the `db-engine-spec-metadata` validator) all pass. ### FOLLOW-UP (not in this PR) - File an issue with `dodopizza/sqlalchemy-kusto` asking about SQLAlchemy 2.0 plans. - Decide whether to patch our own `preset-io/sqlalchemy-aurora-data-api` fork directly, or contribute to `cloud-utils/sqlalchemy-aurora-data-api` issue #43 instead. - `sqlalchemy-solr` and `sqlalchemy-d1` are lower priority — watch for upstream activity. - Once this lands, consider adding an allow-to-fail CI matrix leg that installs SQLAlchemy 2.0 and runs the suite (same pattern as the existing Python `next`-version leg), so step 6 gets continuous signal before anyone commits to the actual core bump. - The `risingwave`/`exasol`/`firebird`/`redshift`/`dremio` ceilings documented here need to flip to their 2.0-only versions in lockstep with the core `sqlalchemy` bump itself (step 6), not before. ### ADDITIONAL INFORMATION - [ ] Has associated issue: - [ ] Required feature flags: - [ ] Changes UI - [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351)) - [ ] Migration is atomic, supports rollback & is backwards-compatible - [ ] Confirm DB migration upgrade and downgrade tested - [ ] Runtime estimates and downtime expectations provided - [ ] Introduces new feature or API - [ ] Removes existing feature or API -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
