aminghadersohi opened a new pull request, #45843: URL: https://github.com/apache/superset/pull/45843
### SUMMARY Automatically migrate saved `databricks+connector://` URIs to the registered SQLAlchemy 2 `databricks://` dialect for workspaces upgrading from 6.0 with a legacy Databricks connection. No API access or manual connection recreation is required. - Rewrite only the scheme prefix; leave the URI suffix and separately encrypted password/encrypted-extra columns untouched. - Mark converted connections in `extra`; downgrade restores only those connections and removes the marker. Use bounded, keyset-paginated batches inside Alembic's transaction. - Keep editing available when a saved driver disappears: use its saved parameter schema, or fall back to an editable SQLAlchemy URI form if no schema exists. - Advertise the installed Databricks driver so its field-based form is available. Read legacy HTTP path/catalog settings from `extra`, preserve path-based schemas, and let subsequent form edits supersede legacy connect arguments. - Keep legacy form saves on the supported URI scheme and update connection examples. The existing URI-to-dynamic-form link remains available. Related independent migration: https://github.com/apache/superset/pull/44278 If both changes land, the pyodbc migration must target `databricks://`, not `databricks+connector://`. Maintainers can decide whether to combine the migrations. This change does not depend on or modify that pull request. ### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF Real Docker Superset, with a synthetic legacy connection and fake token. Before: the edit modal has no connection fields. After: the native field-based form contains the saved host, port, HTTP path, catalog, schema and masked token; the token field is editable. | Before | After | | --- | --- | |  |  | ### TESTING INSTRUCTIONS 1. Before upgrading, save a legacy Databricks connection with `extra.engine_params.connect_args.http_path` and `catalog` populated. 2. Run `superset db upgrade`. Open the saved connection and confirm its host, HTTP path, catalog, schema and masked token. Replace the token or other fields through the form as needed. 3. Create a separate native `databricks://` connection. Run `superset db downgrade f72b6e9c104a`: only the migrated connection should return to the legacy scheme; the native connection must remain unchanged. 4. Run `superset db upgrade` again. Verified in this run: - `pre-commit run`: all applicable staged-file hooks pass, including mypy and frontend type checking. - Targeted backend run (Databricks engine specs, migration, database model tests and single-head guard): **259 passed, 3 skipped**. - DatabaseModal Jest suite: **89 passed**. Both unavailable-driver regression cases fail against master and pass with this change. - Engine-spec regressions and the updated legacy URI-builder assertion fail against master. The migration tests reject the pre-migration no-op behavior (**8 failures** with the rewrite disabled); master has no such migration module. - Docker with SQLAlchemy 2 and `databricks-sqlalchemy==2.0.1`: **119 passed** across the Databricks/migration/single-head tests, including the installed-dialect check. - Docker upgrade/downgrade/re-upgrade: exact URI suffix and encrypted ciphertext preserved; separately created native connection untouched. The actual dialect receives the expected host, token, HTTP path, catalog and schema, intercepted before network I/O. - Playwright against Docker reproduced the empty form before the change and verified populated native fields and an editable token afterward. Migration lineage: ```text superset db heads # Docker, before adding this migration f72b6e9c104a (head) superset db heads # Docker, with this migration b8f3a2c91d64 (head) down_revision = "f72b6e9c104a" git log -1 --format=%H -- superset/migrations/versions/2026-10-22_12-00_f72b6e9c104a_add_oauth2_token_management_indexes.py 1aceca35ed218288dcf2b20ed09dbc053c204a47 ``` Live Databricks queries, charts, SQL Lab execution and remote Test Connection were not run: the acceptance fixture uses a fake token and stops before DBAPI network connection. No real model-provider API was used. ### ADDITIONAL INFORMATION - [ ] Has associated issue: - [ ] Required feature flags: - [x] Changes UI - [x] Includes DB Migration - [x] Migration is atomic, supports rollback & is backwards-compatible - [x] Confirm DB migration upgrade and downgrade tested - [x] Runtime estimates and downtime expectations provided - [ ] Introduces new feature or API - [ ] Removes existing feature or API Runtime/downtime: no DDL; O(number of saved connections), with batches of at most 1,000 rows and no intermediate commits. The single-row Docker upgrade reported `Duration: 00:00:00` for migration execution. Larger installations depend on connection count and metadata database performance; run within the normal upgrade maintenance window. Sibling audit (`git grep -nF 'databricks+connector'`): engine-spec creation examples and both current/6.1 database metadata snapshots use the supported scheme. Remaining occurrences are the migration's source prefix, upgrade documentation, historical 6.0 documentation, and intentional legacy-URI parser/OAuth/modal regression fixtures. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
