villebro opened a new pull request, #43920: URL: https://github.com/apache/superset/pull/43920
### SUMMARY **Umbrella / tracker PR for the "SQL execution on GTF" epic. This is a long-lived feature branch (`villebro/sqllab-gtf`); focused child PRs are reviewed against it and squashed in, then this PR merges the completed epic to `master`.** Opening as a **draft** while the child PRs land. This is the direct follow-up to the Global Async Queries → GTF migration ([#43407](https://github.com/apache/superset/pull/43407)), whose summary noted that *"SQL Lab async query execution … remains on its existing path and is intentionally deferred to a later migration."* This epic is that migration — and it goes further, converging **SQL Lab** and the **unified SQL execution API** ([#36529](https://github.com/apache/superset/pull/36529)) onto a single synchronous execution primitive and a single async orchestrator (GTF). #### Before: three SQL execution stacks 1. **Legacy SQL Lab** — the `sql_lab.get_sql_results` Celery task with *frozen* 6h time limits, a main-thread-only `SIGALRM` sync timeout, DB-row-polling cancellation (the web worker writes `Query.status=STOPPED`; the running worker notices between statement blocks), no heartbeat / orphan reaping, and a 2s `/query/updated_since` frontend poll. 2. **The unified SQL execution API** (`Database.execute()` / `execute_async()`) — meant to be the single "run SQL against a Database" contract, but used in production only by the MCP `execute_sql` tool (sync only). Its `execute_async()` half is a **third**, GTF-agnostic Celery stack (`AsyncQueryHandle`) with no production caller. 3. **GTF** — the shared `@task` / `.schedule()` framework chart-data already runs on after #43407. #### After: one primitive, one orchestrator - **Layer 1 — a synchronous execution *feature*** (`SQLExecutor` / `Database.execute()`), extended to be SQL-Lab-complete (CTAS/CVAS, tmp tables, `LimitingFactor`, `expand_data`, Arrow→results-backend, engine-cancel seam). - **Layer 2 — GTF task bodies**, each a *single* task (`execute_sql_query` PRIVATE + the existing `execute_chart_query` SHARED) that call Layer 1 **in-process**. - **Layer 3 — HTTP handlers** are the only schedulers; `Database.execute_async()` schedules the one SQL task. **Hard invariant:** a GTF task never schedules another task to run its SQL — the execution logic is a shared library both tasks call directly. Chart-data stays one task per `QueryObject`; SQL Lab async is one task per query. The `Query` row remains the SQL-Lab-facing source of truth; the task mirrors its terminal status into `Query.status`, so the existing poll keeps working while the frontend adopts the GTF `task.status` realtime push (reusing the transport chart-data uses), with polling as the correctness backstop. Design doc: [`SQL_EXECUTION_GTF.md`](SQL_EXECUTION_GTF.md) (added on this branch). #### PR-by-PR breakdown (child PRs into `villebro/sqllab-gtf`) 1. **Execution feature foundation** — extend the `superset-core` query options contract; share `apply_ctas`/`apply_limit` into the execution feature; add the `notify_cursor` engine-cancel seam to the shared statement loop; add `caller_owns_timeout` and `existing_query_id` hooks to `SQLExecutor`. 2. **SQL-Lab-complete executor** — CTAS/CVAS, `LimitingFactor`, `expand_data`, and the Arrow→results-backend result mode in `Database.execute()`. 3. **GTF SQL task** — `execute_sql_query` (PRIVATE) + subscription policy + cancellation; repoint `Database.execute_async()` to schedule it; retire the `AsyncQueryHandle` Celery stack. 4. **SQL Lab onto the unified API** — `/execute/` and the stop/poll path drive `Database.execute()` / `execute_async()`; retire `sql_lab.get_sql_results` and the sync/async `SqlJsonExecutor`s. 5. **Frontend realtime** — SQL Lab subscribes to `task.status` (reusing `realtime.ts`/`asyncEvent.ts`), with the poll as backstop; progress bridging. 6. **Chart-data convergence (decoupled)** — reimplement `Database.get_df` on the Layer-1 core so chart-data and SQL Lab share one warehouse path. ### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF _To be added as the frontend/realtime child PR lands._ ### TESTING INSTRUCTIONS Per child PR. End-to-end for the epic: with `GLOBAL_TASK_FRAMEWORK` enabled, run sync and async queries in SQL Lab (including multi-statement and CTAS), Stop a long-running query mid-flight (verify the warehouse query is cancelled on a cancellable engine, e.g. Postgres), let a query time out, and — with `WEBSOCKET_ENABLE` on — confirm completion arrives via `task.status` push, and with it off, via the `/query/updated_since` / `/task/status_changes` poll. ### ADDITIONAL INFORMATION - [ ] Has associated issue: - [x] Required feature flags: `GLOBAL_TASK_FRAMEWORK` (async execution); `WEBSOCKET_ENABLE` for realtime push (poll fallback otherwise) - [x] Changes UI - [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351)) - [x] Introduces new feature or API - [x] Removes existing feature or API -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
