villebro opened a new pull request, #43424:
URL: https://github.com/apache/superset/pull/43424
### SUMMARY
**Step 4 of the GAQ→GTF epic (#43407): the chart-data cutover.** Async
chart-data
requests now run entirely on the **Global Task Framework** instead of the
bespoke
Global Async Queries (GAQ) plumbing — one GTF task per `QueryObject`, a
lightweight
status poll, and a client re-request on completion. This subsumes the old
"frontend re-request" and "remove query-context wrapper" steps into one
cutover.
Because the whole `gaq-to-gtf` branch merges to `master` as a unit, this
takes the
**breaking** simplifications now rather than carrying compatibility shims:
**Backend**
- **Fan-out:** `/chart/data` (async) schedules one **SHARED** GTF task per
`QueryObject`,
keyed by its `query_cache_key` (safe cross-user dedup — the key encodes
RLS/impersonation). Contribution queries `depends_on` the totals query's
task and read
its cached result to normalize. Dedicated task type
**`superset.query_object_v1`**.
The 202 body is `{"task_ids": [...]}`.
- **No coordinator task, no `qc-<hash>` wrapper:** the client aggregates the
query tasks'
own honest statuses itself. Removed the `/api/v1/chart/data/<cache_key>`
replay
endpoint, `QueryContextCacheLoader`, and `result_url`.
- **New poll primitive `GET /api/v1/task/status_changes`:** returns
`{statuses: {uuid: {status, progress}}, cursor}` for tasks the caller can
see
(`TaskFilter`-scoped), changed since an opaque server-issued cursor
(`changed_on >=`,
so no transition is missed; re-delivery is idempotent). No cursor =
baseline
(empty + current watermark; never dumps all tasks). Optional `task_type`
filter.
- **Embedded-guest visibility:** guests have no `ab_user` id, so they now
subscribe to
tasks by a stable, token-derived `guest_key` (HMAC over guest-token claims
keyed with
`SECRET_KEY`); `TaskFilter` honors it. Survives SHARED-scope dedup (two
equivalent
guests collapse to one task and both subscribe). Adds a nullable
`task_subscribers.guest_key` column (reversible migration; `user_id`
relaxed to
nullable — a subscriber is exactly one of `user_id`/`guest_key`).
- **GAQ rip-out:** deleted `AsyncQueryManager`, the `/api/v1/async_event/`
REST API, the
factory, and the legacy `load_chart_data_into_cache` Celery task. Removed
the GAQ
JWT/cookie/transport/stream config **and** the dedicated
`GLOBAL_ASYNC_QUERIES_CACHE_BACKEND` — coordination now runs on
`DISTRIBUTED_COORDINATION_CONFIG` exclusively. Kept the
`GLOBAL_ASYNC_QUERIES` flag and
`GLOBAL_ASYNC_QUERIES_POLLING_DELAY`.
**Frontend**
- Rewrote `asyncEvent.ts` off the GAQ firehose (poll + WebSocket +
`result_url`) to a
single shared cursor-poll of `status_changes` filtered to
`superset.query_object_v1`.
`waitForAsyncData(job, refetch, signal)` awaits the job's `task_ids` to
all reach
`SUCCESS`, then calls `refetch()` (re-issues the original request — served
synchronously from the now-warm per-query DATA cache); any terminal
non-success
rejects; abort cancels the tasks via `/api/v1/task/<uuid>/cancel`.
- Threaded the re-request through `chartAction.exploreJSON`, `FilterValue`,
`FiltersConfigForm`, `DrillByModal`, `ChartVersionPreview`, and
`StatefulChart`'s
`handleAsyncChartData` hook.
**Auth / transport notes**
- Polling and cancel use the caller's normal Superset session
(`@protect()`); embedded
guests use their existing guest token. The GAQ JWT is no longer needed.
- Real-time WebSocket push (previously via the GAQ firehose) is
intentionally retired
here; async chart-data is **polling-only** on the branch. A GTF-native
WebSocket
transport returns in a later epic step with its own channel-token service.
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
N/A — no user-visible UI change (async chart loading behaves the same; the
transport
underneath changed).
### TESTING INSTRUCTIONS
With `GLOBAL_ASYNC_QUERIES` enabled (which also enables
`GLOBAL_TASK_FRAMEWORK`) and
`DISTRIBUTED_COORDINATION_CONFIG` pointed at Redis/Valkey:
- Load a dashboard with multi-query + contribution charts; confirm each
chart resolves
(tasks appear in the Task List as `superset.query_object_v1`, then the
chart renders).
- Confirm a second load short-circuits to a synchronous 200 (warm cache).
- Press Stop mid-load; confirm the outstanding tasks are cancelled.
- Load an embedded (guest-token) dashboard; confirm async charts resolve.
Automated: `pytest tests/unit_tests/tasks tests/unit_tests/coordination
tests/unit_tests/daos/test_tasks.py`
(393 pass locally); frontend `npm run test -- asyncEvent chartActions
StatefulChart FilterValue`.
Migration up/down/up verified reversible on SQLite.
### ADDITIONAL INFORMATION
- [ ] Has associated issue:
- [x] Required feature flags: `GLOBAL_ASYNC_QUERIES` (auto-enables
`GLOBAL_TASK_FRAMEWORK` in a later step)
- [x] Changes UI
- [x] Includes DB Migration (follow approval process in
[SIP-59](https://github.com/apache/superset/issues/13351))
- [x] Migration is atomic, supports rollback & is backwards-compatible
- [x] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [x] Introduces new feature or API (`/api/v1/task/status_changes`;
per-`QueryObject` async tasks)
- [x] Removes existing feature or API (`/api/v1/async_event/`,
`AsyncQueryManager`, `qc-<hash>` replay, `GLOBAL_ASYNC_QUERIES_CACHE_BACKEND` +
GAQ JWT/transport config)
---
**Follow-ups (not this PR):** rip the now-dead
`cache_query_context`/`qc-<hash>` `cache_key()`
write path from `QueryContextProcessor` (harmless dead write today; touches
the
`/chart/data` response shape, so done separately); `async_mode` per-request
opt-in +
`GLOBAL_ASYNC_QUERIES` → `GLOBAL_TASK_FRAMEWORK` auto-enable (next step);
GTF-native
WebSocket transport.
Targets the `gaq-to-gtf` feature branch (part of #43407), not `master`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
