villebro opened a new issue, #44174: URL: https://github.com/apache/superset/issues/44174
*Please make sure you are familiar with the SIP process documented* [here](https://github.com/apache/superset/issues/5602). The SIP will be numbered by a committer upon acceptance. ## [SIP] Proposal to make Async and Realtime by Default ### Motivation Superset has added many valuable capabilities over the years that become available only when async infrastructure is configured: thumbnails, screenshots, Alerts and Reports, async chart data, SQL Lab async execution, exports, cache warmup, and task observability. However, because the required infrastructure remains optional, these capabilities are not consistently available by default. Features that should feel like core Superset behavior can instead depend on whether operators have enabled and wired the right async services. This optionality also creates major developer friction by requiring parallel sync and async paths, increasing maintenance work, and slowing down changes that should apply across the product. This SIP proposes making Redis/Valkey, Celery, and WebSockets part of the supported baseline so these capabilities can be treated as standard product behavior instead of optional deployment extras. The primary benefit is a richer and more dependable experience for Superset users. Long-running work can happen outside web workers, dashboards and SQL Lab can stay responsive under load, and users can track progress, retry failed work, and cancel stuck or expensive workloads instead of waiting for request timeouts. Operators also get clearer separation between interactive web capacity, background worker capacity, coordination, and realtime delivery, which reduces waste from long-held web requests and repeated polling. Centralizing async functionality makes platform improvements apply across domains: better retry, timeout, cancellation, heartbeat, or reaping behavior can benefit every migrated feature. It also reduces the maintenance cost of divergent sync and async paths, letting contributors spend more effort on user-visible improvements. Several areas can benefit immediately: - Chart data can build on the Global Async Queries migration to the Global Task Framework (GTF) and use shared task semantics for async results. - SQL Lab can continue its active GTF consolidation work and use the same task primitives as chart data and other long-running workflows. - Thumbnails, screenshots, Alerts and Reports, cache warmup, and dashboard Excel export can share task orchestration, status, cancellation, retry, locking, and recovery behavior. - Long-running work can consistently move out of web worker processes. - WebSockets can replace repeated long-poll loops with immediate task updates and a reusable realtime integration model. Docker Compose, the Helm chart, and the Kubernetes Operator already support these infrastructure components. Any remaining packaging, configuration, or readiness gaps should be addressed as part of the SIP implementation so the required topology is easy to run through the officially supported deployment paths. ### Proposed Change This SIP makes Redis/Valkey, Celery, and WebSockets required for supported Superset deployments, and routes all long-running core workflows through GTF. This would be a Superset 8.0 breaking change and would not affect the in-progress Superset 7.0 release. The concrete changes are: - Redis/Valkey becomes the required coordinator and realtime backend. Valkey is compatible with the Redis protocol for the capabilities discussed here. Other supported cache backends may still be used for caching, but GTF and the WebSocket server need Redis/Valkey capabilities such as Streams, Pub/Sub, and locking primitives. - Celery workers and Celery beat or equivalent scheduling become required. Celery becomes the execution engine for GTF-managed work. - The Superset WebSocket server becomes required for browser-facing async updates. REST APIs remain available for initial reads, cursor-based catch-up, reconnect recovery, tests, and non-browser clients. - GTF becomes the shared product and developer contract for long-running work: status, progress, results, cancellation, retries, dependencies, heartbeats, reapers, and worker recovery. - GTF's synchronous execution mode is retired. Synchronous helper code may still exist below the task layer, but framework-declared tasks schedule through Celery and long-running product workflows run through workers. - Shared frontend primitives are added for awaiting task IDs, subscribing to task status, catching up from a cursor, and rendering common task states. ### New or Changed Public Interfaces #### Deployment and Configuration Supported deployments will include Redis/Valkey, Celery workers, Celery beat or equivalent scheduling, and the Superset WebSocket server. Docker Compose, the Helm chart, and the Kubernetes Operator should expose these services as first-class supported components with clear defaults, health checks, routing, configuration examples, and upgrade notes. The following configuration areas will move toward always-on async infrastructure: - `CELERY_CONFIG` becomes required and validated for supported deployments. - `DISTRIBUTED_COORDINATION_CONFIG` becomes required and must point at a Redis/Valkey backend. - `WEBSOCKET_ENABLE` becomes unnecessary once WebSockets are part of the supported baseline; during the transition it may remain as a compatibility setting or warning mechanism. - WebSocket URL and authentication settings must be validated when the app starts. - `GLOBAL_TASK_FRAMEWORK` becomes permanently enabled. - `GLOBAL_ASYNC_QUERIES` and related async chart-data flags become permanently enabled or evolve into narrower product controls. - SQL Lab async configuration such as per-database `allow_run_async`, frontend `runAsync`, and `SQLLAB_FORCE_RUN_ASYNC` should be reviewed and either retired or redefined as product behavior controls rather than execution-path selectors. This SIP should also decide whether Redis/Valkey is required only for coordination and realtime delivery, or also as the required Celery broker and results backend. See Open Questions. #### APIs and Extension Points GTF task APIs should become the canonical interface for async status, progress, cancellation, and result settlement. Feature APIs that start long-running work should return GTF task identifiers and enough cursor metadata for realtime subscription and catch-up. Existing feature-specific status endpoints should converge onto GTF once their features migrate. Some read endpoints may remain as resource-specific views of state, while orchestration behavior moves into the shared task layer. The task decorator and task wrapper APIs should clearly document asynchronous production semantics. Framework-declared tasks should be async-only. Direct calls should either be disallowed or reserved for internal test helpers that are separate from production behavior. For chart data and SQL Lab specifically, the SIP should define which public API calls can still return immediate results for cache hits or bounded work, and which calls should always schedule async work. The guiding rule should be that long-running execution runs through workers. #### User Interface Task List should become the shared realtime surface for user-visible async work. SQL Lab, chart data, exports, reports, and screenshots should use consistent task status behavior instead of separate per-feature polling loops. Feature UIs can still present domain-specific labels and controls, but they should consume shared GTF task state and shared WebSocket subscription primitives. ### New dependencies No major new application library dependency is expected. Superset already has Celery-based task execution, Redis/Valkey coordination patterns, and a WebSocket server package. The dependency change is operational: the runtime services listed above become required for supported deployments. Implementation should confirm supported Redis/Valkey versions, broker/result backend combinations, TLS/Sentinel support, and operational expectations for Docker Compose, Helm, and the Kubernetes Operator. ### Migration Plan and Compatibility This should be treated as a staged platform modernization that culminates in Superset 8.0, with a clear deprecation period for deployments and extensions that rely on optional infrastructure or synchronous framework execution. The in-progress Superset 7.0 release would not be affected by the breaking-change portion of this SIP. ### Rejected Alternatives - Continue supporting optional async infrastructure. This preserves a low-infrastructure deployment path, but it keeps core features split across different operating modes. - Keep sync and async execution in GTF. This was useful while Celery remained optional, but GTF can provide a clearer contract by having one production execution model for framework tasks. - Require only part of the stack. Making only Celery, only Redis/Valkey, or only WebSockets mandatory would standardize one layer while leaving others feature-specific. - Migrate tasks opportunistically without a mandatory topology. Incremental migration improves individual features, but it does not create the shared baseline needed to converge duplicate paths. - Keep separate async systems per feature. Per-feature systems can solve local problems, but GTF creates a better foundation for task history, progress, cancellation, retries, dependency graphs, worker recovery, and realtime UI. - Depend on WebSocket pub/sub alone for correctness. WebSockets should be mandatory as the active realtime transport, but clients still need durable state and catch-up semantics. ### Reference Material - SIP issue template: `.github/ISSUE_TEMPLATE/sip.md` - GTF developer docs: `docs/developer_docs/extensions/tasks.md` - Superset default config: `superset/config.py` - GAQ to GTF update notes: `UPDATING.md` - GTF introduction: https://github.com/apache/superset/pull/36368 - GAQ migration to GTF: https://github.com/apache/superset/pull/43407 - Batteries-included Docker image: https://github.com/apache/superset/pull/44100 - Dashboard Excel export async discussion: https://github.com/apache/superset/pull/43805#issuecomment-5636231335 - SQL Lab GTF migration tracker: https://github.com/apache/superset/pull/43928 - SQL Lab GTF task PR: https://github.com/apache/superset/pull/43933 - SQL Lab unified executor and GTF migration PR: https://github.com/apache/superset/pull/43942 - SQL Lab realtime settlement PR: https://github.com/apache/superset/pull/44032 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
