GitHub user dosubot[bot] added a comment to the discussion: Preventing Duplicate Queries During Asynchronous Chart Data Processing
<!-- Greeting --> Hi @kch8306! I'm [Dosu](https://go.dosu.dev/dosubot) and I’m helping the superset team. <!-- Answer --> Superset 3.1 does not have built-in deduplication or distributed locking for async chart queries—when multiple users hit a cold cache, each worker will execute the same query independently, causing the duplicate load you described. Your approach of using a CacheKey table as a lock is a valid pattern for serializing work, and it would prevent duplicate queries at the application level. However, be aware of potential race conditions and deadlocks, especially if a worker crashes or times out before releasing the lock. You may want to add a TTL or cleanup mechanism for stale locks. A more common approach in distributed systems is to use a distributed lock or mutex (for example, via Redis SETNX or Redlock) keyed by the cache key, which avoids the need for a database table and is generally more robust in high-concurrency environments. Superset already uses Redis for caching, so implementing a Redis-based lock could be more idiomatic and performant. If you stick with the DB table approach, make sure to handle edge cases like lock cleanup and error recovery. There’s no evidence of a built-in or recommended deduplication mechanism for async chart queries in Superset as of 3.1, so your solution would be a novel extension to the current architecture. For more on how caching and cache keys work in Superset, see [this discussion](https://github.com/apache/superset/discussions/33944) and [the async query code](https://github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset/tasks/async_queries.py). <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/c8ad7305-5bfb-4dd3-b037-a0d7f8270948?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/c8ad7305-5bfb-4dd3-b037-a0d7f8270948?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/c8ad7305-5bfb-4dd3-b037-a0d7f8270948?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/c8ad7305-5bfb-4dd3-b037-a0d7f8270948?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/c8ad7305-5bfb-4dd3-b037-a0d7f8270948?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/c8ad7305-5bfb-4dd3-b037-a0d7f8270948?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/c8ad7305-5bfb-4dd3-b037-a0d7f8270948?feedback_type=other)</sup> [](https://app.dosu.dev/a28d3c7e-a9d3-459e-9fb6-3a6f9ff4f357/ask?utm_source=github)& nbsp;[](https://go.dosu.dev/discord-bot) [](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/apache/superset/discussions/34316) GitHub link: https://github.com/apache/superset/discussions/34316#discussioncomment-13889955 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
