Umbrella-zero opened a new issue, #36294:
URL: https://github.com/apache/superset/issues/36294
### Bug description
I'm not sure if this is a bug or a configuration issue, but here's what
we're experiencing...
We're running into an issue with the new 5.0.0 release where Celery
completely breaks when we try to use a Redis Cluster as the broker and result
backend. This is taking down two critical features for us:
1.SQL Lab async queries just fail with "Failed to start remote query on a
worker"
2.All scheduled reports stop working and won't generate
### Screenshots/recordings
Here's the relevant part from our superset_config.py
# Trying to use Redis Cluster for Celery
class CeleryConfig(object):
broker_url = "redis://our-redis-cluster-host:port" # Our cluster
connection string
result_backend = "redis://our-redis-cluster-host:port" # Same here
imports = ("superset.sql_lab", "superset.tasks.scheduler",)
worker_prefetch_multiplier = 10
task_acks_late = True
CELERY_CONFIG = CeleryConfig
# Enable async SQL Lab
SQLLAB_EXECUTOR = "superset.sql_lab.CeleryAsyncExecutor"
# Enable reports
FEATURE_FLAGS = {
"ENABLE_ASYNC_QUERIES": True,
"ASYNC_QUERIES": True,
"GLOBAL_ASYNC_QUERIES": False,
}
Troubleshooting we've tried:
We even tried using the solo pool instead of prefork to avoid any potential
cluster issues, by running:
celery --app=superset.tasks.celery_app:app worker --pool=solo -O fair -c 4
But the issue persists - Celery still fails to work properly with the Redis
Cluster setup.
What we expect:
Pretty straightforward - we expect Superset 5.0.0 to properly support Redis
Cluster as Celery backend so async queries and reports work normally.
### Superset version
5.0.0
### Python version
3.11
### Node version
16
### Browser
Chrome
### Additional context
_No response_
### Checklist
- [x] I have searched Superset docs and Slack and didn't find a solution to
my problem.
- [x] I have searched the GitHub issue tracker and didn't find a similar bug
report.
- [x] I have checked Superset's logs for errors and if I found a relevant
Python stacktrace, I included it here as text in the "additional context"
section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
