john-bodley opened a new pull request #10280: URL: https://github.com/apache/incubator-superset/pull/10280
### SUMMARY At Airbnb we had an incident with Celery where tasks weren't being picked up and the CPU was pegged at close to 100%. We configure Celery with the `gevent` pool and the only workaround to remedy the problem was to restart the workers sans `gevent`. We haven't been able to track down the root cause of the problem, though a Google search surfaced a similar issue ( [celery/celery/issues#4999](https://github.com/celery/celery/issues/4999)) which I believe was fixed in `4.4.2` (which is the version Superset currently uses). The issue seems to be that there are no [release note](https://docs.celeryproject.org/en/latest/changelog.html) associated with said release nor does that tag exist within the Celery GitHub repo. Given there have been a number of recent releases (which mostly look like bug fixes) I felt it was prudent that we bumped the version of Celery to the latest stable version. Note we use a MySQL database for Superset and historically have been using the `mysqlclient` DBAPI but we're looking into using `PyMySQL` which fully supports `gevent`. We're not sure if this was the issue, but the Celery workers poll certain async engines, i.e., Presto, every few seconds per task and thus having a fully `gevent` compliant DBAPI may also be benefical. ### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF <!--- Skip this if not applicable --> ### TEST PLAN CI. ### ADDITIONAL INFORMATION <!--- Check any relevant boxes with "x" --> <!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue --> - [ ] Has associated issue: - [ ] Changes UI - [ ] Requires DB Migration. - [ ] Confirm DB Migration upgrade and downgrade tested. - [ ] Introduces new feature or API - [ ] Removes existing feature or API ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
