Gwildor commented on issue #14053: URL: https://github.com/apache/superset/issues/14053#issuecomment-3588571688
We've run into this issue with Superset 5.0. We use Google BigQuery, with a maximum slot amount. This means that if there are too many queries running, BigQuery will put new queries into the queue until there is capacity to pick up more, instead of directly executing them. This is slower, but ensures stable billing, and avoids surprise bills because someone ran an expensive query by accident. When Superset triggers a few longer running queries, the UI just breaks down. The expected behaviour was that the UI would continue to work, but the dashboards would be stuck on spinners. We noticed one new user was creating around 40 charts in one day, as they were creating a few new dashboards with a lot of small elements. In itself this is a good use case of course, we want people to use our product. But the charts did not have a fixed date filter, instead it relied on the (correctly configured) default date filter of the dashboard. This means the dashboards ran fine, as they only picked up the last day, but we assume that while creating the charts, the chart was executing a query to scan the full multiyear data. These (40+) queries stacked, and at some point some queries were running voor 50+ minutes, which the user was obviously not waiting for the results on. This completely clogged our BigQuery slots, and as a result it seems (this is our assumption) all the Python threads were occupied by waiting on the results of these queries, and with no threads left over, the UI just toppled. We have autoscaling with Cloud Run, and it was scaling up (we had about 20 containers each with multiple Gunicorn threads within them), but not fast enough, so for the user the UI would just timeout after a few minutes. We had to kill the long running queries by hand, and then the UI became available again. We don't have async queries enabled, we will do a pilot to see if that helps, as well as seeing if we can properly enforce a partition filter and look into a way to properly cancel/kill queries at the BigQuery side that run longer than 2 minutes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
