Re: [I] Sqllab stuck executing a query [superset]
michael-s-molina commented on issue #27850: URL: https://github.com/apache/superset/issues/27850#issuecomment-2061189724 @justinpark could you help? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org For additional commands, e-mail: notifications-h...@superset.apache.org
Re: [I] Sqllab stuck executing a query [superset]
aavanzyl commented on issue #27850:
URL: https://github.com/apache/superset/issues/27850#issuecomment-2060557307
Hi guys, unfortunately after 6 days of stability, the issue has returned.
This might be related to load in some way.
The behaviour explained
[above](https://github.com/apache/superset/issues/27850#issuecomment-2044267335)
seems better however, I no longer see empty execute being returned
What can I provide from my side to assist in solving the issue?
superset_config.py:
```
import logging
import os
from datetime import timedelta
from typing import Optional
from cachelib.file import FileSystemCache
from celery.schedules import crontab
from flask_caching.backends.rediscache import RedisCache
from flask_appbuilder.security.manager import AUTH_OAUTH
from s3cache.s3cache import S3Cache
logging.basicConfig()
logging.getLogger().setLevel(logging.INFO)
logger = logging.getLogger()
def get_env_variable(var_name: str, default: Optional[str] = None) -> str:
"""Get the environment variable or raise exception."""
try:
return os.environ[var_name]
except KeyError:
if default is not None:
return default
else:
error_msg = "The environment variable {} was missing,
abort...".format(
var_name
)
raise EnvironmentError(error_msg)
SECRET_KEY = get_env_variable("SECRET_KEY")
DATABASE_DIALECT = get_env_variable("DATABASE_DIALECT")
DATABASE_USER = get_env_variable("DATABASE_USER")
DATABASE_PASSWORD = get_env_variable("DATABASE_PASSWORD")
DATABASE_HOST = get_env_variable("DATABASE_HOST")
DATABASE_PORT = get_env_variable("DATABASE_PORT")
DATABASE_DB = get_env_variable("DATABASE_DB")
# The SQLAlchemy connection string.
SQLALCHEMY_DATABASE_URI = "%s://%s:%s@%s:%s/%s" % (
DATABASE_DIALECT,
DATABASE_USER,
DATABASE_PASSWORD,
DATABASE_HOST,
DATABASE_PORT,
DATABASE_DB,
)
REDIS_HOST = get_env_variable("REDIS_HOST")
REDIS_PORT = get_env_variable("REDIS_PORT")
REDIS_CELERY_DB = get_env_variable("REDIS_CELERY_DB", "0")
REDIS_RESULTS_DB = get_env_variable("REDIS_RESULTS_DB", "1")
FILTER_STATE_CACHE_DB = get_env_variable("FILTER_STATE_CACHE_DB", "2")
EXPLORE_FORM_DATA_CACHE_DB = get_env_variable("EXPLORE_FORM_DATA_CACHE_DB",
"3")
RATE_LIMIT_STORAGE_CACHE_DB =
get_env_variable("RATE_LIMIT_STORAGE_CACHE_DB", "4")
class CeleryConfig(object):
broker_url = f"redis://{REDIS_HOST}:{REDIS_PORT}/{REDIS_CELERY_DB}"
imports = (
"superset.sql_lab",
"superset.tasks",
"superset.tasks.scheduler",
)
result_backend = f"redis://{REDIS_HOST}:{REDIS_PORT}/{REDIS_RESULTS_DB}"
worker_log_level = "INFO"
CELERYD_LOG_LEVEL = "INFO"
# worker_prefetch_multiplier = 2
worker_max_tasks_per_child = get_env_variable("MAX_TASKS_PER_CHILD")
worker_max_memory_per_child = get_env_variable("MAX_MEMORY_PER_CHILD")
task_acks_late = False
task_annotations = {
"sql_lab.get_sql_results": {
"rate_limit": "100/s",
},
"email_reports.send": {
"rate_limit": "1/s",
"time_limit": 120,
"soft_time_limit": 150,
"ignore_result": True,
},
}
beat_schedule = {
"reports.scheduler": {
"task": "reports.scheduler",
"schedule": crontab(minute="*", hour="*"),
}
}
CELERY_CONFIG = CeleryConfig
# Caching
S3_CACHE_BUCKET = get_env_variable("S3_CACHE_BUCKET")
RESULTS_BACKEND = S3Cache(S3_CACHE_BUCKET, 'sql_lab_result')
FILTER_STATE_CACHE_CONFIG = {
"CACHE_TYPE": "RedisCache",
"CACHE_DEFAULT_TIMEOUT": 86400,
"CACHE_KEY_PREFIX": "superset_filter_cache",
"CACHE_REDIS_URL":
f"redis://{REDIS_HOST}:{REDIS_PORT}/{FILTER_STATE_CACHE_DB}",
}
EXPLORE_FORM_DATA_CACHE_CONFIG = {
"CACHE_TYPE": "RedisCache",
"CACHE_DEFAULT_TIMEOUT": 86400,
"CACHE_KEY_PREFIX": "superset_form_data_cache",
"CACHE_REDIS_URL":
f"redis://{REDIS_HOST}:{REDIS_PORT}/{EXPLORE_FORM_DATA_CACHE_DB}",
}
RATELIMIT_STORAGE_URL =
f"redis://{REDIS_HOST}:{REDIS_PORT}/{RATE_LIMIT_STORAGE_CACHE_DB}",
FEATURE_FLAGS = {"ALERT_REPORTS": True, "ENABLE_TEMPLATE_PROCESSING": True}
SLACK_API_TOKEN = get_env_variable("SLACK_API_TOKEN")
ALERT_REPORTS_NOTIFICATION_DRY_RUN = False
WEBDRIVER_BASEURL = "http://superset-app:8088;
# The base URL for the email report hyperlinks.
WEBDRIVER_BASEURL_USER_FRIENDLY = "https://superset-dw-ib.naked.insure;
WEBDRIVER_TYPE = "firefox"
WEBDRIVER_OPTION_ARGS = ["--headless"]
MAPBOX_API_KEY = get_env_variable("MAPBOX_API_KEY")
#
Re: [I] Sqllab stuck executing a query [superset]
aavanzyl commented on issue #27850: URL: https://github.com/apache/superset/issues/27850#issuecomment-2049024184 We have had 48 hours of stability with the issue not poping up. Its safe to say its solved. Thank you @michael-s-molina and team for the swift assistance. I appreciate the effort on your side. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org For additional commands, e-mail: notifications-h...@superset.apache.org
Re: [I] Sqllab stuck executing a query [superset]
aavanzyl closed issue #27850: Sqllab stuck executing a query URL: https://github.com/apache/superset/issues/27850 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org For additional commands, e-mail: notifications-h...@superset.apache.org
Re: [I] Sqllab stuck executing a query [superset]
michael-s-molina commented on issue #27850: URL: https://github.com/apache/superset/issues/27850#issuecomment-2044774982 Thanks @aavanzyl. Let us know if things are working after your tests so we can close the issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org For additional commands, e-mail: notifications-h...@superset.apache.org
Re: [I] Sqllab stuck executing a query [superset]
aavanzyl commented on issue #27850: URL: https://github.com/apache/superset/issues/27850#issuecomment-2044267335 @michael-s-molina I have deployed 3.1.2 as asked. The issue currently is not showing, will monitor for a couple of days and let you know. Thank you for the response on my issue. For further information, I can see that when clicking on run, it calls the `execute` API. This in turn takes a while to respond and was an empty response when broken. When things are working as expected, the `execute` API responds with a json object. I also noticed that the `updated_since` API call started before the `execute` had a chance to receive a response, leading me to believe that this might contribute to the issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org For additional commands, e-mail: notifications-h...@superset.apache.org
Re: [I] Sqllab stuck executing a query [superset]
michael-s-molina commented on issue #27850: URL: https://github.com/apache/superset/issues/27850#issuecomment-2034709023 Hi @aavanzyl. Could you check if this problem happens in 3.1.2 which was released today? @justinpark for awareness. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org For additional commands, e-mail: notifications-h...@superset.apache.org
[I] Sqllab stuck executing a query [superset]
aavanzyl opened a new issue, #27850:
URL: https://github.com/apache/superset/issues/27850
### Bug description
When using SQL Lab to execute a query such as `SELECT * from claims.claim
limit 1`, the UI shows `Pending` and never starts increasing the execution
time.
The following is logged in the browser:
Execute:
`
{
"query": {
"changed_on": "2024-04-02T07:16:18.493787",
"dbId": 1,
"db": "Amazon Redshift (Prod)",
"endDttm": null,
"errorMessage": null,
"executedSql": null,
"id": "9E6eWGcfB",
"queryId": 26748,
"limit": 1,
"limitingFactor": "QUERY",
"progress": 0,
"rows": null,
"schema": null,
"ctas": false,
"serverId": 26748,
"sql": "SELECT * from claims.claim limit 1\n",
"sqlEditorId": "1773",
"startDttm": 1712042178240.671000,
"state": "pending",
"tab": "Untitled Query 1",
"tempSchema": null,
"tempTable": null,
"userId": 1,
"user": "Anton van Zyl",
"resultsKey": null,
"trackingUrl": null,
"extra": {}
}
}
`
We get multiple updated since messages back such as:
`{
"result": [
{
"changed_on": "2024-04-02T07:16:18.242144",
"ctas": false,
"db": "Amazon Redshift (Prod)",
"dbId": 1,
"endDttm": null,
"errorMessage": null,
"executedSql": null,
"extra": {},
"id": "9E6eWGcfB",
"limit": 1000,
"limitingFactor": "UNKNOWN",
"progress": 0,
"queryId": 26748,
"resultsKey": null,
"rows": null,
"schema": null,
"serverId": 26748,
"sql": "SELECT * from claims.claim limit 1\n",
"sqlEditorId": "1773",
"startDttm": "1712042178240.671000",
"state": "pending",
"tab": "Untitled Query 1",
"tempSchema": null,
"tempTable": null,
"trackingUrl": null,
"user": "Anton van Zyl",
"userId": 1
}
]
}
`
But never receive and `/results` call back.
My current remedy is to restart the instance and cache, which seems to
intermittently solve the issue.
This issue only started recently, with us updating to 3.1.0 and then 3.1.1.
I have checked the logs, nothing pops up as out of the ordinary.
### How to reproduce the bug
Go to SQL Lab and run any query will result in the query being stuck in
`Pending`
### Screenshots/recordings
The current UI state being shown to the end user:
### Superset version
3.1.1
### Python version
I don't know
### Node version
I don't know
### Browser
Chrome
### Additional context
We are running the latest 3.1.1 superset from a docker container hosted on
ECS in AWS. The instance size has 12gigs of memory and connects to a Redshift
database.
Superset Caching is set up with Redis 7.1.0 running on elasticache in AWS.
### Checklist
- [X] I have searched Superset docs and Slack and didn't find a solution to
my problem.
- [X] I have searched the GitHub issue tracker and didn't find a similar bug
report.
- [X] I have checked Superset's logs for errors and if I found a relevant
Python stacktrace, I included it here as text in the "additional context"
section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org
For additional commands, e-mail: notifications-h...@superset.apache.org
