Yann-OAF opened a new pull request #13116: URL: https://github.com/apache/superset/pull/13116
### SUMMARY This PR adds an optional kubernetes `Deployment` to run the Celery beat, which is needed in order to trigger the scheduled alerts and reports (as per the excellent unpublished guide at https://github.com/apache/superset/blob/4fa3b6c7185629b87c27fc2c0e5435d458f7b73d/docs/src/pages/docs/installation/email_reports.mdx). It is off by default, and needs to be enabled with `supersetBeat.enabled`. The new pod is defined pretty much exactly like the worker pod except that: * `replicas` is always 1, since this needs to be a singleton * The command is different Note that for the chart to be able to execute reports, we still need to check all the other boxes, in particular: * Make sure one webdriver is installed - for this, the commands mentioned in the guide above can be added as a custom `supersetWorker.command` (this is only needed in the worker container) with the existing chart, with something like this in your `values.yaml`: ``` supersetWorker: command: - /bin/sh - -c - | # Install chrome webdriver # See https://github.com/apache/superset/blob/4fa3b6c7185629b87c27fc2c0e5435d458f7b73d/docs/src/pages/docs/installation/email_reports.mdx apt update wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb apt install -y --no-install-recommends ./google-chrome-stable_current_amd64.deb wget https://chromedriver.storage.googleapis.com/88.0.4324.96/chromedriver_linux64.zip unzip chromedriver_linux64.zip chmod +x chromedriver mv chromedriver /usr/bin apt autoremove -yqq --purge apt clean rm -f google-chrome-stable_current_amd64.deb chromedriver_linux64.zip # Run . {{ .Values.configMountPath }}/superset_bootstrap.sh; celery --app=superset.tasks.celery_app:app worker ``` * Perform all the required Celery setup by overriding the `superset_config.py`. This is now possible with the latest chart from `master` by specifying overrides in your `values.yaml` such as: ``` configOverrides: celery_conf: | from celery.schedules import crontab class CeleryConfig(object): BROKER_URL = f"redis://{env('REDIS_HOST')}:{env('REDIS_PORT')}/0" CELERY_IMPORTS = ('superset.sql_lab', ) CELERY_RESULT_BACKEND = f"redis://{env('REDIS_HOST')}:{env('REDIS_PORT')}/0" CELERY_ANNOTATIONS = {'tasks.add': {'rate_limit': '10/s'}} CELERY_IMPORTS = ('superset.sql_lab', "superset.tasks", "superset.tasks.thumbnails", ) CELERY_ANNOTATIONS = { 'sql_lab.get_sql_results': { 'rate_limit': '100/s', }, 'email_reports.send': { 'rate_limit': '1/s', 'time_limit': 600, 'soft_time_limit': 600, 'ignore_result': True, }, } CELERYBEAT_SCHEDULE = { 'reports.scheduler': { 'task': 'reports.scheduler', 'schedule': crontab(minute='*', hour='*'), }, 'reports.prune_log': { 'task': 'reports.prune_log', 'schedule': crontab(minute=0, hour=0), }, 'cache-warmup-hourly': { 'task': 'cache-warmup', 'schedule': crontab(minute='*/30', hour='*'), 'kwargs': { 'strategy_name': 'top_n_dashboards', 'top_n': 10, 'since': '7 days ago', }, } } CELERY_CONFIG = CeleryConfig reports: | EMAIL_PAGE_RENDER_WAIT = 60 WEBDRIVER_BASEURL = "http://superset:8088/"; WEBDRIVER_BASEURL_USER_FRIENDLY = "https://superset.qa.oneacrefund.org/"; WEBDRIVER_TYPE= "chrome" WEBDRIVER_OPTION_ARGS = [ "--force-device-scale-factor=2.0", "--high-dpi-support=2.0", "--headless", "--disable-gpu", "--disable-dev-shm-usage", "--no-sandbox", "--disable-setuid-sandbox", "--disable-extensions", ] feature_flags: | import ast FEATURE_FLAGS = { "ALERT_REPORTS": True, } EMAIL_NOTIFICATIONS = ast.literal_eval(os.getenv("EMAIL_NOTIFICATIONS", "True")) SMTP_HOST = os.getenv("SMTP_HOST","localhost") SMTP_STARTTLS = ast.literal_eval(os.getenv("SMTP_STARTTLS", "True")) SMTP_SSL = ast.literal_eval(os.getenv("SMTP_SSL", "False")) SMTP_USER = os.getenv("SMTP_USER","superset") SMTP_PORT = os.getenv("SMTP_PORT",25) SMTP_PASSWORD = os.getenv("SMTP_PASSWORD","superset") SMTP_MAIL_FROM = os.getenv("SMTP_MAIL_FROM","[email protected]") SLACK_API_TOKEN = os.getenv("SLACK_API_TOKEN",None) extraSecretEnv: SLACK_API_TOKEN: xoxb-... SMTP_PASSWORD: ... extraEnv: SMTP_HOST: smtp.gmail.com SMTP_USER: ... SMTP_PORT: "587" SMTP_MAIL_FROM: ... ``` The above would probably need to be added to the doc together with PR #13104 for the Kubernetes case. ### TEST PLAN * Update `values.yaml` with `supersetBeat.enabled: true` and upgrade your chart release * See your new pod running! ### ADDITIONAL INFORMATION - [x] Has associated issue: #13104 - [ ] Changes UI - [ ] Requires DB Migration. - [ ] Confirm DB Migration upgrade and downgrade tested. - [ ] Introduces new feature or API - [ ] Removes existing feature or API ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
