paul-bormans commented on issue #19192: URL: https://github.com/apache/airflow/issues/19192#issuecomment-1128824014
We are seeing the same issue running v2.2.4 on K8S - Using the LocalExecutor - Most tasks are using operator: KubernetesPodOperator and some of these are streaming apps so run infinitely. We first increased the setting "scheduler_health_check_threshold" from default 30 to 180 (i.e. seconds), that prevented at least that Airflow kills tasks because of the timeout (somehow related?).  I suspected too much logging being written maybe but there is only a few GB of logging and with log-groomer running (enabled by default in the chart) retention should be fine. Also i couldn't find any DB issue's, it's rather small even; few MB's only. For now the issue is resolved by a simple redeploy, btw reusing the DB. Paul -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
