James Meickle created AIRFLOW-2684:
--------------------------------------
Summary: scheduler_heartbeat_sec is not respected
Key: AIRFLOW-2684
URL: https://issues.apache.org/jira/browse/AIRFLOW-2684
Project: Apache Airflow
Issue Type: Bug
Components: scheduler
Affects Versions: 1.10
Reporter: James Meickle
In our staging environment, we leave Airflow debug logs on. We're generating
immense amounts of logs because for both our 1.9 cluster and 1.10 cluster,
scheduler_heartbeat_sec is not being respected. It's set to 5 in both
environments, but produces logs like this every second:
```
airflow_scheduler-stdout.log: [2018-06-27 13:47:20,272] \{{jobs.py:1685}} DEBUG
- Ran scheduling loop in 0.00 seconds
airflow_scheduler-stdout.log: [2018-06-27 13:47:20,273] \{{jobs.py:1686}} DEBUG
- Sleeping for 1.00 seconds
airflow_scheduler-stdout.log: [2018-06-27 13:47:21,273] \{{jobs.py:1607}} DEBUG
- Starting Loop...
airflow_scheduler-stdout.log: [2018-06-27 13:47:21,274] \{{jobs.py:1627}} INFO
- Heartbeating the process manager
airflow_scheduler-stdout.log: [2018-06-27 13:47:21,274]
\{{dag_processing.py:523}} DEBUG - Queuing the following files for processing:
airflow_scheduler-stdout.log: [2018-06-27 13:47:21,275] \{{jobs.py:1662}} INFO
- Heartbeating the executor
airflow_scheduler-stdout.log: [2018-06-27 13:47:21,275]
\{{base_executor.py:103}} DEBUG - 0 running task instances
airflow_scheduler-stdout.log: [2018-06-27 13:47:21,275]
\{{base_executor.py:104}} DEBUG - 0 in queue
airflow_scheduler-stdout.log: [2018-06-27 13:47:21,276]
\{{base_executor.py:105}} DEBUG - 32 open slots
airflow_scheduler-stdout.log: [2018-06-27 13:47:21,276]
\{{base_executor.py:132}} DEBUG - Calling the <class
'airflow.executors.celery_executor.CeleryExecutor'> sync method
airflow_scheduler-stdout.log: [2018-06-27 13:47:21,276]
\{{celery_executor.py:80}} DEBUG - Inquiring about 0 celery task(s)
airflow_scheduler-stdout.log: [2018-06-27 13:47:21,277] \{{jobs.py:1685}} DEBUG
- Ran scheduling loop in 0.00 seconds
airflow_scheduler-stdout.log: [2018-06-27 13:47:21,277] \{{jobs.py:1686}} DEBUG
- Sleeping for 1.00 seconds
```
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
