[
https://issues.apache.org/jira/browse/AIRFLOW-2201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16418144#comment-16418144
]
Jake Boggan edited comment on AIRFLOW-2201 at 3/28/18 9:27 PM:
---------------------------------------------------------------
I am having an identical problem with 1.8.2 - setting
{{worker_refresh_batch_size = 0 }}
doesn't seem to help as some have suggested.
was (Author: jakeboggan):
I am having an identical problem with 1.8.2 - setting
`worker_refresh_batch_size = 0` doesn't seem to help as some have suggested.
> Scheduler processes dags every second
> -------------------------------------
>
> Key: AIRFLOW-2201
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2201
> Project: Apache Airflow
> Issue Type: Bug
> Affects Versions: 1.9.0
> Reporter: John Arnold
> Priority: Critical
>
> Building latest master.
>
> The scheduler appears to be running continuously, not respecting
> scheduler_heartbeat_sec = 30
>
> Logs are filled up with messages like:
> Mar8 23:20:36 netdocker1-eastus2 INFO :[2018-03-08 23:20:36,826]
> \{dag_processing.py:468} INFO - Processor for
> /var/lib/airflow/dags/test_nt.py finished
> Mar8 23:20:36 netdocker1-eastus2 INFO :[2018-03-08 23:20:36,831]
> \{dag_processing.py:543} INFO - Started a process (PID: 64) to generate tasks
> for /var/lib/airflow/dags/test_nt.py
> Mar8 23:20:37 netdocker1-eastus2 INFO :[2018-03-08 23:20:37,834]
> \{dag_processing.py:468} INFO - Processor for
> /var/lib/airflow/dags/test_nt.py finished
> Mar8 23:20:37 netdocker1-eastus2 INFO :[2018-03-08 23:20:37,838]
> \{dag_processing.py:543} INFO - Started a process (PID: 66) to generate tasks
> for /var/lib/airflow/dags/test_nt.py
> Mar8 23:20:38 netdocker1-eastus2 INFO :[2018-03-08 23:20:38,842]
> \{dag_processing.py:468} INFO - Processor for
> /var/lib/airflow/dags/test_nt.py finished
> Mar8 23:20:38 netdocker1-eastus2 INFO :[2018-03-08 23:20:38,846]
> \{dag_processing.py:543} INFO - Started a process (PID: 68) to generate tasks
> for /var/lib/airflow/dags/test_nt.py
> Mar8 23:20:39 netdocker1-eastus2 INFO :[2018-03-08 23:20:39,850]
> \{dag_processing.py:468} INFO - Processor for
> /var/lib/airflow/dags/test_nt.py finished
> Mar8 23:20:39 netdocker1-eastus2 INFO :[2018-03-08 23:20:39,856]
> \{dag_processing.py:543} INFO - Started a process (PID: 70) to generate tasks
> for /var/lib/airflow/dags/test_nt.py
> Mar8 23:20:40 netdocker1-eastus2 INFO :[2018-03-08 23:20:40,863]
> \{dag_processing.py:468} INFO - Processor for
> /var/lib/airflow/dags/test_nt.py finished
> Mar8 23:20:40 netdocker1-eastus2 INFO :[2018-03-08 23:20:40,867]
> \{dag_processing.py:543} INFO - Started a process (PID: 72) to generate tasks
> for /var/lib/airflow/dags/test_nt.py
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
