[ 
https://issues.apache.org/jira/browse/AIRFLOW-5506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17001848#comment-17001848
 ] 

Matsubara Yuya edited comment on AIRFLOW-5506 at 12/22/19 9:39 AM:
-------------------------------------------------------------------

Using with the KubernetesExecutor, I am facing the same issue, too.

When running over 4000 tasks inside one DAG, DAG does not run properly. 
Triggering DAG, DAG state come into Running state. But, not scheduling tasks 
and DAG is staying into Running state eternally ( DAG Runs = 'running', Recent 
Tasks state = 'none' on the WebUI) .

Reverting to 1.10.5 not fixes the issue. 

*The scheduler logs* ( kubectl logs airflow-85bfcd8c86-v4lps scheduler ) shows 
repeatedly:

[2019-12-22 08:42:42,278] {{scheduler_job.py:210}} WARNING - Killing PID XXXXXX.

*DAG log* :

[2019-12-22 09:09:07,007] {{scheduler_job.py:1507}} INFO - DAG(s) 
dict_keys(['d20manytest']) retrieved from /root/airflow/dags/d20.py
 [2019-12-22 09:09:07,118] {{scheduler_job.py:1212}} INFO - Processing 
d20manytest
 [2019-12-22 09:09:07,133] {{scheduler_job.py:1225}} INFO - Created None
 [2019-12-22 09:09:07,149] {{scheduler_job.py:690}} INFO - Examining DAG run 
<DagRun d20manytest @ 2019-12-20 14:00:00+00:00: 
scheduled__2019-12-20T14:00:00+00:00, externally triggered: False>
 [2019-12-22 09:11:05,188] {{logging_mixin.py:90}} INFO - [2019-12-22 
09:11:05,187] {{settings.py:175}} INFO - settings.configure_orm(): Using pool 
settings. pool_size=0, max_overflow=10, pool_recycle=1800, pid=21046
 [2019-12-22 09:11:05,190] {{scheduler_job.py:142}} INFO - Started process 
(PID=21046) to work on /root/airflow/dags/d20.py
 [2019-12-22 09:11:05,192] {{scheduler_job.py:1495}} INFO - Processing file 
/root/airflow/dags/d20.py for tasks to queue
 [2019-12-22 09:11:05,193] {{logging_mixin.py:90}} INFO - [2019-12-22 
09:11:05,193] {{dagbag.py:90}} INFO - Filling up the DagBag from 
/root/airflow/dags/d20.py

Any idea why would that happen?

 


was (Author: yuya):
Using with the KubernetesExecutor, I am facing the same issue, too.

When running over 4000 tasks inside one DAG, DAG does not run properly. 
Triggering DAG, DAG state come into Running state. But, not scheduling tasks 
and DAG is staying into Running state eternally ( DAG Runs = 'running', Recent 
Tasks state = 'none' on the WebUI) .

Reverting to 1.10.5 not fixes the issue. 

*The scheduler logs* ( kubectl logs airflow-85bfcd8c86-v4lps scheduler ) shows 
repeatedly:

[2019-12-22 08:42:42,278] {{scheduler_job.py:210}} WARNING - Killing PID XXXXXX.

*DAG log* :

[2019-12-22 09:09:07,007] {{scheduler_job.py:1507}} INFO - DAG(s) 
dict_keys(['d20manytest']) retrieved from /root/airflow/dags/d20.py
 [2019-12-22 09:09:07,118] {{scheduler_job.py:1212}} INFO - Processing 
d20manytest
 [2019-12-22 09:09:07,133] {{scheduler_job.py:1225}} INFO - Created None
 [2019-12-22 09:09:07,149] {{scheduler_job.py:690}} INFO - Examining DAG run 
<DagRun d20manytest @ 2019-12-20 14:00:00+00:00: 
scheduled__2019-12-20T14:00:00+00:00, externally triggered: False>
 [2019-12-22 09:11:05,188] {{logging_mixin.py:90}} INFO - [2019-12-22 
09:11:05,187] {{settings.py:175}} INFO - settings.configure_orm(): Using pool 
settings. pool_size=0, max_overflow=10, pool_recycle=1800, pid=21046
 [2019-12-22 09:11:05,190] {{scheduler_job.py:142}} INFO - Started process 
(PID=21046) to work on /root/airflow/dags/d20.py
 [2019-12-22 09:11:05,192] {{scheduler_job.py:1495}} INFO - Processing file 
/root/airflow/dags/d20.py for tasks to queue
 [2019-12-22 09:11:05,193] {{logging_mixin.py:90}} INFO - [2019-12-22 
09:11:05,193] {{dagbag.py:90}} INFO - Filling up the DagBag from 
/root/airflow/dags/d20.py

 

> Airflow scheduler stuck
> -----------------------
>
>                 Key: AIRFLOW-5506
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5506
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: scheduler
>    Affects Versions: 1.10.4, 1.10.5
>            Reporter: t oo
>            Priority: Major
>
> re-post of 
> [https://stackoverflow.com/questions/57713394/airflow-scheduler-stuck] and 
> slack discussion
>  
>  
> I'm testing the use of Airflow, and after triggering a (seemingly) large 
> number of DAGs at the same time, it seems to just fail to schedule anything 
> and starts killing processes. These are the logs the scheduler prints:
> {{[2019-08-29 11:17:13,542] \{scheduler_job.py:214} WARNING - Killing PID 
> 199809
> [2019-08-29 11:17:13,544] \{scheduler_job.py:214} WARNING - Killing PID 199809
> [2019-08-29 11:17:44,614] \{scheduler_job.py:214} WARNING - Killing PID 2992
> [2019-08-29 11:17:44,614] \{scheduler_job.py:214} WARNING - Killing PID 2992
> [2019-08-29 11:18:15,692] \{scheduler_job.py:214} WARNING - Killing PID 5174
> [2019-08-29 11:18:15,693] \{scheduler_job.py:214} WARNING - Killing PID 5174
> [2019-08-29 11:18:46,765] \{scheduler_job.py:214} WARNING - Killing PID 22410
> [2019-08-29 11:18:46,766] \{scheduler_job.py:214} WARNING - Killing PID 22410
> [2019-08-29 11:19:17,845] \{scheduler_job.py:214} WARNING - Killing PID 42177
> [2019-08-29 11:19:17,846] \{scheduler_job.py:214} WARNING - Killing PID 42177
> ...}}
> I'm using a LocalExecutor with a PostgreSQL backend DB. It seems to be 
> happening only after I'm triggering a large number (>100) of DAGs at about 
> the same time using external triggering. As in:
> {{airflow trigger_dag DAG_NAME}}
> After waiting for it to finish killing whatever processes he is killing, he 
> starts executing all of the tasks properly. I don't even know what these 
> processes were, as I can't really see them after they are killed...
> Did anyone encounter this kind of behavior? Any idea why would that happen?
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to