[
https://issues.apache.org/jira/browse/AIRFLOW-934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15924852#comment-15924852
]
Bolke de Bruin commented on AIRFLOW-934:
----------------------------------------
Try with 1.8.0rc5 - 1.6.1. is not supported anymore.
> airflow delayed the task to start
> ----------------------------------
>
> Key: AIRFLOW-934
> URL: https://issues.apache.org/jira/browse/AIRFLOW-934
> Project: Apache Airflow
> Issue Type: Bug
> Components: DagRun
> Affects Versions: Airflow 1.6.2
> Reporter: Yajun Dong
> Priority: Blocker
>
> we have a complex DAG which includes many tasks. but recently we found some
> tasks start delayed. for instance:
> start_task(which will start at 00:00) --> create_cluster(will finished at
> 00:11) --> wait_task(start at 00:16) ,
> note:
> 1. wait_task only has one upstream that is create_cluster
> 2. the server that the airflow hosts has enough memory, and
> celeryd_concurrency is 20.
> below is the log of wait_task:
> [2017-03-02 00:16:39,602] {models.py:124} INFO - Filling up the DagBag from
> /home/ubuntu/airflow/dags/etl_prod/etl_prod.py
> [2017-03-02 00:16:39,603] {models.py:197} INFO - Importing
> /home/ubuntu/airflow/dags/etl_prod/etl_prod.py
> [2017-03-02 00:16:39,613] {models.py:284} INFO - Loaded DAG <DAG: etl_prod>
> [2017-03-02 00:16:40,333] {models.py:124} INFO - Filling up the DagBag from
> /home/ubuntu/airflow/dags/etl_prod/etl_prod.py
> [2017-03-02 00:16:40,333] {models.py:197} INFO - Importing
> /home/ubuntu/airflow/dags/etl_prod/etl_prod.py
> [2017-03-02 00:16:40,345] {models.py:284} INFO - Loaded DAG <DAG: etl_prod>
> [2017-03-02 00:16:40,373] {models.py:936} INFO -
> --------------------------------------------------------------------------------
> New run starting @2017-03-02T00:16:40.369560
> --------------------------------------------------------------------------------
> [2017-03-02 00:16:40,402] {models.py:951} INFO - Queuing into pool None
> [2017-03-02 00:22:31,161] {models.py:124} INFO - Filling up the DagBag from
> /home/ubuntu/airflow/dags/etl_prod/etl_prod.py
> [2017-03-02 00:22:31,162] {models.py:197} INFO - Importing
> /home/ubuntu/airflow/dags/etl_prod/etl_prod.py
> [2017-03-02 00:22:31,172] {models.py:284} INFO - Loaded DAG <DAG: etl_prod>
> [2017-03-02 00:22:31,863] {models.py:124} INFO - Filling up the DagBag from
> /home/ubuntu/airflow/dags/etl_prod/etl_prod.py
> [2017-03-02 00:22:31,863] {models.py:197} INFO - Importing
> /home/ubuntu/airflow/dags/etl_prod/etl_prod.py
> [2017-03-02 00:22:31,874] {models.py:284} INFO - Loaded DAG <DAG: etl_prod>
> [2017-03-02 00:22:31,901] {models.py:936} INFO -
> --------------------------------------------------------------------------------
> New run starting @2017-03-02T00:22:31.897547
> --------------------------------------------------------------------------------
> [2017-03-02 00:22:31,911] {models.py:974} INFO - Executing
> <Task(BashOperator): wait_mins> on 2017-03-01 00:00:00
> [2017-03-02 00:22:31,922] {bash_operator.py:52} INFO - tmp dir root location:
> /tmp
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
