[
https://issues.apache.org/jira/browse/AIRFLOW-1738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christopher Groskopf updated AIRFLOW-1738:
------------------------------------------
Description:
We're building integration tests around Airflow that upload a new {{dag.py}}
file, trigger it's execution (using
{{airflow.api.common.experimental.trigger_dag}}), wait for it to complete, and
then cleanup. However, we've run into a peculiar set of circumstances where if
we do this multiple times in sequence, Airflow will sometime hang indefinitely
when we try to trigger our next dag.
I've traced this and determined that the issue only happens if remove the dag
files after each test. I speculate that this is because triggering a dag
invokes {{create_dagrun}} which in turn invokes {{verify_integrity}}, which I
believe may be choking on the sudden disappearance of the DAG files. I'm not
certain this is correct, but it's the best explanation I've been able to come
up with.
was:
We're building integration tests around Airflow that upload a new {{dag.py}}
file, trigger it's execution (using
{{airflow.api.common.experimental.trigger_dag}}), wait for it to complete, and
then cleanup. However, we've run into a peculiar set of circumstances where if
we do this multiple times in sequence, Airflow will sometime hang indefinitely
when we try to trigger our next dag.
I've traced this and determined that the issue only happens if remove the dag
files after each test. I speculate that this is because triggering a dag
invokes {{create_dagrun}} which in turn invokes {{verify_integrity}}, which I
believe may be choking on the sudden disappearance of the DAG files. I'm
certain this is correct, but it's the best explanation I've been able to come
up with.
> Deleting dag.py after manual run causes Airflow to hang on next manual run
> --------------------------------------------------------------------------
>
> Key: AIRFLOW-1738
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1738
> Project: Apache Airflow
> Issue Type: Bug
> Components: DagRun
> Environment: Airflow 1.8.2, running on Alpine Docker images with
> Celery.
> Reporter: Christopher Groskopf
> Priority: Minor
>
> We're building integration tests around Airflow that upload a new {{dag.py}}
> file, trigger it's execution (using
> {{airflow.api.common.experimental.trigger_dag}}), wait for it to complete,
> and then cleanup. However, we've run into a peculiar set of circumstances
> where if we do this multiple times in sequence, Airflow will sometime hang
> indefinitely when we try to trigger our next dag.
> I've traced this and determined that the issue only happens if remove the dag
> files after each test. I speculate that this is because triggering a dag
> invokes {{create_dagrun}} which in turn invokes {{verify_integrity}}, which I
> believe may be choking on the sudden disappearance of the DAG files. I'm not
> certain this is correct, but it's the best explanation I've been able to come
> up with.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
