[
https://issues.apache.org/jira/browse/AIRFLOW-6853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Johannes Wienke updated AIRFLOW-6853:
-------------------------------------
Summary: ExternalTaskSensor check_existence does not work with packaged
DAGs (was: ExternalTaskSensor does not work with packaged DAGs)
> ExternalTaskSensor check_existence does not work with packaged DAGs
> -------------------------------------------------------------------
>
> Key: AIRFLOW-6853
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6853
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
> Affects Versions: 1.10.9
> Reporter: Johannes Wienke
> Priority: Major
>
> The current implementation of ExternalTaskSensor uses the file location of a
> DAG to determine if it still exists. This is realized by the following code
> fragment:
> {code:python}
> if not os.path.exists(dag_to_wait.fileloc):
> raise AirflowException('The external DAG '
> '{} was
> deleted.'.format(self.external_dag_id)
> {code}
> In case the DAG to check is part of a Zip archive, fileloc looks something
> like: {{/usr/local/airflow/dags/analytics.zip/create_keyframes.py}}. This can
> obviously not be checked with {{os.path.exists}} and therefore the sensor is
> currently unable to handle packaged DAGs at all.
> This also contradicts the explicit note in the DAG model:
> {noformat}
> # The location of the file containing the DAG object
> # Note: Do not depend on fileloc pointing to a file; in the case of a
> # packaged DAG, it will point to the subpath of the DAG within the
> # associated zip.
> {noformat}
> I suspect, the sensor should instead use {{is_active}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
