[
https://issues.apache.org/jira/browse/AIRFLOW-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16929083#comment-16929083
]
Wesley edited comment on AIRFLOW-2843 at 9/13/19 10:08 AM:
-----------------------------------------------------------
Using the operator stumbled upon the case where the `external_dag_id` and
`external_task_id` exist but the DAG is disabled, so the ExternalTaskSensor
task instance keep poking for a DagRun that does not exist.
Perhaps the `check_existence` should check DagRun instead? or the name should
be more specific like `check_dag_existence`/ `check_task_existence`?
What you guys think about this use case?
was (Author: wesleybatista):
Using the operator stumbled upon the case where the `external_dag_id` and
`external_task_id` exist but the DAG is disabled, so the ExternalTaskSensor
task instance keep polling for a DagRun that does not exist.
Perhaps the `check_existence` should check DagRun instead? or the name should
be more specific like `check_dag_existence`/ `check_task_existence`?
What you guys think about this use case?
> ExternalTaskSensor: Add option to cease waiting immediately if the external
> DAG/task doesn't exist
> --------------------------------------------------------------------------------------------------
>
> Key: AIRFLOW-2843
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2843
> Project: Apache Airflow
> Issue Type: Improvement
> Components: operators
> Reporter: Xiaodong DENG
> Assignee: Xiaodong DENG
> Priority: Minor
> Fix For: 1.10.3
>
>
> h2. Background
> *ExternalTaskSensor* will keep waiting (given restrictions of retries,
> poke_interval, etc), even if the external DAG/task specified doesn't exist at
> all. In some cases, this waiting may still make sense as new DAG may backfill.
> But it may be good to provide an option to cease waiting immediately if the
> external DAG/task specified doesn't exist.
> h2. Proposal
> Provide an argument "check_existence". Set to *True* to check if the external
> DAG/task exists, and immediately cease waiting if the external DAG/task does
> not exist.
> The default value is set to *False* (no check or ceasing will happen) so it
> will not affect any existing DAGs or user expectation.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
