Raphael Lopez Kaufman created AIRFLOW-1414:
----------------------------------------------
Summary: Add support for retriggering dependent workflows
Key: AIRFLOW-1414
URL: https://issues.apache.org/jira/browse/AIRFLOW-1414
Project: Apache Airflow
Issue Type: Improvement
Reporter: Raphael Lopez Kaufman
Currently when using an ExternalTaskSensor sensor, to have a dag wait for the
completion of a task in another dag, there is no way, when clearing the task
that is depended on, to also clear the ExternalTaskSensor task (and its
downstream/upstream tasks).
However, that might be important when teams have separate responsibilities and
create multi-staged data pipelines. Let's say there is a team responsible for
transforming logs to make them available in hadoop land. They maintain a hourly
dag processing the raw logs. Other teams wait on the completion of this dag
using an ExternalTaskSensor. If the logging team realizes that some logs where
corrupted, not only do they have to rerun the task that processed these logs
but also all the dags that use these logs to generate higher level data.
Introducing the notion of descendants can solve this problem. The descendants
of dag A are the dags that have an ExternalTaskSensor pointing to a task of dag
A. Then, when clearing a task in the view, an option can be added to also clear
the tasks of the descendants (having the upstream, downstream, recursive,
future and past flags act accordingly).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
