Raphael Lopez Kaufman commented on AIRFLOW-1414:

[~bolke] Any though on that this (the corresponding PR is 
We are trying to switch from Oozie to Airflow at Booking.com and would really 
need this feature (or maybe there's already a way to achieve this) for us to go 
forward with the migration.

> Add support for retriggering dependent workflows
> ------------------------------------------------
>                 Key: AIRFLOW-1414
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1414
>             Project: Apache Airflow
>          Issue Type: Improvement
>            Reporter: Raphael Lopez Kaufman
> Currently when using an ExternalTaskSensor sensor, to have a dag wait for the 
> completion of a task in another dag, there is no way, when clearing the task 
> that is depended on, to also clear the ExternalTaskSensor task (and its 
> downstream/upstream tasks).
> However, that might be important when teams have separate responsibilities 
> and create multi-staged data pipelines. Let's say there is a team responsible 
> for transforming logs to make them available in hadoop land. They maintain a 
> hourly dag processing the raw logs. Other teams wait on the completion of 
> this dag using an ExternalTaskSensor. If the logging team realizes that some 
> logs where corrupted, not only do they have to rerun the task that processed 
> these logs but also all the dags that use these logs to generate higher level 
> data.
> Introducing the notion of descendants can solve this problem. The descendants 
> of dag A are the dags that have an ExternalTaskSensor pointing to a task of 
> dag A. Then, when clearing a task in the view, an option can be added to also 
> clear the tasks of the descendants (having the upstream, downstream, 
> recursive, future and past flags act accordingly).

This message was sent by Atlassian JIRA

Reply via email to