[
https://issues.apache.org/jira/browse/AIRFLOW-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16980019#comment-16980019
]
ASF GitHub Bot commented on AIRFLOW-2279:
-----------------------------------------
yuqian90 commented on pull request #6633: [AIRFLOW-2279] Clear tasks across
DAGs if marked by ExternalTaskMarker
URL: https://github.com/apache/airflow/pull/6633
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title.
- https://issues.apache.org/jira/browse/AIRFLOW-2279
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
There's a clear need for having a way to clear tasks across DAGs when there
are dependencies set up by ExternalTaskSensor. This PR adds an
ExternalTaskMarker to indicate such need. And when ExternalTaskMarker is used,
the "Clear" action by the user will look for the external tasks recursively and
clear them. There's no need to change the UI because the "Recursive" option in
the Clear form fits this nicely.
This is primarily addressing AIRFLOW-2279. However, there are various other
users in the community using ExternalTaskSensor to set up cross-DAG
dependencies and many of them asked for this kind of feature.
https://issues.apache.org/jira/browse/AIRFLOW-2279
https://issues.apache.org/jira/browse/AIRFLOW-1414
https://issues.apache.org/jira/browse/AIRFLOW-1446
https://issues.apache.org/jira/browse/AIRFLOW-4005
https://www.mikulskibartosz.name/using-sensors-in-airflow/
https://forum.astronomer.io/t/dependencies-across-dags/332
https://geoffruddock.com/how-to-use-external-task-sensor-in-airflow/
https://stackoverflow.com/questions/38022323/how-to-set-dependencies-between-dags-in-airflow
https://issues.apache.org/jira/browse/AIRFLOW-1825
This inactive PR tried to implement this feature. However it was abandoned.
And I feel my approach is simpler and more flexible. And it does not need UI
changes.
https://github.com/apache/airflow/pull/2444
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Tests added to test_external_task_sensor.py
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Clearing Tasks Across DAGs
> --------------------------
>
> Key: AIRFLOW-2279
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2279
> Project: Apache Airflow
> Issue Type: Improvement
> Reporter: Achal Soni
> Priority: Major
> Attachments: cross_dag_ui_screenshot.png
>
>
> At Stripe, we commonly have discrete dags that depend on each other by
> leveraging ExternalTaskSensors. We also find ourselves routinely wanting to
> not only clear tasks and their downstream tasks in a particular dag, but also
> their downstream tasks in their dependent dags (linked by
> ExternalTaskSensors).
> We currently have extended Airflow to handle this by modifying the webapp and
> cli tool to optionally clear dependent tasks across multiple dags (see
> attached screenshot).
> We want to open the floor for discussion with the larger Airflow community
> about the usage of ExternalTaskSensors and specifically how to handle
> clearing across dags. We are interested in learning more about the accepted
> practices in this regard, and are very open/willing to contribute in this
> area if there is interest!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
