mnojek commented on a change in pull request #22311:
URL: https://github.com/apache/airflow/pull/22311#discussion_r841184322
##########
File path: docs/apache-airflow/best-practices.rst
##########
@@ -247,18 +247,26 @@ each parameter by following the links):
Example of watcher pattern with trigger rules
---------------------------------------------
-The watcher pattern is how we call a DAG with a task that is "watching" the
states of the other tasks. It's primary purpose is to fail a DAG Run when any
other task fail.
+The watcher pattern is how we call a DAG with a task that is "watching" the
states of the other tasks.
+It's primary purpose is to fail a DAG Run when any other task fail.
The need came from the Airflow system tests that are DAGs with different tasks
(similarly like a test containing steps).
-Normally, when any task fails, all other tasks are not executed and the whole
DAG Run gets failed status too. But when we use trigger rules, we can disrupt
the normal flow of running tasks and the whole DAG may represent different
status that we expect.
-For example, we can have a teardown task (with trigger rule set to
``"all_done"``) that will be executed regardless of the state of the other
tasks (e.g. to clean up the resources). In such situation, the DAG would always
run this task and the DAG Run will get the status of this particular task, so
we can potentially lose the information about failing tasks.
-If we want to ensure that the DAG with teardown task would fail if any task
fails, we need to use the watcher pattern.
-The watcher task is a task that will always fail if triggered, but it needs to
be triggered only if any other task fails. It needs to have a trigger rule set
to ``"one_failed"`` and it needs also to be a downstream task for all other
tasks in the DAG.
-Thanks to this, if every other task will pass, the watcher will be skipped,
but when something fails, the watcher task will be executed and fail making the
DAG Run fail too.
+Normally, when any task fails, all other tasks are not executed and the whole
DAG Run gets failed status too. But
+when we use trigger rules, we can disrupt the normal flow of running tasks and
the whole DAG may represent different
+status that we expect. For example, we can have a teardown task (with trigger
rule set to ``TriggerRule.ALL_DONE``)
+that will be executed regardless of the state of the other tasks (e.g. to
clean up the resources). In such
+situation, the DAG would always run this task and the DAG Run will get the
status of this particular task, so we can
+potentially lose the information about failing tasks. If we want to ensure
that the DAG with teardown task would fail
+if any task fails, we need to use the watcher pattern. The watcher task is a
task that will always fail if
+triggered, but it needs to be triggered only if any other task fails. It needs
to have a trigger rule set to
+``TriggerRule.ONE_FAILED`` and it needs also to be a downstream task for all
other tasks in the DAG. Thanks to
+this, if every other task will pass, the watcher will be skipped, but when
something fails, the watcher task will be
Review comment:
No, the teardown task will be executed because by design it has a
trigger rule set to "all done" which means it is executed always when all
parent tasks are executed. The watcher is a child for every task so it doesn't
count into the tasks that need to be executed to trigger the teardown. It is
skipped only if everything passed.
And the watcher is triggered only if any of its parent tasks failed, and
because all tasks are parent for the watcher, if any of them fails, the watcher
is triggered.
Hope it answers your question. If you still have concerns, please comment
and I will try to help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
