list(dag.tasks) >> watcher() may be more accurate to not rely on the implementation details of tasks property (it can be generator in the future).
Wdyt? On Thu, 10 Feb 2022 at 23:00, Jarek Potiuk <[email protected]> wrote: > One more thing, callbacks (daniel) - also bad thing is that we really need > it to set the "dag status" not run anything. The "watcher" is mainly there > to propagate the "failure" status to Dag to get the Dag "fail" when any of > the tasks fail - if we have callbacks for all tasks, The group > approach also asks for more "noise" to the example, The nice thing about > adding watcher is that it's just "completely separate" part of the DAGand > we can easily mark it as "boilerplate" without raising too many questions > from the users watching the examples. > > Dennis -> I think the cases you described are already there in "all_done", > and other triggering rules. I guess what I only aimed at here is how to set > the dependencies to "all other" tasks (no matter what the triggering rule > is) and the proposal by Ash seems to "just work". > > J. > > > On Thu, Feb 10, 2022 at 9:48 PM Jarek Potiuk <[email protected]> wrote: > >> @dstandish.- Yeah - callback is a bit too "invasive". I thought about it >> but at least some examples use default_args and adding "callback" there >> would mix the concerns and make it difficult to extract parts of the >> examples in our documentation. >> >> @ash - yeah. I realized that when you posted it that the >> will work in >> this case only because of the sequence of processing of the operator :). I >> have not thought about it before, but when you posted it, it was kinda >> obvious :). >> It is a bit "implicit" and probably I would not rely on this behaviour >> elsewhere, but it actually plays very well with the use-case of system >> tests. >> >> J. >> >> >> On Thu, Feb 10, 2022 at 8:08 PM Daniel Standish >> <[email protected]> wrote: >> >>> The other thing that comes to mind is you can add your "normal" tasks >>> to a task group and then do `my_group >> watcher` >>> >>> Also I noticed that dag can take success / failure callbacks. Maybe >>> the "watcher" task makes sense as a callback. >>> >>> -- Eugene
