potiuk commented on PR #29406:
URL: https://github.com/apache/airflow/pull/29406#issuecomment-1449441326
I think it needs a bit of anylysis and design. It's hard to get the logic
around it without viualising it, but @uranusjr is right that some thinking
should be done around it. I guess we have two options:
* fail everything regardless from the rules (this is the current solution)
* be "smart" about it and include triggering rules in it
How the "smart" should look like - hard to say. probably each rule should be
analysed how it should behave and good strategy for each rule shoudl be worked
out (if that's even possible) - trying to logically attempt to follow the rules
and choose for example a strategy of failing currently run tasks and
propagating it furhter. That would be rather complex and there are some
non-obvious traps we can fall in (what happens with the tasks that succeed when
one failed?). And what about the tasks that follow them?
I think it very much depends on the definition of "fail fast". If we define
it as "fail all the tasks if any of those tasks fail", then current solution is
good. If we try to follow triggering rules, then well, it's complex and likely
"fali fast" is hard to define in general ("fail fast all the currently eligible
tasks and propagagate it across the DAG").
But maybe there is a middle-ground. Maybe we can only make fail-fast work if
the only triggering rules we have are "all_success" (i.e. default). And if you
add another triggering rule to a "fail-fast" DAG, this could cause an error.
I think there is a big class of DAGs which fall into that category and those
are precisely the DAGs where "fail fast" defined as "fail all the tasks if any
of them fail" makes perfect sense.
@uranusjr ? WDYT? I guess that would only require to add the error in case
any non-default triggering rule is added for "fail-fast" dag (and properly
documenting it).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
