I'm confused that a failed status on short_circuit doesn't list its
associated DagRun as failed. I was under the impression a DagRun was
given
failed status if any of its tasks failed.
Thanks for the comments, Laura! Yes, we were slightly surprised by the
effect of the short circuit operator as well. Looking at the
documentation, though, it appears consistent enough, since no task
actually fails:
"The ShortCircuitOperator is derived from the PythonOperator. It
evaluates a condition and short-circuits the workflow if the condition
is False. Any downstream tasks are marked with a state of “skipped”. If
the condition is True, downstream tasks proceed as normal." (from [1])
From the code [2]:
def execute(self, context):
condition = super(ShortCircuitOperator, self).execute(context)
if condition:
logging.info('Proceeding with downstream tasks...')
return
else:
logging.info('Skipping downstream tasks...')
...
logging.info("Done.")
Perhaps it would be worth considering a "fail_on_false" flag for the
operator? That might be something we'll try ourselves as a simple custom
extension of this operator.
Regards
ap
[1]
https://pythonhosted.org/airflow/code.html#airflow.operators.ShortCircuitOperator
[2]
https://pythonhosted.org/airflow/_modules/python_operator.html#ShortCircuitOperator
