Andrei-Alin Popescu created AIRFLOW-2680:
--------------------------------------------
Summary: Don't automatically percolate skipped stae
Key: AIRFLOW-2680
URL: https://issues.apache.org/jira/browse/AIRFLOW-2680
Project: Apache Airflow
Issue Type: Improvement
Reporter: Andrei-Alin Popescu
Dear Airflow Maintainers,
As part of our workflow, we have cases where all the upstream of a certain task
A can be skipped. In this case, airflow seems to automatically mark A as
skipped.
However, this does not quite work for us, since there are changes external to
the DAG which A needs to process, regardless of whether its upstream ran or
not. Additionally, we require A to get into an "upstream_failed" state and not
run if any its upstream tasks failed.
I don't see a trigger rule to cover this, so what would be the best way to
achieve this? I was thinking we could attach a DummyOperator as an upstream to
A, which in a way marks the fact that A depends on some external data and needs
to run anyway, but this can get really ugly for big DAGs.
I was also thinking we could have a new trigger_rule, e.g. "no_failure", which
would only trigger tasks if no upstream has failed. It differs from
"all_success" in that it will also trigger if all upstream has been skipped,
rather than percolating the skipped state on.
I'd really appreciate your feedback on this, and I'd like to know if in fact
there is already a good way of doing this with airflow that I don't know of.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
