[
https://issues.apache.org/jira/browse/AIRFLOW-992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dan Davydov updated AIRFLOW-992:
--------------------------------
Description:
We are seeing skipped tasks not being propagated correctly:
E.g.
A->B
`-->C
Task A depends on task B and C
If B gets skipped and C gets run then:
Expected:
A will get skipped
EDIT: Upon further investigation this was caused by a change in the semantics
of ALL_SUCCESS, which I have these feelings about:
Intuitively you would expect to skip any task that has dependencies that
weren't run by default, i.e. the trigger rule is called ALL_SUCCESS and skipped
tasks are not successful ones, and that was also the old behavior in 1.7.3.
This is going to break some use cases which could be alright, but I feel these
new semantics make less sense than before so it's a bad reason to break
existing use cases.
I will get started on a PR for a new ALL_SUCCESS_NOT_SKIPPED trigger rule but
again I feel this is hacky and really we should have the old ALL_SUCCESS
(default) and a new ALL_SUCCESS_OR_SKIPPED trigger rule if desired.
Actual:
A gets run
It's possible the reason for the change in semantics (which was to fix a bug)
is fixed already by https://github.com/apache/incubator-airflow/pull/2154 and
the only thing we need to do is revert
https://github.com/apache/incubator-airflow/pull/2125
[~bolke]
was:
We are seeing skipped tasks not being propagated correctly:
E.g.
A->B
`-->C
Task A depends on task B and C
If B gets skipped and C gets run then:
Expected:
A will get skipped
EDIT: Upon further investigation this was caused by a change in the semantics
of ALL_SUCCESS, which I have these feelings about:
Intuitively you would expect to skip any task that has dependencies that
weren't run by default, i.e. the trigger rule is called ALL_SUCCESS and skipped
tasks are not successful ones, and that was also the old behavior in 1.7.3.
This is going to break some use cases which could be alright, but I feel these
new semantics make less sense than before so it's a bad reason to break
existing use cases.
I will get started on a PR for a new ALL_SUCCESS_NOT_SKIPPED trigger rule but
again I feel this is hacky and really we should have the old ALL_SUCCESS
(default) and a new ALL_SUCCESS_OR_SKIPPED trigger rule if desired.
Actual:
A gets run
[~bolke]
> Skipped tasks do not propagate correctly
> ----------------------------------------
>
> Key: AIRFLOW-992
> URL: https://issues.apache.org/jira/browse/AIRFLOW-992
> Project: Apache Airflow
> Issue Type: Bug
> Reporter: Dan Davydov
> Priority: Critical
>
> We are seeing skipped tasks not being propagated correctly:
> E.g.
> A->B
> `-->C
> Task A depends on task B and C
> If B gets skipped and C gets run then:
> Expected:
> A will get skipped
> EDIT: Upon further investigation this was caused by a change in the semantics
> of ALL_SUCCESS, which I have these feelings about:
> Intuitively you would expect to skip any task that has dependencies that
> weren't run by default, i.e. the trigger rule is called ALL_SUCCESS and
> skipped tasks are not successful ones, and that was also the old behavior in
> 1.7.3.
> This is going to break some use cases which could be alright, but I feel
> these new semantics make less sense than before so it's a bad reason to break
> existing use cases.
> I will get started on a PR for a new ALL_SUCCESS_NOT_SKIPPED trigger rule but
> again I feel this is hacky and really we should have the old ALL_SUCCESS
> (default) and a new ALL_SUCCESS_OR_SKIPPED trigger rule if desired.
> Actual:
> A gets run
> It's possible the reason for the change in semantics (which was to fix a bug)
> is fixed already by https://github.com/apache/incubator-airflow/pull/2154 and
> the only thing we need to do is revert
> https://github.com/apache/incubator-airflow/pull/2125
> [~bolke]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
