[jira] [Commented] (AIRFLOW-4453) none_failed trigger rule cascading skipped state to downstream tasks
[
https://issues.apache.org/jira/browse/AIRFLOW-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286082#comment-17286082
]
N. M. commented on AIRFLOW-4453:
This has either regressed in 1.10.12 or there are still corner cases where it
is not solved.
In a simple three-task DAG where, the first step is the
[GoogleCloudStoragePrefixSensor|https://airflow.apache.org/docs/apache-airflow/1.10.12/_modules/airflow/contrib/sensors/gcs_sensor.html],
followed by a processing task and ending with a heartbeat check operator:
check_for_late_data >> run_statekeeper >> passive_check
The passive_check task is configured with NONE_FAILED:
passive_check = PassiveCheckOperator(task_id="passive_check", dag=dag,
trigger_rule=TriggerRule.NONE_FAILED)
The GCS sensor operator exits thusly if it finds no keys:
[2021-02-12 00:32:18,130] \{taskinstance.py:1025} INFO - Marking task as
SKIPPED.dag_id=pipeline_v1, task_id=check_for_late_data,
execution_date=20210211T003000, start_date=20210212T003017, end_date=
[2021-02-12 00:32:18,130] \{taskinstance.py:1070} INFO - Marking task as
SUCCESS.dag_id=pipeline_v1, task_id=check_for_late_data,
execution_date=20210211T003000, start_date=20210212T003017,
end_date=20210212T003218
...but the final step never fires:
!image-2021-02-17-12-41-23-809.png!
If we put a shared dummy start task in front, the same happens:
!image-2021-02-17-13-50-32-787.png!
This is airflow 1.10.12, installed from the airflow-7.16.0 helm chart, using
the celery executor.
> none_failed trigger rule cascading skipped state to downstream tasks
>
>
> Key: AIRFLOW-4453
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4453
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG, scheduler
>Affects Versions: 1.10.3, 1.10.4, 1.10.5, 1.10.6, 1.10.7
>Reporter: Dmytro Kulyk
>Assignee: Kaxil Naik
>Priority: Major
> Labels: skipped
> Fix For: 1.10.5, 1.10.10
>
> Attachments: 3_step.png, cube_update.py,
> image-2019-05-02-18-11-28-307.png, image-2021-02-17-12-41-23-809.png,
> image-2021-02-17-13-50-32-787.png, simple_skip.png
>
>
> Task with trigger_rule = 'none_failed' cascading *skipped *status to
> downstream task
> * task have multiple upstream tasks
> * trigger_rule set to 'none_failed'
> * some of upstream tasks can be skipped due to *latest only*
> Basing on documentation this shouldn't happen
> !image-2019-05-02-18-11-28-307.png|width=655,height=372!
> DAG attached
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4453) none_failed trigger rule cascading skipped state to downstream tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068699#comment-17068699 ] ASF subversion and git services commented on AIRFLOW-4453: -- Commit d85db992b6378d9ad39133dc14374f71abdd4c09 in airflow's branch refs/heads/v1-10-test from yuqian90 [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=d85db99 ] [AIRFLOW-4453] Make behavior of `none_failed` consistent with documentation (#7464) The documentation for the `none_failed` trigger rule (https://airflow.apache.org/docs/stable/concepts.html#trigger-rules) describes its behavior as "all parents have not failed (`failed` or `upstream_failed`) i.e. all parents have succeeded or been skipped." With that definition in mind, there is no reason that the check for `none_failed` should ever have to check for skipped upstream tasks or set the current task to `skipped`. The current behavior causes the rule to skip if all upstream tasks have skipped, which is more than a little confusing. This fixes the behavior to be consistent with the documentation. Co-authored-by: root (cherry picked from commit f0c65e32103fa0fe91a4890d75ba31a89aa226e7) > none_failed trigger rule cascading skipped state to downstream tasks > > > Key: AIRFLOW-4453 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4453 > Project: Apache Airflow > Issue Type: Bug > Components: DAG, scheduler >Affects Versions: 1.10.3, 1.10.4, 1.10.5, 1.10.6, 1.10.7 >Reporter: Dmytro Kulyk >Assignee: Kaxil Naik >Priority: Major > Labels: skipped > Fix For: 1.10.5, 1.10.10 > > Attachments: 3_step.png, cube_update.py, > image-2019-05-02-18-11-28-307.png, simple_skip.png > > > Task with trigger_rule = 'none_failed' cascading *skipped *status to > downstream task > * task have multiple upstream tasks > * trigger_rule set to 'none_failed' > * some of upstream tasks can be skipped due to *latest only* > Basing on documentation this shouldn't happen > !image-2019-05-02-18-11-28-307.png|width=655,height=372! > DAG attached -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4453) none_failed trigger rule cascading skipped state to downstream tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068629#comment-17068629 ] ASF subversion and git services commented on AIRFLOW-4453: -- Commit c824f381384dbc7693cba80ceb10f25b256a8b6a in airflow's branch refs/heads/v1-10-test from yuqian90 [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=c824f38 ] [AIRFLOW-4453] Make behavior of `none_failed` consistent with documentation (#7464) The documentation for the `none_failed` trigger rule (https://airflow.apache.org/docs/stable/concepts.html#trigger-rules) describes its behavior as "all parents have not failed (`failed` or `upstream_failed`) i.e. all parents have succeeded or been skipped." With that definition in mind, there is no reason that the check for `none_failed` should ever have to check for skipped upstream tasks or set the current task to `skipped`. The current behavior causes the rule to skip if all upstream tasks have skipped, which is more than a little confusing. This fixes the behavior to be consistent with the documentation. Co-authored-by: root (cherry picked from commit f0c65e32103fa0fe91a4890d75ba31a89aa226e7) > none_failed trigger rule cascading skipped state to downstream tasks > > > Key: AIRFLOW-4453 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4453 > Project: Apache Airflow > Issue Type: Bug > Components: DAG, scheduler >Affects Versions: 1.10.3, 1.10.4, 1.10.5, 1.10.6, 1.10.7 >Reporter: Dmytro Kulyk >Assignee: Kaxil Naik >Priority: Major > Labels: skipped > Fix For: 1.10.5, 1.10.10 > > Attachments: 3_step.png, cube_update.py, > image-2019-05-02-18-11-28-307.png, simple_skip.png > > > Task with trigger_rule = 'none_failed' cascading *skipped *status to > downstream task > * task have multiple upstream tasks > * trigger_rule set to 'none_failed' > * some of upstream tasks can be skipped due to *latest only* > Basing on documentation this shouldn't happen > !image-2019-05-02-18-11-28-307.png|width=655,height=372! > DAG attached -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4453) none_failed trigger rule cascading skipped state to downstream tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068210#comment-17068210 ] ASF subversion and git services commented on AIRFLOW-4453: -- Commit 1605f1def43c86f84a1c1a7e7732d46bb0e1814a in airflow's branch refs/heads/v1-10-test from yuqian90 [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=1605f1d ] [AIRFLOW-4453] Make behavior of `none_failed` consistent with documentation (#7464) The documentation for the `none_failed` trigger rule (https://airflow.apache.org/docs/stable/concepts.html#trigger-rules) describes its behavior as "all parents have not failed (`failed` or `upstream_failed`) i.e. all parents have succeeded or been skipped." With that definition in mind, there is no reason that the check for `none_failed` should ever have to check for skipped upstream tasks or set the current task to `skipped`. The current behavior causes the rule to skip if all upstream tasks have skipped, which is more than a little confusing. This fixes the behavior to be consistent with the documentation. Co-authored-by: root (cherry picked from commit f0c65e32103fa0fe91a4890d75ba31a89aa226e7) > none_failed trigger rule cascading skipped state to downstream tasks > > > Key: AIRFLOW-4453 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4453 > Project: Apache Airflow > Issue Type: Bug > Components: DAG, scheduler >Affects Versions: 1.10.3, 1.10.4, 1.10.5, 1.10.6, 1.10.7 >Reporter: Dmytro Kulyk >Assignee: Kaxil Naik >Priority: Major > Labels: skipped > Fix For: 1.10.5, 1.10.10 > > Attachments: 3_step.png, cube_update.py, > image-2019-05-02-18-11-28-307.png, simple_skip.png > > > Task with trigger_rule = 'none_failed' cascading *skipped *status to > downstream task > * task have multiple upstream tasks > * trigger_rule set to 'none_failed' > * some of upstream tasks can be skipped due to *latest only* > Basing on documentation this shouldn't happen > !image-2019-05-02-18-11-28-307.png|width=655,height=372! > DAG attached -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4453) none_failed trigger rule cascading skipped state to downstream tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068095#comment-17068095 ] ASF subversion and git services commented on AIRFLOW-4453: -- Commit eb19111e21b4b63b51e206b56d3f57d8b37d5b89 in airflow's branch refs/heads/v1-10-test from yuqian90 [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=eb19111 ] [AIRFLOW-4453] Make behavior of `none_failed` consistent with documentation (#7464) The documentation for the `none_failed` trigger rule (https://airflow.apache.org/docs/stable/concepts.html#trigger-rules) describes its behavior as "all parents have not failed (`failed` or `upstream_failed`) i.e. all parents have succeeded or been skipped." With that definition in mind, there is no reason that the check for `none_failed` should ever have to check for skipped upstream tasks or set the current task to `skipped`. The current behavior causes the rule to skip if all upstream tasks have skipped, which is more than a little confusing. This fixes the behavior to be consistent with the documentation. Co-authored-by: root (cherry picked from commit f0c65e32103fa0fe91a4890d75ba31a89aa226e7) > none_failed trigger rule cascading skipped state to downstream tasks > > > Key: AIRFLOW-4453 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4453 > Project: Apache Airflow > Issue Type: Bug > Components: DAG, scheduler >Affects Versions: 1.10.3, 1.10.4, 1.10.5, 1.10.6, 1.10.7 >Reporter: Dmytro Kulyk >Assignee: Kaxil Naik >Priority: Major > Labels: skipped > Fix For: 1.10.5, 1.10.10 > > Attachments: 3_step.png, cube_update.py, > image-2019-05-02-18-11-28-307.png, simple_skip.png > > > Task with trigger_rule = 'none_failed' cascading *skipped *status to > downstream task > * task have multiple upstream tasks > * trigger_rule set to 'none_failed' > * some of upstream tasks can be skipped due to *latest only* > Basing on documentation this shouldn't happen > !image-2019-05-02-18-11-28-307.png|width=655,height=372! > DAG attached -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4453) none_failed trigger rule cascading skipped state to downstream tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068047#comment-17068047 ] ASF subversion and git services commented on AIRFLOW-4453: -- Commit f0c65e32103fa0fe91a4890d75ba31a89aa226e7 in airflow's branch refs/heads/master from yuqian90 [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=f0c65e3 ] [AIRFLOW-4453] Make behavior of `none_failed` consistent with documentation (#7464) The documentation for the `none_failed` trigger rule (https://airflow.apache.org/docs/stable/concepts.html#trigger-rules) describes its behavior as "all parents have not failed (`failed` or `upstream_failed`) i.e. all parents have succeeded or been skipped." With that definition in mind, there is no reason that the check for `none_failed` should ever have to check for skipped upstream tasks or set the current task to `skipped`. The current behavior causes the rule to skip if all upstream tasks have skipped, which is more than a little confusing. This fixes the behavior to be consistent with the documentation. Co-authored-by: root > none_failed trigger rule cascading skipped state to downstream tasks > > > Key: AIRFLOW-4453 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4453 > Project: Apache Airflow > Issue Type: Bug > Components: DAG, scheduler >Affects Versions: 1.10.3, 1.10.4, 1.10.5, 1.10.6, 1.10.7 >Reporter: Dmytro Kulyk >Assignee: Kaxil Naik >Priority: Major > Labels: skipped > Fix For: 1.10.5 > > Attachments: 3_step.png, cube_update.py, > image-2019-05-02-18-11-28-307.png, simple_skip.png > > > Task with trigger_rule = 'none_failed' cascading *skipped *status to > downstream task > * task have multiple upstream tasks > * trigger_rule set to 'none_failed' > * some of upstream tasks can be skipped due to *latest only* > Basing on documentation this shouldn't happen > !image-2019-05-02-18-11-28-307.png|width=655,height=372! > DAG attached -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4453) none_failed trigger rule cascading skipped state to downstream tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068046#comment-17068046 ] ASF GitHub Bot commented on AIRFLOW-4453: - kaxil commented on pull request #7464: [AIRFLOW-4453] Make behavior of `none_failed` consistent with documentation URL: https://github.com/apache/airflow/pull/7464 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > none_failed trigger rule cascading skipped state to downstream tasks > > > Key: AIRFLOW-4453 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4453 > Project: Apache Airflow > Issue Type: Bug > Components: DAG, scheduler >Affects Versions: 1.10.3, 1.10.4, 1.10.5, 1.10.6, 1.10.7 >Reporter: Dmytro Kulyk >Assignee: Kaxil Naik >Priority: Major > Labels: skipped > Fix For: 1.10.5 > > Attachments: 3_step.png, cube_update.py, > image-2019-05-02-18-11-28-307.png, simple_skip.png > > > Task with trigger_rule = 'none_failed' cascading *skipped *status to > downstream task > * task have multiple upstream tasks > * trigger_rule set to 'none_failed' > * some of upstream tasks can be skipped due to *latest only* > Basing on documentation this shouldn't happen > !image-2019-05-02-18-11-28-307.png|width=655,height=372! > DAG attached -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4453) none_failed trigger rule cascading skipped state to downstream tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066328#comment-17066328 ] ASF GitHub Bot commented on AIRFLOW-4453: - yuqian90 commented on pull request #7464: [AIRFLOW-4453] Make behavior of `none_failed` consistent with documentation URL: https://github.com/apache/airflow/pull/7464 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > none_failed trigger rule cascading skipped state to downstream tasks > > > Key: AIRFLOW-4453 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4453 > Project: Apache Airflow > Issue Type: Bug > Components: DAG, scheduler >Affects Versions: 1.10.3, 1.10.4, 1.10.5, 1.10.6, 1.10.7 >Reporter: Dmytro Kulyk >Assignee: Kaxil Naik >Priority: Major > Labels: skipped > Fix For: 1.10.5 > > Attachments: 3_step.png, cube_update.py, > image-2019-05-02-18-11-28-307.png, simple_skip.png > > > Task with trigger_rule = 'none_failed' cascading *skipped *status to > downstream task > * task have multiple upstream tasks > * trigger_rule set to 'none_failed' > * some of upstream tasks can be skipped due to *latest only* > Basing on documentation this shouldn't happen > !image-2019-05-02-18-11-28-307.png|width=655,height=372! > DAG attached -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4453) none_failed trigger rule cascading skipped state to downstream tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066329#comment-17066329 ] ASF GitHub Bot commented on AIRFLOW-4453: - yuqian90 commented on pull request #7464: [AIRFLOW-4453] Make behavior of `none_failed` consistent with documentation URL: https://github.com/apache/airflow/pull/7464 This is to pick up the work started in https://github.com/apache/airflow/pull/7098. I discussed with @TV4Fun and opened this new PR to continue the work. - Make behavior of `none_failed` consistent with documentation. Do not skip itself when all parents are skipped. - Add `none_failed_or_skipped` which behaves like the old `none_failed` trigger rule (i.e. all parents are success or skipped and at least one parent is success). One use case of `none_failed_or_skipped` is the join task in a workflow with nested `BranchPythonOperator` --- Issue link: [AIRFLOW-4453](https://issues.apache.org/jira/browse/AIRFLOW-4453) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > none_failed trigger rule cascading skipped state to downstream tasks > > > Key: AIRFLOW-4453 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4453 > Project: Apache Airflow > Issue Type: Bug > Components: DAG, scheduler >Affects Versions: 1.10.3, 1.10.4, 1.10.5, 1.10.6, 1.10.7 >Reporter: Dmytro Kulyk >Assignee: Kaxil Naik >Priority: Major > Labels: skipped > Fix For: 1.10.5 > > Attachments: 3_step.png, cube_update.py, > image-2019-05-02-18-11-28-307.png, simple_skip.png > > > Task with trigger_rule = 'none_failed' cascading *skipped *status to > downstream task > * task have multiple upstream tasks > * trigger_rule set to 'none_failed' > * some of upstream tasks can be skipped due to *latest only* > Basing on documentation this shouldn't happen > !image-2019-05-02-18-11-28-307.png|width=655,height=372! > DAG attached -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4453) none_failed trigger rule cascading skipped state to downstream tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17041484#comment-17041484 ] ASF GitHub Bot commented on AIRFLOW-4453: - TV4Fun commented on pull request #7098: [AIRFLOW-4453] Make behavior of `none_failed` consistent with documentation URL: https://github.com/apache/airflow/pull/7098 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > none_failed trigger rule cascading skipped state to downstream tasks > > > Key: AIRFLOW-4453 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4453 > Project: Apache Airflow > Issue Type: Bug > Components: DAG, scheduler >Affects Versions: 1.10.3, 1.10.4, 1.10.5, 1.10.6, 1.10.7 >Reporter: Dmytro Kulyk >Assignee: Kaxil Naik >Priority: Major > Labels: skipped > Fix For: 1.10.5 > > Attachments: 3_step.png, cube_update.py, > image-2019-05-02-18-11-28-307.png, simple_skip.png > > > Task with trigger_rule = 'none_failed' cascading *skipped *status to > downstream task > * task have multiple upstream tasks > * trigger_rule set to 'none_failed' > * some of upstream tasks can be skipped due to *latest only* > Basing on documentation this shouldn't happen > !image-2019-05-02-18-11-28-307.png|width=655,height=372! > DAG attached -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4453) none_failed trigger rule cascading skipped state to downstream tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039786#comment-17039786 ] ASF GitHub Bot commented on AIRFLOW-4453: - yuqian90 commented on pull request #7464: [AIRFLOW-4453] Make behavior of `none_failed` consistent with documentation URL: https://github.com/apache/airflow/pull/7464 This is to pick up the work started in https://github.com/apache/airflow/pull/7098. I discussed with @TV4Fun and opened this new PR to continue the work. - Make behavior of `none_failed` consistent with documentation. Do not skip itself when all parents are skipped. - Fixing ``none_failed`` breaks the use case where ``BranchPythonOperator`` are nested. So added a fix for creating the ``join`` tasks in ``create_branch_join()``. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > none_failed trigger rule cascading skipped state to downstream tasks > > > Key: AIRFLOW-4453 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4453 > Project: Apache Airflow > Issue Type: Bug > Components: DAG, scheduler >Affects Versions: 1.10.3, 1.10.4, 1.10.5, 1.10.6, 1.10.7 >Reporter: Dmytro Kulyk >Assignee: Kaxil Naik >Priority: Major > Labels: skipped > Fix For: 1.10.5 > > Attachments: 3_step.png, cube_update.py, > image-2019-05-02-18-11-28-307.png, simple_skip.png > > > Task with trigger_rule = 'none_failed' cascading *skipped *status to > downstream task > * task have multiple upstream tasks > * trigger_rule set to 'none_failed' > * some of upstream tasks can be skipped due to *latest only* > Basing on documentation this shouldn't happen > !image-2019-05-02-18-11-28-307.png|width=655,height=372! > DAG attached -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4453) none_failed trigger rule cascading skipped state to downstream tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17012033#comment-17012033 ] Dmytro Kulyk commented on AIRFLOW-4453: --- i believe all pre 1.10.8 versions are also impacted? > none_failed trigger rule cascading skipped state to downstream tasks > > > Key: AIRFLOW-4453 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4453 > Project: Apache Airflow > Issue Type: Bug > Components: DAG, scheduler >Affects Versions: 1.10.3 >Reporter: Dmytro Kulyk >Assignee: Kaxil Naik >Priority: Major > Labels: skipped > Fix For: 1.10.5 > > Attachments: 3_step.png, cube_update.py, > image-2019-05-02-18-11-28-307.png, simple_skip.png > > > Task with trigger_rule = 'none_failed' cascading *skipped *status to > downstream task > * task have multiple upstream tasks > * trigger_rule set to 'none_failed' > * some of upstream tasks can be skipped due to *latest only* > Basing on documentation this shouldn't happen > !image-2019-05-02-18-11-28-307.png|width=655,height=372! > DAG attached -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4453) none_failed trigger rule cascading skipped state to downstream tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010257#comment-17010257 ] Joel Croteau commented on AIRFLOW-4453: --- [~kamil.bregula], added an extra step with a dummy rule to my test workflow. `step_3` executed. The rest of the behavior was unchanged. !3_step.png! > none_failed trigger rule cascading skipped state to downstream tasks > > > Key: AIRFLOW-4453 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4453 > Project: Apache Airflow > Issue Type: Bug > Components: DAG >Affects Versions: 1.10.3 >Reporter: Dmytro Kulyk >Assignee: Kaxil Naik >Priority: Major > Labels: skipped > Fix For: 1.10.5 > > Attachments: 3_step.png, cube_update.py, > image-2019-05-02-18-11-28-307.png, simple_skip.png > > > Task with trigger_rule = 'none_failed' cascading *skipped *status to > downstream task > * task have multiple upstream tasks > * trigger_rule set to 'none_failed' > * some of upstream tasks can be skipped due to *latest only* > Basing on documentation this shouldn't happen > !image-2019-05-02-18-11-28-307.png|width=655,height=372! > DAG attached -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4453) none_failed trigger rule cascading skipped state to downstream tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010244#comment-17010244 ] ASF GitHub Bot commented on AIRFLOW-4453: - TV4Fun commented on pull request #7098: [AIRFLOW-4453] Make behavior of `none_failed` consistent with documentation URL: https://github.com/apache/airflow/pull/7098 The documentation for the `none_failed` trigger rule (https://airflow.apache.org/docs/stable/concepts.html#trigger-rules) describes its behavior as "all parents have not failed (`failed` or `upstream_failed`) i.e. all parents have succeeded or been skipped." With that definition in mind, there is no reason that the check for `none_failed` should ever have to check for skipped upstream tasks or set the current task to `skipped`. The current behavior causes the rule to skip if all upstream tasks have skipped, which is more than a little confusing. This fixes the behavior to be consistent with the documentation. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) - [X] Description above provides context of the change - [X] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [X] Unit tests coverage for changes (not needed for documentation changes) - [X] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [X] Relevant documentation is updated including usage instructions. - [X] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > none_failed trigger rule cascading skipped state to downstream tasks > > > Key: AIRFLOW-4453 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4453 > Project: Apache Airflow > Issue Type: Bug > Components: DAG >Affects Versions: 1.10.3 >Reporter: Dmytro Kulyk >Assignee: Kaxil Naik >Priority: Major > Labels: skipped > Fix For: 1.10.5 > > Attachments: cube_update.py, image-2019-05-02-18-11-28-307.png, > simple_skip.png > > > Task with trigger_rule = 'none_failed' cascading *skipped *status to > downstream task > * task have multiple upstream tasks > * trigger_rule set to 'none_failed' > * some of upstream tasks can be skipped due to *latest only* > Basing on documentation this shouldn't happen > !image-2019-05-02-18-11-28-307.png|width=655,height=372! > DAG attached -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4453) none_failed trigger rule cascading skipped state to downstream tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010240#comment-17010240 ] Kamil Bregula commented on AIRFLOW-4453: Perfectly. Can you create PR then? > none_failed trigger rule cascading skipped state to downstream tasks > > > Key: AIRFLOW-4453 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4453 > Project: Apache Airflow > Issue Type: Bug > Components: DAG >Affects Versions: 1.10.3 >Reporter: Dmytro Kulyk >Assignee: Kaxil Naik >Priority: Major > Labels: skipped > Fix For: 1.10.5 > > Attachments: cube_update.py, image-2019-05-02-18-11-28-307.png, > simple_skip.png > > > Task with trigger_rule = 'none_failed' cascading *skipped *status to > downstream task > * task have multiple upstream tasks > * trigger_rule set to 'none_failed' > * some of upstream tasks can be skipped due to *latest only* > Basing on documentation this shouldn't happen > !image-2019-05-02-18-11-28-307.png|width=655,height=372! > DAG attached -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4453) none_failed trigger rule cascading skipped state to downstream tasks
[
https://issues.apache.org/jira/browse/AIRFLOW-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010239#comment-17010239
]
Kamil Bregula commented on AIRFLOW-4453:
Can you check what happens if you add one more task at the end?
{code:python}
step_3 = DummyOperator(task_id='step_3, trigger_rule='dummy')
step 1 >> step_3
{code}
> none_failed trigger rule cascading skipped state to downstream tasks
>
>
> Key: AIRFLOW-4453
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4453
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG
>Affects Versions: 1.10.3
>Reporter: Dmytro Kulyk
>Assignee: Kaxil Naik
>Priority: Major
> Labels: skipped
> Fix For: 1.10.5
>
> Attachments: cube_update.py, image-2019-05-02-18-11-28-307.png,
> simple_skip.png
>
>
> Task with trigger_rule = 'none_failed' cascading *skipped *status to
> downstream task
> * task have multiple upstream tasks
> * trigger_rule set to 'none_failed'
> * some of upstream tasks can be skipped due to *latest only*
> Basing on documentation this shouldn't happen
> !image-2019-05-02-18-11-28-307.png|width=655,height=372!
> DAG attached
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4453) none_failed trigger rule cascading skipped state to downstream tasks
[
https://issues.apache.org/jira/browse/AIRFLOW-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010232#comment-17010232
]
Joel Croteau commented on AIRFLOW-4453:
---
The problem seems to be [this
code|https://github.com/kaxil/airflow/blob/master/airflow/ti_deps/deps/trigger_rule_dep.py#L157]:
{code:python}
elif tr == TR.NONE_FAILED:
if upstream_failed or failed:
ti.set_state(State.UPSTREAM_FAILED, session)
elif skipped == upstream:
ti.set_state(State.SKIPPED, session)
{code}
This causes a task to skip if all upstream dependencies are skipped, which
directly contradicts [the
documentation|https://airflow.apache.org/docs/stable/concepts.html#trigger-rules],
which says "`none_failed`: all parents have not failed (`failed` or
`upstream_failed`) i.e. all parents have succeeded or been skipped." There
shouldn't be any reason that `none_failed` should ever have to look at skipped
upstream tasks, so we should really just get rid of the second check altogether.
> none_failed trigger rule cascading skipped state to downstream tasks
>
>
> Key: AIRFLOW-4453
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4453
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG
>Affects Versions: 1.10.3
>Reporter: Dmytro Kulyk
>Assignee: Kaxil Naik
>Priority: Major
> Labels: skipped
> Fix For: 1.10.5
>
> Attachments: cube_update.py, image-2019-05-02-18-11-28-307.png,
> simple_skip.png
>
>
> Task with trigger_rule = 'none_failed' cascading *skipped *status to
> downstream task
> * task have multiple upstream tasks
> * trigger_rule set to 'none_failed'
> * some of upstream tasks can be skipped due to *latest only*
> Basing on documentation this shouldn't happen
> !image-2019-05-02-18-11-28-307.png|width=655,height=372!
> DAG attached
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4453) none_failed trigger rule cascading skipped state to downstream tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010228#comment-17010228 ] Kamil Bregula commented on AIRFLOW-4453: I am so sorry, my bad. I will correct it in a moment then. > none_failed trigger rule cascading skipped state to downstream tasks > > > Key: AIRFLOW-4453 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4453 > Project: Apache Airflow > Issue Type: Bug > Components: DAG >Affects Versions: 1.10.3 >Reporter: Dmytro Kulyk >Assignee: Kaxil Naik >Priority: Major > Labels: skipped > Fix For: 1.10.5 > > Attachments: cube_update.py, image-2019-05-02-18-11-28-307.png, > simple_skip.png > > > Task with trigger_rule = 'none_failed' cascading *skipped *status to > downstream task > * task have multiple upstream tasks > * trigger_rule set to 'none_failed' > * some of upstream tasks can be skipped due to *latest only* > Basing on documentation this shouldn't happen > !image-2019-05-02-18-11-28-307.png|width=655,height=372! > DAG attached -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4453) none_failed trigger rule cascading skipped state to downstream tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010208#comment-17010208 ] Joel Croteau commented on AIRFLOW-4453: --- [~kamil.bregula], this ticket is about a bug in the `none_failed` trigger rule. The issue you linked is about `none_skipped`. I'm not seeing anything in your PR that changes the behavior of `none_failed`. Can you explained what was fixed here? > none_failed trigger rule cascading skipped state to downstream tasks > > > Key: AIRFLOW-4453 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4453 > Project: Apache Airflow > Issue Type: Bug > Components: DAG >Affects Versions: 1.10.3 >Reporter: Dmytro Kulyk >Assignee: Kaxil Naik >Priority: Major > Labels: skipped > Fix For: 1.10.5 > > Attachments: cube_update.py, image-2019-05-02-18-11-28-307.png, > simple_skip.png > > > Task with trigger_rule = 'none_failed' cascading *skipped *status to > downstream task > * task have multiple upstream tasks > * trigger_rule set to 'none_failed' > * some of upstream tasks can be skipped due to *latest only* > Basing on documentation this shouldn't happen > !image-2019-05-02-18-11-28-307.png|width=655,height=372! > DAG attached -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4453) none_failed trigger rule cascading skipped state to downstream tasks
[
https://issues.apache.org/jira/browse/AIRFLOW-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17007203#comment-17007203
]
Joel Croteau commented on AIRFLOW-4453:
---
I can actually reproduce this on 1.10.3 with a much simpler DAG:
{code:java}
import datetime
from airflow.models import SkipMixin, BaseOperator, DAG
from airflow.operators.dummy_operator import DummyOperator
class SkipOperator(BaseOperator, SkipMixin):
def execute(self, context=None):
# Skip all downstream tasks
ti = context['ti']
dag_run = ti.get_dagrun()
task = ti.task
downstream_tasks = task.downstream_list
if downstream_tasks:
self.skip(dag_run, ti.execution_date, downstream_tasks)
with DAG(
schedule_interval=None,
dag_id='simple_skip',
start_date=datetime.datetime(2019, 12, 18),
) as dag:
skip_operator = SkipOperator(task_id='skip_operator')
step_1 = DummyOperator(task_id='step_1')
step_2 = DummyOperator(task_id='step_2', trigger_rule='none_failed')
skip_operator >> step_1 >> step_2
{code}
This also skips on 1.10.3. I haven't tried it in newer versions.
!simple_skip.png!
> none_failed trigger rule cascading skipped state to downstream tasks
>
>
> Key: AIRFLOW-4453
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4453
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG
>Affects Versions: 1.10.3
>Reporter: Dmytro Kulyk
>Priority: Major
> Labels: skipped
> Attachments: cube_update.py, image-2019-05-02-18-11-28-307.png,
> simple_skip.png
>
>
> Task with trigger_rule = 'none_failed' cascading *skipped *status to
> downstream task
> * task have multiple upstream tasks
> * trigger_rule set to 'none_failed'
> * some of upstream tasks can be skipped due to *latest only*
> Basing on documentation this shouldn't happen
> !image-2019-05-02-18-11-28-307.png|width=655,height=372!
> DAG attached
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4453) none_failed trigger rule cascading skipped state to downstream tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17007170#comment-17007170 ] Joel Croteau commented on AIRFLOW-4453: --- [https://xkcd.com/979/.|https://xkcd.com/979/] Definitely seeing this in 1.10.3 (what Cloud Composer is using). I don't know if it's been fixed in newer versions. > none_failed trigger rule cascading skipped state to downstream tasks > > > Key: AIRFLOW-4453 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4453 > Project: Apache Airflow > Issue Type: Bug > Components: DAG >Affects Versions: 1.10.3 >Reporter: Dmytro Kulyk >Priority: Major > Labels: skipped > Attachments: cube_update.py, image-2019-05-02-18-11-28-307.png > > > Task with trigger_rule = 'none_failed' cascading *skipped *status to > downstream task > * task have multiple upstream tasks > * trigger_rule set to 'none_failed' > * some of upstream tasks can be skipped due to *latest only* > Basing on documentation this shouldn't happen > !image-2019-05-02-18-11-28-307.png|width=655,height=372! > DAG attached -- This message was sent by Atlassian Jira (v8.3.4#803005)
