[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17104495#comment-17104495 ] ASF subversion and git services commented on AIRFLOW-4549: -- Commit 2ec01300990edcb5b96c8198c20ae8c8877313e1 in airflow's branch refs/heads/master from Teddy Hartanto [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=2ec0130 ] [AIRFLOW-4549] Allow skipped tasks to satisfy wait_for_downstream (#7735) Previously, tasks that were in SUCCESS or SKIPPED state satisfy the depends_on_past check, but only tasks that were in the SUCCESS state satisfy the wait_for_downstream check. The inconsistency in behavior made the API less intuitive to users. > wait_for_downstream does not respect skipped tasks > -- > > Key: AIRFLOW-4549 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4549 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Dima Kamalov >Assignee: Teddy Hartanto >Priority: Major > > See > [http://mail-archives.apache.org/mod_mbox/airflow-dev/201609.mbox/%3ccaheep7utgpjvkgww9_9n5fupnu+pskf3rmbvxugk5dxb6bh...@mail.gmail.com%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17104231#comment-17104231 ] ASF GitHub Bot commented on AIRFLOW-4549: - TeddyHartanto edited a comment on pull request #7735: URL: https://github.com/apache/airflow/pull/7735#issuecomment-62600 @kaxil @milton0825 seems like we're all good... except for the quarantined tests. We just ignore that, right? Do you guys need me to do anything else on my end (squashing etc)? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > wait_for_downstream does not respect skipped tasks > -- > > Key: AIRFLOW-4549 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4549 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Dima Kamalov >Assignee: Teddy Hartanto >Priority: Major > > See > [http://mail-archives.apache.org/mod_mbox/airflow-dev/201609.mbox/%3ccaheep7utgpjvkgww9_9n5fupnu+pskf3rmbvxugk5dxb6bh...@mail.gmail.com%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17104207#comment-17104207 ] ASF GitHub Bot commented on AIRFLOW-4549: - TeddyHartanto commented on pull request #7735: URL: https://github.com/apache/airflow/pull/7735#issuecomment-62600 @kaxil @milton0825 seems like we're all good... except for the quarantined tests. I'm not totally sure what "quarantined tests" mean, but seems like it's a bit flaky, isn't it? Do we just ignore it? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > wait_for_downstream does not respect skipped tasks > -- > > Key: AIRFLOW-4549 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4549 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Dima Kamalov >Assignee: Teddy Hartanto >Priority: Major > > See > [http://mail-archives.apache.org/mod_mbox/airflow-dev/201609.mbox/%3ccaheep7utgpjvkgww9_9n5fupnu+pskf3rmbvxugk5dxb6bh...@mail.gmail.com%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102549#comment-17102549 ] ASF GitHub Bot commented on AIRFLOW-4549: - TeddyHartanto commented on a change in pull request #7735: URL: https://github.com/apache/airflow/pull/7735#discussion_r422125930 ## File path: tests/dags/test_issue_1225.py ## @@ -47,12 +47,7 @@ def fail(): dag=dag1, pool='test_backfill_pooled_task_pool',) -# DAG tests depends_on_past dependencies -dag2 = DAG(dag_id='test_depends_on_past', default_args=default_args) -dag2_task1 = DummyOperator( -task_id='test_dop_task', -dag=dag2, -depends_on_past=True,) +# dag2 has been moved to test_past_dagrun_deps.py Review comment: Good catch! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > wait_for_downstream does not respect skipped tasks > -- > > Key: AIRFLOW-4549 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4549 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Dima Kamalov >Assignee: Teddy Hartanto >Priority: Major > > See > [http://mail-archives.apache.org/mod_mbox/airflow-dev/201609.mbox/%3ccaheep7utgpjvkgww9_9n5fupnu+pskf3rmbvxugk5dxb6bh...@mail.gmail.com%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102228#comment-17102228 ] ASF GitHub Bot commented on AIRFLOW-4549: - TeddyHartanto commented on a change in pull request #7735: URL: https://github.com/apache/airflow/pull/7735#discussion_r421923733 ## File path: tests/dags/test_issue_1225.py ## @@ -47,12 +47,7 @@ def fail(): dag=dag1, pool='test_backfill_pooled_task_pool',) -# DAG tests depends_on_past dependencies -dag2 = DAG(dag_id='test_depends_on_past', default_args=default_args) -dag2_task1 = DummyOperator( -task_id='test_dop_task', -dag=dag2, -depends_on_past=True,) +# dag2 has been moved to test_past_dagrun_deps.py Review comment: Sharp eyes! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > wait_for_downstream does not respect skipped tasks > -- > > Key: AIRFLOW-4549 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4549 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Dima Kamalov >Assignee: Teddy Hartanto >Priority: Major > > See > [http://mail-archives.apache.org/mod_mbox/airflow-dev/201609.mbox/%3ccaheep7utgpjvkgww9_9n5fupnu+pskf3rmbvxugk5dxb6bh...@mail.gmail.com%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102227#comment-17102227 ] ASF GitHub Bot commented on AIRFLOW-4549: - TeddyHartanto commented on a change in pull request #7735: URL: https://github.com/apache/airflow/pull/7735#discussion_r421923733 ## File path: tests/dags/test_issue_1225.py ## @@ -47,12 +47,7 @@ def fail(): dag=dag1, pool='test_backfill_pooled_task_pool',) -# DAG tests depends_on_past dependencies -dag2 = DAG(dag_id='test_depends_on_past', default_args=default_args) -dag2_task1 = DummyOperator( -task_id='test_dop_task', -dag=dag2, -depends_on_past=True,) +# dag2 has been moved to test_past_dagrun_deps.py Review comment: Sharp eyes! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > wait_for_downstream does not respect skipped tasks > -- > > Key: AIRFLOW-4549 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4549 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Dima Kamalov >Assignee: Teddy Hartanto >Priority: Major > > See > [http://mail-archives.apache.org/mod_mbox/airflow-dev/201609.mbox/%3ccaheep7utgpjvkgww9_9n5fupnu+pskf3rmbvxugk5dxb6bh...@mail.gmail.com%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102226#comment-17102226 ] ASF GitHub Bot commented on AIRFLOW-4549: - TeddyHartanto commented on a change in pull request #7735: URL: https://github.com/apache/airflow/pull/7735#discussion_r421923688 ## File path: tests/models/test_dagrun.py ## @@ -552,3 +561,131 @@ def with_all_tasks_removed(dag): dagrun.verify_integrity() flaky_ti.refresh_from_db() self.assertEqual(State.NONE, flaky_ti.state) + +def test_depends_on_past(self): +# dag_id = 'test_depends_on_past' Review comment: OK got it. Since the tests pass anyways without any scheduler deadlock, there shouldn't be any problem. I will remove the comments This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > wait_for_downstream does not respect skipped tasks > -- > > Key: AIRFLOW-4549 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4549 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Dima Kamalov >Assignee: Teddy Hartanto >Priority: Major > > See > [http://mail-archives.apache.org/mod_mbox/airflow-dev/201609.mbox/%3ccaheep7utgpjvkgww9_9n5fupnu+pskf3rmbvxugk5dxb6bh...@mail.gmail.com%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102169#comment-17102169 ] ASF GitHub Bot commented on AIRFLOW-4549: - kaxil commented on a change in pull request #7735: URL: https://github.com/apache/airflow/pull/7735#discussion_r421894092 ## File path: tests/models/test_dagrun.py ## @@ -552,3 +561,131 @@ def with_all_tasks_removed(dag): dagrun.verify_integrity() flaky_ti.refresh_from_db() self.assertEqual(State.NONE, flaky_ti.state) + +def test_depends_on_past(self): +# dag_id = 'test_depends_on_past' Review comment: yes, remove the comments in the PR, should be fine This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > wait_for_downstream does not respect skipped tasks > -- > > Key: AIRFLOW-4549 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4549 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Dima Kamalov >Assignee: Teddy Hartanto >Priority: Major > > See > [http://mail-archives.apache.org/mod_mbox/airflow-dev/201609.mbox/%3ccaheep7utgpjvkgww9_9n5fupnu+pskf3rmbvxugk5dxb6bh...@mail.gmail.com%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102165#comment-17102165 ] ASF GitHub Bot commented on AIRFLOW-4549: - kaxil commented on a change in pull request #7735: URL: https://github.com/apache/airflow/pull/7735#discussion_r421893773 ## File path: tests/models/test_dagrun.py ## @@ -552,3 +561,131 @@ def with_all_tasks_removed(dag): dagrun.verify_integrity() flaky_ti.refresh_from_db() self.assertEqual(State.NONE, flaky_ti.state) + +def test_depends_on_past(self): +# dag_id = 'test_depends_on_past' Review comment: Scheduler deadlock in which case, there are known issues of Scheduler deadlock if MySql. Other than that I am not sure if there are any This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > wait_for_downstream does not respect skipped tasks > -- > > Key: AIRFLOW-4549 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4549 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Dima Kamalov >Assignee: Teddy Hartanto >Priority: Major > > See > [http://mail-archives.apache.org/mod_mbox/airflow-dev/201609.mbox/%3ccaheep7utgpjvkgww9_9n5fupnu+pskf3rmbvxugk5dxb6bh...@mail.gmail.com%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102164#comment-17102164 ] ASF GitHub Bot commented on AIRFLOW-4549: - kaxil commented on a change in pull request #7735: URL: https://github.com/apache/airflow/pull/7735#discussion_r421893536 ## File path: tests/dags/test_issue_1225.py ## @@ -47,12 +47,7 @@ def fail(): dag=dag1, pool='test_backfill_pooled_task_pool',) -# DAG tests depends_on_past dependencies -dag2 = DAG(dag_id='test_depends_on_past', default_args=default_args) -dag2_task1 = DummyOperator( -task_id='test_dop_task', -dag=dag2, -depends_on_past=True,) +# dag2 has been moved to test_past_dagrun_deps.py Review comment: ```suggestion # dag2 has been moved to test_prev_dagrun_deps.py ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > wait_for_downstream does not respect skipped tasks > -- > > Key: AIRFLOW-4549 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4549 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Dima Kamalov >Assignee: Teddy Hartanto >Priority: Major > > See > [http://mail-archives.apache.org/mod_mbox/airflow-dev/201609.mbox/%3ccaheep7utgpjvkgww9_9n5fupnu+pskf3rmbvxugk5dxb6bh...@mail.gmail.com%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101679#comment-17101679 ] ASF GitHub Bot commented on AIRFLOW-4549: - TeddyHartanto commented on a change in pull request #7735: URL: https://github.com/apache/airflow/pull/7735#discussion_r421504185 ## File path: tests/dags/test_issue_1225.py ## @@ -47,12 +47,7 @@ def fail(): dag=dag1, pool='test_backfill_pooled_task_pool',) -# DAG tests depends_on_past dependencies -dag2 = DAG(dag_id='test_depends_on_past', default_args=default_args) -dag2_task1 = DummyOperator( -task_id='test_dop_task', -dag=dag2, -depends_on_past=True,) +# dag2 has been moved to test_past_dagrun_deps.py Review comment: To further support this argument, both `depends_on_past=True` and `wait_for_downstream=True` are used in a file named `prev_dagrun_dep.py`. Oh btw I just noticed that the test file then should be called `test_prev_dagrun_deps.py` instead of `test_past_dagrun_deps.py`. On it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > wait_for_downstream does not respect skipped tasks > -- > > Key: AIRFLOW-4549 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4549 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Dima Kamalov >Assignee: Teddy Hartanto >Priority: Major > > See > [http://mail-archives.apache.org/mod_mbox/airflow-dev/201609.mbox/%3ccaheep7utgpjvkgww9_9n5fupnu+pskf3rmbvxugk5dxb6bh...@mail.gmail.com%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101675#comment-17101675 ] ASF GitHub Bot commented on AIRFLOW-4549: - TeddyHartanto commented on a change in pull request #7735: URL: https://github.com/apache/airflow/pull/7735#discussion_r421499807 ## File path: tests/dags/test_issue_1225.py ## @@ -47,12 +47,7 @@ def fail(): dag=dag1, pool='test_backfill_pooled_task_pool',) -# DAG tests depends_on_past dependencies -dag2 = DAG(dag_id='test_depends_on_past', default_args=default_args) -dag2_task1 = DummyOperator( -task_id='test_dop_task', -dag=dag2, -depends_on_past=True,) +# dag2 has been moved to test_past_dagrun_deps.py Review comment: Yes, indeed. It only has a DAG with `depends_on_past=True`. But, because I needed to add a new DAG with `wait_for_downstream=True` anyway in a file called `tests/dags/test_past_dagrun_deps.py`, it made more sense to put the `depends_on_past=True` DAG in the same file because they are logically related. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > wait_for_downstream does not respect skipped tasks > -- > > Key: AIRFLOW-4549 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4549 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Dima Kamalov >Assignee: Teddy Hartanto >Priority: Major > > See > [http://mail-archives.apache.org/mod_mbox/airflow-dev/201609.mbox/%3ccaheep7utgpjvkgww9_9n5fupnu+pskf3rmbvxugk5dxb6bh...@mail.gmail.com%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101672#comment-17101672 ] ASF GitHub Bot commented on AIRFLOW-4549: - TeddyHartanto commented on a change in pull request #7735: URL: https://github.com/apache/airflow/pull/7735#discussion_r421499807 ## File path: tests/dags/test_issue_1225.py ## @@ -47,12 +47,7 @@ def fail(): dag=dag1, pool='test_backfill_pooled_task_pool',) -# DAG tests depends_on_past dependencies -dag2 = DAG(dag_id='test_depends_on_past', default_args=default_args) -dag2_task1 = DummyOperator( -task_id='test_dop_task', -dag=dag2, -depends_on_past=True,) +# dag2 has been moved to test_past_dagrun_deps.py Review comment: But, because I needed to add a new DAG with `wait_for_downstream=True` anyway in a file called `tests/dags/test_past_dagrun_deps.py`, it made more sense to put the `depends_on_past=True` DAG in the same file because they are logically related. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > wait_for_downstream does not respect skipped tasks > -- > > Key: AIRFLOW-4549 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4549 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Dima Kamalov >Assignee: Teddy Hartanto >Priority: Major > > See > [http://mail-archives.apache.org/mod_mbox/airflow-dev/201609.mbox/%3ccaheep7utgpjvkgww9_9n5fupnu+pskf3rmbvxugk5dxb6bh...@mail.gmail.com%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101671#comment-17101671 ] ASF GitHub Bot commented on AIRFLOW-4549: - TeddyHartanto commented on a change in pull request #7735: URL: https://github.com/apache/airflow/pull/7735#discussion_r421500793 ## File path: tests/dags/test_issue_1225.py ## @@ -47,12 +47,7 @@ def fail(): dag=dag1, pool='test_backfill_pooled_task_pool',) -# DAG tests depends_on_past dependencies -dag2 = DAG(dag_id='test_depends_on_past', default_args=default_args) -dag2_task1 = DummyOperator( -task_id='test_dop_task', -dag=dag2, -depends_on_past=True,) +# dag2 has been moved to test_past_dagrun_deps.py Review comment: depends_on_past=True ## File path: tests/dags/test_issue_1225.py ## @@ -47,12 +47,7 @@ def fail(): dag=dag1, pool='test_backfill_pooled_task_pool',) -# DAG tests depends_on_past dependencies -dag2 = DAG(dag_id='test_depends_on_past', default_args=default_args) -dag2_task1 = DummyOperator( -task_id='test_dop_task', -dag=dag2, -depends_on_past=True,) +# dag2 has been moved to test_past_dagrun_deps.py Review comment: depends_on_past=True This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > wait_for_downstream does not respect skipped tasks > -- > > Key: AIRFLOW-4549 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4549 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Dima Kamalov >Assignee: Teddy Hartanto >Priority: Major > > See > [http://mail-archives.apache.org/mod_mbox/airflow-dev/201609.mbox/%3ccaheep7utgpjvkgww9_9n5fupnu+pskf3rmbvxugk5dxb6bh...@mail.gmail.com%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks
[
https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101667#comment-17101667
]
ASF GitHub Bot commented on AIRFLOW-4549:
-
TeddyHartanto commented on a change in pull request #7735:
URL: https://github.com/apache/airflow/pull/7735#discussion_r421499807
##
File path: tests/dags/test_issue_1225.py
##
@@ -47,12 +47,7 @@ def fail():
dag=dag1,
pool='test_backfill_pooled_task_pool',)
-# DAG tests depends_on_past dependencies
-dag2 = DAG(dag_id='test_depends_on_past', default_args=default_args)
-dag2_task1 = DummyOperator(
-task_id='test_dop_task',
-dag=dag2,
-depends_on_past=True,)
+# dag2 has been moved to test_past_dagrun_deps.py
Review comment:
Yes, it only has a DAG with `depends_on_past=True`. But because this DAG
is closely related to the test DAG with `wait_for_downstream=True`, I reasoned
that we'd be better off to put them together in a new file called
`tests/dags/test_past_dagrun_deps.py`, because anyway these 2 flags are used in
`prev_dagrun_deps.py`:
```python
if previous_ti.state not in {State.SKIPPED, State.SUCCESS}:
yield self._failing_status(
reason="depends_on_past is true for this task, but the
previous task "
"instance {0} is in the state '{1}' which is not a
successful "
"state.".format(previous_ti, previous_ti.state))
previous_ti.task = ti.task
if (ti.task.wait_for_downstream and
not previous_ti.are_dependents_done(session=session)):
yield self._failing_status(
reason="The tasks downstream of the previous task instance
{0} haven't "
"completed (and wait_for_downstream is
True).".format(previous_ti))
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> wait_for_downstream does not respect skipped tasks
> --
>
> Key: AIRFLOW-4549
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4549
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
>Reporter: Dima Kamalov
>Assignee: Teddy Hartanto
>Priority: Major
>
> See
> [http://mail-archives.apache.org/mod_mbox/airflow-dev/201609.mbox/%3ccaheep7utgpjvkgww9_9n5fupnu+pskf3rmbvxugk5dxb6bh...@mail.gmail.com%3E]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101620#comment-17101620 ] ASF GitHub Bot commented on AIRFLOW-4549: - kaxil commented on a change in pull request #7735: URL: https://github.com/apache/airflow/pull/7735#discussion_r421463685 ## File path: tests/dags/test_issue_1225.py ## @@ -47,12 +47,7 @@ def fail(): dag=dag1, pool='test_backfill_pooled_task_pool',) -# DAG tests depends_on_past dependencies -dag2 = DAG(dag_id='test_depends_on_past', default_args=default_args) -dag2_task1 = DummyOperator( -task_id='test_dop_task', -dag=dag2, -depends_on_past=True,) +# dag2 has been moved to test_past_dagrun_deps.py Review comment: I can't see `wait_for_downstream=True` in `tests/dags/test_issue_1225.py` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > wait_for_downstream does not respect skipped tasks > -- > > Key: AIRFLOW-4549 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4549 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Dima Kamalov >Assignee: Teddy Hartanto >Priority: Major > > See > [http://mail-archives.apache.org/mod_mbox/airflow-dev/201609.mbox/%3ccaheep7utgpjvkgww9_9n5fupnu+pskf3rmbvxugk5dxb6bh...@mail.gmail.com%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101329#comment-17101329 ] ASF GitHub Bot commented on AIRFLOW-4549: - TeddyHartanto commented on a change in pull request #7735: URL: https://github.com/apache/airflow/pull/7735#discussion_r421209778 ## File path: tests/dags/test_issue_1225.py ## @@ -47,12 +47,7 @@ def fail(): dag=dag1, pool='test_backfill_pooled_task_pool',) -# DAG tests depends_on_past dependencies -dag2 = DAG(dag_id='test_depends_on_past', default_args=default_args) -dag2_task1 = DummyOperator( -task_id='test_dop_task', -dag=dag2, -depends_on_past=True,) +# dag2 has been moved to test_past_dagrun_deps.py Review comment: Because `tests/dags/test_issue_1225.py` isn't descriptive. I think it's clearer if we move it to `tests/dags/test_past_dagrun_deps.py`, which houses the DAGs with `depends_on_past=True`, and `wait_for_downstream=True`. OA did the same thing and I agree. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > wait_for_downstream does not respect skipped tasks > -- > > Key: AIRFLOW-4549 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4549 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Dima Kamalov >Assignee: Teddy Hartanto >Priority: Major > > See > [http://mail-archives.apache.org/mod_mbox/airflow-dev/201609.mbox/%3ccaheep7utgpjvkgww9_9n5fupnu+pskf3rmbvxugk5dxb6bh...@mail.gmail.com%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101326#comment-17101326 ] ASF GitHub Bot commented on AIRFLOW-4549: - TeddyHartanto commented on a change in pull request #7735: URL: https://github.com/apache/airflow/pull/7735#discussion_r421208697 ## File path: tests/models/test_dagrun.py ## @@ -552,3 +561,131 @@ def with_all_tasks_removed(dag): dagrun.verify_integrity() flaky_ti.refresh_from_db() self.assertEqual(State.NONE, flaky_ti.state) + +def test_depends_on_past(self): +# dag_id = 'test_depends_on_past' Review comment: Oh, actually, the tests that I included are the modified versions of the original tests. The main difference are as follow: 1. Original Author (OA) uses `dag.run()`, while I uses `task_instance.run()` 2. OA mentioned that he encountered a "scheduler deadlock", which I didn't encounter. I wonder if you know anything about the scheduler deadlock? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > wait_for_downstream does not respect skipped tasks > -- > > Key: AIRFLOW-4549 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4549 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Dima Kamalov >Assignee: Teddy Hartanto >Priority: Major > > See > [http://mail-archives.apache.org/mod_mbox/airflow-dev/201609.mbox/%3ccaheep7utgpjvkgww9_9n5fupnu+pskf3rmbvxugk5dxb6bh...@mail.gmail.com%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101325#comment-17101325 ] ASF GitHub Bot commented on AIRFLOW-4549: - TeddyHartanto commented on a change in pull request #7735: URL: https://github.com/apache/airflow/pull/7735#discussion_r421208697 ## File path: tests/models/test_dagrun.py ## @@ -552,3 +561,131 @@ def with_all_tasks_removed(dag): dagrun.verify_integrity() flaky_ti.refresh_from_db() self.assertEqual(State.NONE, flaky_ti.state) + +def test_depends_on_past(self): +# dag_id = 'test_depends_on_past' Review comment: Oh, actually, the tests that I included are the modified versions of the original tests. The main difference are as follow: 1. Original Author (OA) uses `dag.run()`, while I uses `task_instance.run()` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > wait_for_downstream does not respect skipped tasks > -- > > Key: AIRFLOW-4549 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4549 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Dima Kamalov >Assignee: Teddy Hartanto >Priority: Major > > See > [http://mail-archives.apache.org/mod_mbox/airflow-dev/201609.mbox/%3ccaheep7utgpjvkgww9_9n5fupnu+pskf3rmbvxugk5dxb6bh...@mail.gmail.com%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101264#comment-17101264 ] ASF GitHub Bot commented on AIRFLOW-4549: - kaxil commented on a change in pull request #7735: URL: https://github.com/apache/airflow/pull/7735#discussion_r421151508 ## File path: tests/models/test_dagrun.py ## @@ -552,3 +561,131 @@ def with_all_tasks_removed(dag): dagrun.verify_integrity() flaky_ti.refresh_from_db() self.assertEqual(State.NONE, flaky_ti.state) + +def test_depends_on_past(self): +# dag_id = 'test_depends_on_past' Review comment: If the test don't take a significant amount of time to run then we can have these tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > wait_for_downstream does not respect skipped tasks > -- > > Key: AIRFLOW-4549 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4549 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Dima Kamalov >Assignee: Teddy Hartanto >Priority: Major > > See > [http://mail-archives.apache.org/mod_mbox/airflow-dev/201609.mbox/%3ccaheep7utgpjvkgww9_9n5fupnu+pskf3rmbvxugk5dxb6bh...@mail.gmail.com%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101258#comment-17101258 ] ASF GitHub Bot commented on AIRFLOW-4549: - kaxil commented on a change in pull request #7735: URL: https://github.com/apache/airflow/pull/7735#discussion_r421147807 ## File path: tests/dags/test_issue_1225.py ## @@ -47,12 +47,7 @@ def fail(): dag=dag1, pool='test_backfill_pooled_task_pool',) -# DAG tests depends_on_past dependencies -dag2 = DAG(dag_id='test_depends_on_past', default_args=default_args) -dag2_task1 = DummyOperator( -task_id='test_dop_task', -dag=dag2, -depends_on_past=True,) +# dag2 has been moved to test_past_dagrun_deps.py Review comment: Why do we need to move it from here ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > wait_for_downstream does not respect skipped tasks > -- > > Key: AIRFLOW-4549 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4549 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Dima Kamalov >Assignee: Teddy Hartanto >Priority: Major > > See > [http://mail-archives.apache.org/mod_mbox/airflow-dev/201609.mbox/%3ccaheep7utgpjvkgww9_9n5fupnu+pskf3rmbvxugk5dxb6bh...@mail.gmail.com%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100869#comment-17100869 ] ASF GitHub Bot commented on AIRFLOW-4549: - TeddyHartanto commented on pull request #7735: URL: https://github.com/apache/airflow/pull/7735#issuecomment-624699155 @kaxil btw I think @milton0825 is on vacation. So, we'll wait until he's available? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > wait_for_downstream does not respect skipped tasks > -- > > Key: AIRFLOW-4549 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4549 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Dima Kamalov >Assignee: Teddy Hartanto >Priority: Major > > See > [http://mail-archives.apache.org/mod_mbox/airflow-dev/201609.mbox/%3ccaheep7utgpjvkgww9_9n5fupnu+pskf3rmbvxugk5dxb6bh...@mail.gmail.com%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100868#comment-17100868 ] ASF GitHub Bot commented on AIRFLOW-4549: - TeddyHartanto commented on a change in pull request #7735: URL: https://github.com/apache/airflow/pull/7735#discussion_r420857303 ## File path: tests/models/test_taskinstance.py ## @@ -855,6 +855,25 @@ def test_check_task_dependencies(self, trigger_rule, successes, skipped, self.assertEqual(completed, expect_completed) self.assertEqual(ti.state, expect_state) +@parameterized.expand([ Review comment: Added! :) I really couldn't remember why I omitted it previously This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > wait_for_downstream does not respect skipped tasks > -- > > Key: AIRFLOW-4549 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4549 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Dima Kamalov >Assignee: Teddy Hartanto >Priority: Major > > See > [http://mail-archives.apache.org/mod_mbox/airflow-dev/201609.mbox/%3ccaheep7utgpjvkgww9_9n5fupnu+pskf3rmbvxugk5dxb6bh...@mail.gmail.com%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100866#comment-17100866 ] ASF GitHub Bot commented on AIRFLOW-4549: - TeddyHartanto commented on a change in pull request #7735: URL: https://github.com/apache/airflow/pull/7735#discussion_r420857303 ## File path: tests/models/test_taskinstance.py ## @@ -855,6 +855,25 @@ def test_check_task_dependencies(self, trigger_rule, successes, skipped, self.assertEqual(completed, expect_completed) self.assertEqual(ti.state, expect_state) +@parameterized.expand([ Review comment: Added! :) I really couldn't remember why I omitted it This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > wait_for_downstream does not respect skipped tasks > -- > > Key: AIRFLOW-4549 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4549 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Dima Kamalov >Assignee: Teddy Hartanto >Priority: Major > > See > [http://mail-archives.apache.org/mod_mbox/airflow-dev/201609.mbox/%3ccaheep7utgpjvkgww9_9n5fupnu+pskf3rmbvxugk5dxb6bh...@mail.gmail.com%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100842#comment-17100842 ] ASF GitHub Bot commented on AIRFLOW-4549: - TeddyHartanto commented on a change in pull request #7735: URL: https://github.com/apache/airflow/pull/7735#discussion_r420829036 ## File path: tests/models/test_taskinstance.py ## @@ -855,6 +855,25 @@ def test_check_task_dependencies(self, trigger_rule, successes, skipped, self.assertEqual(completed, expect_completed) self.assertEqual(ti.state, expect_state) +@parameterized.expand([ Review comment: Ah good point there. I somehow reasoned this wasn't necessary. I couldn't quite remember what the exact reason was. But it could be a misjudgement because now that I take a second look at it, I think we should add it in. Thanks for catching that! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > wait_for_downstream does not respect skipped tasks > -- > > Key: AIRFLOW-4549 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4549 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Dima Kamalov >Assignee: Teddy Hartanto >Priority: Major > > See > [http://mail-archives.apache.org/mod_mbox/airflow-dev/201609.mbox/%3ccaheep7utgpjvkgww9_9n5fupnu+pskf3rmbvxugk5dxb6bh...@mail.gmail.com%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks
[
https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100612#comment-17100612
]
ASF GitHub Bot commented on AIRFLOW-4549:
-
kaxil commented on a change in pull request #7735:
URL: https://github.com/apache/airflow/pull/7735#discussion_r420652841
##
File path: tests/models/test_taskinstance.py
##
@@ -855,6 +855,25 @@ def test_check_task_dependencies(self, trigger_rule,
successes, skipped,
self.assertEqual(completed, expect_completed)
self.assertEqual(ti.state, expect_state)
+@parameterized.expand([
Review comment:
There was another test in the original PR:
```python
def test_respects_prev_dagrun_dep(self):
dag = models.DAG(dag_id='test_dag')
task = DummyOperator(dag=dag, task_id='test_task',
start_date=DEFAULT_DATE)
ti = TI(task, DEFAULT_DATE)
failing_status = [TIDepStatus('test fail status name', False, 'test
fail reason')]
passing_status = [TIDepStatus('test pass status name', True, 'test
passing reason')]
with
patch('airflow.ti_deps.deps.prev_dagrun_dep.PrevDagrunDep.get_dep_statuses',
return_value=failing_status):
self.assertFalse(ti.are_dependencies_met())
with
patch('airflow.ti_deps.deps.prev_dagrun_dep.PrevDagrunDep.get_dep_statuses',
return_value=passing_status):
self.assertTrue(ti.are_dependencies_met())
```
Any reason why we didn't include it?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> wait_for_downstream does not respect skipped tasks
> --
>
> Key: AIRFLOW-4549
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4549
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
>Reporter: Dima Kamalov
>Assignee: Teddy Hartanto
>Priority: Major
>
> See
> [http://mail-archives.apache.org/mod_mbox/airflow-dev/201609.mbox/%3ccaheep7utgpjvkgww9_9n5fupnu+pskf3rmbvxugk5dxb6bh...@mail.gmail.com%3E]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098778#comment-17098778 ] ASF GitHub Bot commented on AIRFLOW-4549: - TeddyHartanto commented on pull request #7735: URL: https://github.com/apache/airflow/pull/7735#issuecomment-623337588 FYI: this PR deprecates the outdated PR #5308 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > wait_for_downstream does not respect skipped tasks > -- > > Key: AIRFLOW-4549 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4549 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Dima Kamalov >Assignee: Teddy Hartanto >Priority: Major > > See > [http://mail-archives.apache.org/mod_mbox/airflow-dev/201609.mbox/%3ccaheep7utgpjvkgww9_9n5fupnu+pskf3rmbvxugk5dxb6bh...@mail.gmail.com%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094132#comment-17094132 ] ASF GitHub Bot commented on AIRFLOW-4549: - stale[bot] commented on pull request #5308: URL: https://github.com/apache/airflow/pull/5308#issuecomment-620366004 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > wait_for_downstream does not respect skipped tasks > -- > > Key: AIRFLOW-4549 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4549 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Dima Kamalov >Assignee: Teddy Hartanto >Priority: Major > > See > [http://mail-archives.apache.org/mod_mbox/airflow-dev/201609.mbox/%3ccaheep7utgpjvkgww9_9n5fupnu+pskf3rmbvxugk5dxb6bh...@mail.gmail.com%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060177#comment-17060177 ] ASF GitHub Bot commented on AIRFLOW-4549: - TeddyHartanto commented on pull request #7735: [AIRFLOW-4549] Allow skipped tasks to satisfy wait_for_downstream URL: https://github.com/apache/airflow/pull/7735 Before this fix, tasks that are in SUCCESS or SKIPPED state satisfy the depends_on_past check, but only tasks that are in SUCCESS state satisfy the wait_for_downstream check. That makes the API inconsistent and unintuitive. --- Issue link: [AIRFLOW-4549](https://issues.apache.org/jira/browse/AIRFLOW-4549) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > wait_for_downstream does not respect skipped tasks > -- > > Key: AIRFLOW-4549 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4549 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Dima Kamalov >Assignee: Teddy Hartanto >Priority: Major > > See > [http://mail-archives.apache.org/mod_mbox/airflow-dev/201609.mbox/%3ccaheep7utgpjvkgww9_9n5fupnu+pskf3rmbvxugk5dxb6bh...@mail.gmail.com%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060176#comment-17060176 ] ASF GitHub Bot commented on AIRFLOW-4549: - TeddyHartanto commented on pull request #7735: [AIRFLOW-4549] Allow skipped tasks to satisfy wait_for_downstream URL: https://github.com/apache/airflow/pull/7735 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > wait_for_downstream does not respect skipped tasks > -- > > Key: AIRFLOW-4549 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4549 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Dima Kamalov >Assignee: Teddy Hartanto >Priority: Major > > See > [http://mail-archives.apache.org/mod_mbox/airflow-dev/201609.mbox/%3ccaheep7utgpjvkgww9_9n5fupnu+pskf3rmbvxugk5dxb6bh...@mail.gmail.com%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060173#comment-17060173 ] ASF GitHub Bot commented on AIRFLOW-4549: - TeddyHartanto commented on pull request #7735: [AIRFLOW-4549] Allow skipped tasks to satisfy wait_for_downstream URL: https://github.com/apache/airflow/pull/7735 Before this fix, tasks that are in SUCCESS or SKIPPED state satisfy the depends_on_past check, but only tasks that are in SUCCESS state satisfy the wait_for_downstream check. That makes the API inconsistent and unintuitive. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > wait_for_downstream does not respect skipped tasks > -- > > Key: AIRFLOW-4549 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4549 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Dima Kamalov >Assignee: Teddy Hartanto >Priority: Major > > See > [http://mail-archives.apache.org/mod_mbox/airflow-dev/201609.mbox/%3ccaheep7utgpjvkgww9_9n5fupnu+pskf3rmbvxugk5dxb6bh...@mail.gmail.com%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks
[
https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16844509#comment-16844509
]
ASF GitHub Bot commented on AIRFLOW-4549:
-
dima-asana commented on pull request #5308: [AIRFLOW-4549] skipped tasks should
be ok for wait_for_downstream
URL: https://github.com/apache/airflow/pull/5308
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-4549
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
this PR updates wait_for_downstream dependency to be satisfied for tasks
that are skipped. This is desirable because
(a) depends_on_past, which this is interacts with, allows skipped tasks
(b) documentation for wait_for_downstream mentions it is to avoid resource
conflicts, which are de facto avoided when a task is skipped
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
TaskInstanceTest.test_wait_for_downstream
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> wait_for_downstream does not respect skipped tasks
> --
>
> Key: AIRFLOW-4549
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4549
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
>Reporter: Dima Kamalov
>Assignee: Dima Kamalov
>Priority: Major
>
> See
> [http://mail-archives.apache.org/mod_mbox/airflow-dev/201609.mbox/%3ccaheep7utgpjvkgww9_9n5fupnu+pskf3rmbvxugk5dxb6bh...@mail.gmail.com%3E]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
