[jira] [Comment Edited] (AIRFLOW-1894) Rebase and migrate existing Airflow GCP operators to google-python-cloud
[ https://issues.apache.org/jira/browse/AIRFLOW-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637794#comment-16637794 ] Feng Lu edited comment on AIRFLOW-1894 at 10/4/18 4:16 AM: --- [~yiga2] it's on our to-do list, the current focus is to move all GCP operators from contrib to core. I am also working on a GCP operator development guide([https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY|https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY/edit?ts=5bb379b4]), soon to be shared with the community. was (Author: fenglu): [~yiga2] it's on our to-do list, the current focus is to move all GCP operators from contrib to core. I am also working on a GCP operator development [guide|[https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY|https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY/edit?ts=5bb379b4]], soon to be shared with the community. > Rebase and migrate existing Airflow GCP operators to google-python-cloud > > > Key: AIRFLOW-1894 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1894 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib >Affects Versions: 2.0.0 >Reporter: Feng Lu >Assignee: Feng Lu >Priority: Minor > > [google-api-python-client|https://github.com/google/google-api-python-client] > is in maintenance mode and it's recommended that > [google-cloud-python|https://github.com/GoogleCloudPlatform/google-cloud-python] > should be used whenever possible. Given that we don't have feature parity > between the two libraries, this issue is created to track the long-term > migration efforts moving from google-api-python-client to > google-cloud-python. Here are some general guidelines we try to follow in > this cleanup process: > - add google-cloud-python dependency as part of gcp_api extra packages (make > sure there is no dependency conflict between the two). > - new operators shall be based on google-cloud-python if possible. > - migrate existing GCP operators when the underlying GCP service is available > in google-cloud-python. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-1894) Rebase and migrate existing Airflow GCP operators to google-python-cloud
[ https://issues.apache.org/jira/browse/AIRFLOW-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637794#comment-16637794 ] Feng Lu edited comment on AIRFLOW-1894 at 10/4/18 4:16 AM: --- [~yiga2] it's on our to-do list, the current focus is to move all GCP operators from contrib to core. I am also working on a GCP operator development [guide|[https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY|https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY/edit?ts=5bb379b4]], soon to be shared with the community. was (Author: fenglu): [~yiga2] it's on our to-do list, the current focus is to move all GCP operators from contrib to core. I am also working on a GCP operator [development |http://example.com][guide||[https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY/edit?ts=5bb379b4#|https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY/edit?ts=5bb379b4]] soon to be shared with the community. > Rebase and migrate existing Airflow GCP operators to google-python-cloud > > > Key: AIRFLOW-1894 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1894 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib >Affects Versions: 2.0.0 >Reporter: Feng Lu >Assignee: Feng Lu >Priority: Minor > > [google-api-python-client|https://github.com/google/google-api-python-client] > is in maintenance mode and it's recommended that > [google-cloud-python|https://github.com/GoogleCloudPlatform/google-cloud-python] > should be used whenever possible. Given that we don't have feature parity > between the two libraries, this issue is created to track the long-term > migration efforts moving from google-api-python-client to > google-cloud-python. Here are some general guidelines we try to follow in > this cleanup process: > - add google-cloud-python dependency as part of gcp_api extra packages (make > sure there is no dependency conflict between the two). > - new operators shall be based on google-cloud-python if possible. > - migrate existing GCP operators when the underlying GCP service is available > in google-cloud-python. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-1894) Rebase and migrate existing Airflow GCP operators to google-python-cloud
[ https://issues.apache.org/jira/browse/AIRFLOW-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637794#comment-16637794 ] Feng Lu edited comment on AIRFLOW-1894 at 10/4/18 4:15 AM: --- [~yiga2] it's on our to-do list, the current focus is to move all GCP operators from contrib to core. I am also working on a GCP operator [development |http://example.com][guide||[https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY/edit?ts=5bb379b4#|https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY/edit?ts=5bb379b4]] soon to be shared with the community. was (Author: fenglu): [~yiga2] it's on our to-do list, the current focus is to move all GCP operators from contrib to core. I am also working on a GCP operator development [guide||[https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY/edit?ts=5bb379b4#|https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY/edit?ts=5bb379b4]] soon to be shared with the community. > Rebase and migrate existing Airflow GCP operators to google-python-cloud > > > Key: AIRFLOW-1894 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1894 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib >Affects Versions: 2.0.0 >Reporter: Feng Lu >Assignee: Feng Lu >Priority: Minor > > [google-api-python-client|https://github.com/google/google-api-python-client] > is in maintenance mode and it's recommended that > [google-cloud-python|https://github.com/GoogleCloudPlatform/google-cloud-python] > should be used whenever possible. Given that we don't have feature parity > between the two libraries, this issue is created to track the long-term > migration efforts moving from google-api-python-client to > google-cloud-python. Here are some general guidelines we try to follow in > this cleanup process: > - add google-cloud-python dependency as part of gcp_api extra packages (make > sure there is no dependency conflict between the two). > - new operators shall be based on google-cloud-python if possible. > - migrate existing GCP operators when the underlying GCP service is available > in google-cloud-python. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-1894) Rebase and migrate existing Airflow GCP operators to google-python-cloud
[ https://issues.apache.org/jira/browse/AIRFLOW-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637794#comment-16637794 ] Feng Lu edited comment on AIRFLOW-1894 at 10/4/18 4:14 AM: --- [~yiga2] it's on our to-do list, the current focus is to move all GCP operators from contrib to core. I am also working on a GCP operator development [guide||[https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY/edit?ts=5bb379b4#|https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY/edit?ts=5bb379b4]] soon to be shared with the community. was (Author: fenglu): [~yiga2] it's on our to-do list, the current focus is to move all GCP operators from contrib to core. I am also working on a GCP operator development [guide|[https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY/edit?ts=5bb379b4#|https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY/edit?ts=5bb379b4]] soon to be shared with the community. > Rebase and migrate existing Airflow GCP operators to google-python-cloud > > > Key: AIRFLOW-1894 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1894 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib >Affects Versions: 2.0.0 >Reporter: Feng Lu >Assignee: Feng Lu >Priority: Minor > > [google-api-python-client|https://github.com/google/google-api-python-client] > is in maintenance mode and it's recommended that > [google-cloud-python|https://github.com/GoogleCloudPlatform/google-cloud-python] > should be used whenever possible. Given that we don't have feature parity > between the two libraries, this issue is created to track the long-term > migration efforts moving from google-api-python-client to > google-cloud-python. Here are some general guidelines we try to follow in > this cleanup process: > - add google-cloud-python dependency as part of gcp_api extra packages (make > sure there is no dependency conflict between the two). > - new operators shall be based on google-cloud-python if possible. > - migrate existing GCP operators when the underlying GCP service is available > in google-cloud-python. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-1894) Rebase and migrate existing Airflow GCP operators to google-python-cloud
[ https://issues.apache.org/jira/browse/AIRFLOW-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637794#comment-16637794 ] Feng Lu edited comment on AIRFLOW-1894 at 10/4/18 4:14 AM: --- [~yiga2] it's on our to-do list, the current focus is to move all GCP operators from contrib to core. I am also working on a GCP operator development [guide][[https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY/edit?ts=5bb379b4#|https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY/edit?ts=5bb379b4]] soon to be shared with the community. was (Author: fenglu): [~yiga2] it's on our to-do list, the current focus is to move all GCP operators from contrib to core. I am also working on a GCP operator development [guide][[https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY/edit?ts=5bb379b4#|https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY/edit?ts=5bb379b4],] soon to be shared with the community. > Rebase and migrate existing Airflow GCP operators to google-python-cloud > > > Key: AIRFLOW-1894 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1894 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib >Affects Versions: 2.0.0 >Reporter: Feng Lu >Assignee: Feng Lu >Priority: Minor > > [google-api-python-client|https://github.com/google/google-api-python-client] > is in maintenance mode and it's recommended that > [google-cloud-python|https://github.com/GoogleCloudPlatform/google-cloud-python] > should be used whenever possible. Given that we don't have feature parity > between the two libraries, this issue is created to track the long-term > migration efforts moving from google-api-python-client to > google-cloud-python. Here are some general guidelines we try to follow in > this cleanup process: > - add google-cloud-python dependency as part of gcp_api extra packages (make > sure there is no dependency conflict between the two). > - new operators shall be based on google-cloud-python if possible. > - migrate existing GCP operators when the underlying GCP service is available > in google-cloud-python. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-1894) Rebase and migrate existing Airflow GCP operators to google-python-cloud
[ https://issues.apache.org/jira/browse/AIRFLOW-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637794#comment-16637794 ] Feng Lu edited comment on AIRFLOW-1894 at 10/4/18 4:14 AM: --- [~yiga2] it's on our to-do list, the current focus is to move all GCP operators from contrib to core. I am also working on a GCP operator development [guide|[https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY/edit?ts=5bb379b4#|https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY/edit?ts=5bb379b4]] soon to be shared with the community. was (Author: fenglu): [~yiga2] it's on our to-do list, the current focus is to move all GCP operators from contrib to core. I am also working on a GCP operator development [guide][[https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY/edit?ts=5bb379b4#|https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY/edit?ts=5bb379b4]] soon to be shared with the community. > Rebase and migrate existing Airflow GCP operators to google-python-cloud > > > Key: AIRFLOW-1894 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1894 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib >Affects Versions: 2.0.0 >Reporter: Feng Lu >Assignee: Feng Lu >Priority: Minor > > [google-api-python-client|https://github.com/google/google-api-python-client] > is in maintenance mode and it's recommended that > [google-cloud-python|https://github.com/GoogleCloudPlatform/google-cloud-python] > should be used whenever possible. Given that we don't have feature parity > between the two libraries, this issue is created to track the long-term > migration efforts moving from google-api-python-client to > google-cloud-python. Here are some general guidelines we try to follow in > this cleanup process: > - add google-cloud-python dependency as part of gcp_api extra packages (make > sure there is no dependency conflict between the two). > - new operators shall be based on google-cloud-python if possible. > - migrate existing GCP operators when the underlying GCP service is available > in google-cloud-python. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1894) Rebase and migrate existing Airflow GCP operators to google-python-cloud
[ https://issues.apache.org/jira/browse/AIRFLOW-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637794#comment-16637794 ] Feng Lu commented on AIRFLOW-1894: -- [~yiga2] it's on our to-do list, the current focus is to move all GCP operators from contrib to core. I am also working on a GCP operator development [guide|[https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY/edit?ts=5bb379b4#],] soon to be shared with the community. > Rebase and migrate existing Airflow GCP operators to google-python-cloud > > > Key: AIRFLOW-1894 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1894 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib >Affects Versions: 2.0.0 >Reporter: Feng Lu >Assignee: Feng Lu >Priority: Minor > > [google-api-python-client|https://github.com/google/google-api-python-client] > is in maintenance mode and it's recommended that > [google-cloud-python|https://github.com/GoogleCloudPlatform/google-cloud-python] > should be used whenever possible. Given that we don't have feature parity > between the two libraries, this issue is created to track the long-term > migration efforts moving from google-api-python-client to > google-cloud-python. Here are some general guidelines we try to follow in > this cleanup process: > - add google-cloud-python dependency as part of gcp_api extra packages (make > sure there is no dependency conflict between the two). > - new operators shall be based on google-cloud-python if possible. > - migrate existing GCP operators when the underlying GCP service is available > in google-cloud-python. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-1894) Rebase and migrate existing Airflow GCP operators to google-python-cloud
[ https://issues.apache.org/jira/browse/AIRFLOW-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637794#comment-16637794 ] Feng Lu edited comment on AIRFLOW-1894 at 10/4/18 4:13 AM: --- [~yiga2] it's on our to-do list, the current focus is to move all GCP operators from contrib to core. I am also working on a GCP operator development [guide][[https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY/edit?ts=5bb379b4#|https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY/edit?ts=5bb379b4],] soon to be shared with the community. was (Author: fenglu): [~yiga2] it's on our to-do list, the current focus is to move all GCP operators from contrib to core. I am also working on a GCP operator development [guide|[https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY/edit?ts=5bb379b4#],] soon to be shared with the community. > Rebase and migrate existing Airflow GCP operators to google-python-cloud > > > Key: AIRFLOW-1894 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1894 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib >Affects Versions: 2.0.0 >Reporter: Feng Lu >Assignee: Feng Lu >Priority: Minor > > [google-api-python-client|https://github.com/google/google-api-python-client] > is in maintenance mode and it's recommended that > [google-cloud-python|https://github.com/GoogleCloudPlatform/google-cloud-python] > should be used whenever possible. Given that we don't have feature parity > between the two libraries, this issue is created to track the long-term > migration efforts moving from google-api-python-client to > google-cloud-python. Here are some general guidelines we try to follow in > this cleanup process: > - add google-cloud-python dependency as part of gcp_api extra packages (make > sure there is no dependency conflict between the two). > - new operators shall be based on google-cloud-python if possible. > - migrate existing GCP operators when the underlying GCP service is available > in google-cloud-python. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1894) Rebase and migrate existing Airflow GCP operators to google-python-cloud
[ https://issues.apache.org/jira/browse/AIRFLOW-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637753#comment-16637753 ] Yannick Einsweiler commented on AIRFLOW-1894: - Has the migration fallen off the radar ? With composer being GA, I thought the operators would all be based on the Client library by now. We have written our own since then. > Rebase and migrate existing Airflow GCP operators to google-python-cloud > > > Key: AIRFLOW-1894 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1894 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib >Affects Versions: 2.0.0 >Reporter: Feng Lu >Assignee: Feng Lu >Priority: Minor > > [google-api-python-client|https://github.com/google/google-api-python-client] > is in maintenance mode and it's recommended that > [google-cloud-python|https://github.com/GoogleCloudPlatform/google-cloud-python] > should be used whenever possible. Given that we don't have feature parity > between the two libraries, this issue is created to track the long-term > migration efforts moving from google-api-python-client to > google-cloud-python. Here are some general guidelines we try to follow in > this cleanup process: > - add google-cloud-python dependency as part of gcp_api extra packages (make > sure there is no dependency conflict between the two). > - new operators shall be based on google-cloud-python if possible. > - migrate existing GCP operators when the underlying GCP service is available > in google-cloud-python. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] feng-tao commented on issue #3997: [AIRFLOW-3153] send dag last_run to statsd
feng-tao commented on issue #3997: [AIRFLOW-3153] send dag last_run to statsd URL: https://github.com/apache/incubator-airflow/pull/3997#issuecomment-426870875 PTAL @kaxil This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #3991: [WIP] Backfill missing sensor tests.
codecov-io edited a comment on issue #3991: [WIP] Backfill missing sensor tests. URL: https://github.com/apache/incubator-airflow/pull/3991#issuecomment-426509904 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3991?src=pr=h1) Report > Merging [#3991](https://codecov.io/gh/apache/incubator-airflow/pull/3991?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/7103c2aba01a7e891715102636c8ae31f4741172?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3991?src=pr=tree) ```diff @@Coverage Diff @@ ## master#3991 +/- ## == + Coverage 15.22% 15.22% +<.01% == Files 199 199 Lines 1594715945 -2 == Hits 2428 2428 + Misses 1351913517 -2 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3991?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/sensors/sql\_sensor.py](https://codecov.io/gh/apache/incubator-airflow/pull/3991/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL3NxbF9zZW5zb3IucHk=) | `0% <0%> (ø)` | :arrow_up: | | [airflow/sensors/s3\_key\_sensor.py](https://codecov.io/gh/apache/incubator-airflow/pull/3991/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL3MzX2tleV9zZW5zb3IucHk=) | `0% <0%> (ø)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3991?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3991?src=pr=footer). Last update [7103c2a...9b73fa1](https://codecov.io/gh/apache/incubator-airflow/pull/3991?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-2934) Pools not respected for internal subdag tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-2934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Victor Vianna resolved AIRFLOW-2934. Resolution: Fixed Thanks! > Pools not respected for internal subdag tasks > - > > Key: AIRFLOW-2934 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2934 > Project: Apache Airflow > Issue Type: Bug > Components: pools, subdag >Affects Versions: 1.9.0 > Environment: Linux victorvianna 4.14.65-1-MANJARO #1 SMP PREEMPT Sat > Aug 18 13:29:56 UTC 2018 x86_64 GNU/Linux > Python 3.6.6 >Reporter: Victor Vianna >Priority: Blocker > Labels: pool, subdag > Attachments: Screenshot from 2018-08-22 12-32-32.png, dag_pool.py > > > I'm trying to have some subdags execute one task at a time. The way I found > was to create a first pool for the SubdagOperators (pool1 in the attached > code file) and a second one for the internal tasks (pool2). However, it > appears that pools for subdag elements are not being respected. Running > airflow 1.9.0 with LocalExecutor. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3155) Add ability to filter by a last modified time in GoogleCloudStorageToGoogleCloudStorageOperator
Brandon Kvarda created AIRFLOW-3155: --- Summary: Add ability to filter by a last modified time in GoogleCloudStorageToGoogleCloudStorageOperator Key: AIRFLOW-3155 URL: https://issues.apache.org/jira/browse/AIRFLOW-3155 Project: Apache Airflow Issue Type: Improvement Components: gcp Affects Versions: 2.0.0 Reporter: Brandon Kvarda Assignee: Brandon Kvarda Currently the GoogleCloudStorageToGoogleCloudStorageOperator doesn't support filtering objects based on a last modified time/date. This would add the ability to further filter source object(s) to copy/move based on a last modified time threshold (for example, if the objects were updated after the last run at 10:00 yesterday, then copy/move them; otherwise, do not.) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3154) Failed attempt to send SLA miss email blocks scheduling for DAG with miss
Wilson Lian created AIRFLOW-3154: Summary: Failed attempt to send SLA miss email blocks scheduling for DAG with miss Key: AIRFLOW-3154 URL: https://issues.apache.org/jira/browse/AIRFLOW-3154 Project: Apache Airflow Issue Type: Bug Components: scheduler Affects Versions: 1.9.0 Reporter: Wilson Lian Haven't tested for non-Sendgrid email backends, but when [email]email_backend = airflow.contrib.utils.sendgrid.send_email, and a DAG's SLA miss email fails to send (e.g., authorization error), the scheduler stops scheduling tasks for that DAG. Other DAGs still run fine. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] feng-tao commented on issue #3937: [AIRFLOW-3103][AIRFLOW-3147] Update flask-appbuilder
feng-tao commented on issue #3937: [AIRFLOW-3103][AIRFLOW-3147] Update flask-appbuilder URL: https://github.com/apache/incubator-airflow/pull/3937#issuecomment-426836985 @ashb , I think this pr is ready. And it needs to be checked in to unblock the CI. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feng-tao opened a new pull request #3997: [AIRFLOW-3153] send dag last_run to statsd
feng-tao opened a new pull request #3997: [AIRFLOW-3153] send dag last_run to
statsd
URL: https://github.com/apache/incubator-airflow/pull/3997
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3153
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Lyft has been running with this pr for over an year and numerous production
issues have been detected by the stats(e.g setting pageduty on the last run
time if it exceeds for certain threshold).
This PR adds statds logging for the DAG generation in Airflow, recording
the time spent processing each file; and
the last time it was processed (both as a unix timestamp and as an interval
in seconds).
```
Stats.gauge('last_runtime.example_bash_operator', 1.622376)
Stats.gauge('last_run.unixtime.example_bash_operator', 1512670855)
Stats.gauge('last_run.seconds_ago.example_bash_operator', 0.641343)
Stats.gauge('last_runtime.example_bash_operator', 1.629494)
Stats.gauge('last_run.unixtime.example_bash_operator', 1512670886)
Stats.gauge('last_run.seconds_ago.example_bash_operator', 0.526443)
```
Credit to original PR owner(@betodealmeida) at lyft
And fix some flake8 error
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Add stats, no need for test.
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-3153) send dag last_run to statsd
[
https://issues.apache.org/jira/browse/AIRFLOW-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637594#comment-16637594
]
ASF GitHub Bot commented on AIRFLOW-3153:
-
feng-tao opened a new pull request #3997: [AIRFLOW-3153] send dag last_run to
statsd
URL: https://github.com/apache/incubator-airflow/pull/3997
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3153
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Lyft has been running with this pr for over an year and numerous production
issues have been detected by the stats(e.g setting pageduty on the last run
time if it exceeds for certain threshold).
This PR adds statds logging for the DAG generation in Airflow, recording
the time spent processing each file; and
the last time it was processed (both as a unix timestamp and as an interval
in seconds).
```
Stats.gauge('last_runtime.example_bash_operator', 1.622376)
Stats.gauge('last_run.unixtime.example_bash_operator', 1512670855)
Stats.gauge('last_run.seconds_ago.example_bash_operator', 0.641343)
Stats.gauge('last_runtime.example_bash_operator', 1.629494)
Stats.gauge('last_run.unixtime.example_bash_operator', 1512670886)
Stats.gauge('last_run.seconds_ago.example_bash_operator', 0.526443)
```
Credit to original PR owner(@betodealmeida) at lyft
And fix some flake8 error
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Add stats, no need for test.
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> send dag last_run to statsd
> ---
>
> Key: AIRFLOW-3153
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3153
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Tao Feng
>Assignee: Tao Feng
>Priority: Minor
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (AIRFLOW-3004) Add configuration option to disable schedules
[ https://issues.apache.org/jira/browse/AIRFLOW-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacob Greenfield closed AIRFLOW-3004. - Resolution: Implemented > Add configuration option to disable schedules > - > > Key: AIRFLOW-3004 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3004 > Project: Apache Airflow > Issue Type: Improvement > Components: configuration, scheduler >Reporter: Jacob Greenfield >Assignee: Jacob Greenfield >Priority: Minor > > We have a particular use case where we'd like there to be a configuration > option that controls the scheduler and prevents use of the cron schedules > globally for all DAGs, while still allowing manual submission (trigger_dag) > and task instances being scheduled. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3152) Kubernetes Pod Operator should support init containers
[ https://issues.apache.org/jira/browse/AIRFLOW-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriraam AS updated AIRFLOW-3152: Affects Version/s: 1.10.1 > Kubernetes Pod Operator should support init containers > -- > > Key: AIRFLOW-3152 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3152 > Project: Apache Airflow > Issue Type: Sub-task >Affects Versions: 1.10.1 >Reporter: Sriraam AS >Priority: Major > > The pod generator has support for init containers, but the kubernetes pod > operator doesn't support init containers, yet. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3152) Kubernetes Pod Operator should support init containers
[ https://issues.apache.org/jira/browse/AIRFLOW-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriraam AS updated AIRFLOW-3152: Priority: Major (was: Minor) > Kubernetes Pod Operator should support init containers > -- > > Key: AIRFLOW-3152 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3152 > Project: Apache Airflow > Issue Type: Sub-task >Affects Versions: 1.10.1 >Reporter: Sriraam AS >Priority: Major > > The pod generator has support for init containers, but the kubernetes pod > operator doesn't support init containers, yet. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3152) Kubernetes Pod Operator should support init containers
Sriraam AS created AIRFLOW-3152: --- Summary: Kubernetes Pod Operator should support init containers Key: AIRFLOW-3152 URL: https://issues.apache.org/jira/browse/AIRFLOW-3152 Project: Apache Airflow Issue Type: Sub-task Reporter: Sriraam AS The pod generator has support for init containers, but the kubernetes pod operator doesn't support init containers, yet. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-3004) Add configuration option to disable schedules
[ https://issues.apache.org/jira/browse/AIRFLOW-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng reassigned AIRFLOW-3004: - Assignee: Tao Feng (was: Jacob Greenfield) > Add configuration option to disable schedules > - > > Key: AIRFLOW-3004 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3004 > Project: Apache Airflow > Issue Type: Improvement > Components: configuration, scheduler >Reporter: Jacob Greenfield >Assignee: Tao Feng >Priority: Minor > > We have a particular use case where we'd like there to be a configuration > option that controls the scheduler and prevents use of the cron schedules > globally for all DAGs, while still allowing manual submission (trigger_dag) > and task instances being scheduled. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3004) Add configuration option to disable schedules
[ https://issues.apache.org/jira/browse/AIRFLOW-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637562#comment-16637562 ] Tao Feng commented on AIRFLOW-3004: --- PR merged. Please close the ticket. > Add configuration option to disable schedules > - > > Key: AIRFLOW-3004 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3004 > Project: Apache Airflow > Issue Type: Improvement > Components: configuration, scheduler >Reporter: Jacob Greenfield >Assignee: Jacob Greenfield >Priority: Minor > > We have a particular use case where we'd like there to be a configuration > option that controls the scheduler and prevents use of the cron schedules > globally for all DAGs, while still allowing manual submission (trigger_dag) > and task instances being scheduled. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-3004) Add configuration option to disable schedules
[ https://issues.apache.org/jira/browse/AIRFLOW-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng reassigned AIRFLOW-3004: - Assignee: Jacob Greenfield (was: Tao Feng) > Add configuration option to disable schedules > - > > Key: AIRFLOW-3004 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3004 > Project: Apache Airflow > Issue Type: Improvement > Components: configuration, scheduler >Reporter: Jacob Greenfield >Assignee: Jacob Greenfield >Priority: Minor > > We have a particular use case where we'd like there to be a configuration > option that controls the scheduler and prevents use of the cron schedules > globally for all DAGs, while still allowing manual submission (trigger_dag) > and task instances being scheduled. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3004) Add configuration option to disable schedules
[
https://issues.apache.org/jira/browse/AIRFLOW-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637558#comment-16637558
]
ASF GitHub Bot commented on AIRFLOW-3004:
-
feng-tao closed pull request #3899: [AIRFLOW-3004] Add config disabling
scheduler cron
URL: https://github.com/apache/incubator-airflow/pull/3899
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/config_templates/default_airflow.cfg
b/airflow/config_templates/default_airflow.cfg
index 000dd67a13..e71b7fda5d 100644
--- a/airflow/config_templates/default_airflow.cfg
+++ b/airflow/config_templates/default_airflow.cfg
@@ -484,6 +484,10 @@ max_threads = 2
authenticate = False
+# Turn off scheduler use of cron intervals by setting this to False.
+# DAGs submitted manually in the web UI or with trigger_dag will still run.
+use_job_schedule = True
+
[ldap]
# set this to ldaps://:
uri =
diff --git a/airflow/jobs.py b/airflow/jobs.py
index 916ec1f243..78b9baa4f5 100644
--- a/airflow/jobs.py
+++ b/airflow/jobs.py
@@ -785,7 +785,7 @@ def create_dag_run(self, dag, session=None):
for a DAG based on scheduling interval
Returns DagRun if one is scheduled. Otherwise returns None.
"""
-if dag.schedule_interval:
+if dag.schedule_interval and conf.getboolean('scheduler',
'USE_JOB_SCHEDULE'):
active_runs = DagRun.find(
dag_id=dag.dag_id,
state=State.RUNNING,
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add configuration option to disable schedules
> -
>
> Key: AIRFLOW-3004
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3004
> Project: Apache Airflow
> Issue Type: Improvement
> Components: configuration, scheduler
>Reporter: Jacob Greenfield
>Assignee: Jacob Greenfield
>Priority: Minor
>
> We have a particular use case where we'd like there to be a configuration
> option that controls the scheduler and prevents use of the cron schedules
> globally for all DAGs, while still allowing manual submission (trigger_dag)
> and task instances being scheduled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] feng-tao closed pull request #3899: [AIRFLOW-3004] Add config disabling scheduler cron
feng-tao closed pull request #3899: [AIRFLOW-3004] Add config disabling
scheduler cron
URL: https://github.com/apache/incubator-airflow/pull/3899
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/config_templates/default_airflow.cfg
b/airflow/config_templates/default_airflow.cfg
index 000dd67a13..e71b7fda5d 100644
--- a/airflow/config_templates/default_airflow.cfg
+++ b/airflow/config_templates/default_airflow.cfg
@@ -484,6 +484,10 @@ max_threads = 2
authenticate = False
+# Turn off scheduler use of cron intervals by setting this to False.
+# DAGs submitted manually in the web UI or with trigger_dag will still run.
+use_job_schedule = True
+
[ldap]
# set this to ldaps://:
uri =
diff --git a/airflow/jobs.py b/airflow/jobs.py
index 916ec1f243..78b9baa4f5 100644
--- a/airflow/jobs.py
+++ b/airflow/jobs.py
@@ -785,7 +785,7 @@ def create_dag_run(self, dag, session=None):
for a DAG based on scheduling interval
Returns DagRun if one is scheduled. Otherwise returns None.
"""
-if dag.schedule_interval:
+if dag.schedule_interval and conf.getboolean('scheduler',
'USE_JOB_SCHEDULE'):
active_runs = DagRun.find(
dag_id=dag.dag_id,
state=State.RUNNING,
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] feng-tao edited a comment on issue #3899: [AIRFLOW-3004] Add config disabling scheduler cron
feng-tao edited a comment on issue #3899: [AIRFLOW-3004] Add config disabling scheduler cron URL: https://github.com/apache/incubator-airflow/pull/3899#issuecomment-426822017 LGTM. The use case makes sense. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feng-tao commented on issue #3899: [AIRFLOW-3004] Add config disabling scheduler cron
feng-tao commented on issue #3899: [AIRFLOW-3004] Add config disabling scheduler cron URL: https://github.com/apache/incubator-airflow/pull/3899#issuecomment-426822017 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feng-tao edited a comment on issue #3993: [AIRFLOW-1354] Fix pool_link() return value
feng-tao edited a comment on issue #3993: [AIRFLOW-1354] Fix pool_link() return value URL: https://github.com/apache/incubator-airflow/pull/3993#issuecomment-426821189 flake8 fails. And is it an issue for www_rbac as well? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feng-tao commented on issue #3993: [AIRFLOW-1354] Fix pool_link() return value
feng-tao commented on issue #3993: [AIRFLOW-1354] Fix pool_link() return value URL: https://github.com/apache/incubator-airflow/pull/3993#issuecomment-426821189 flake8 fails. And is it another issue for www_rbac as well? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feng-tao commented on issue #3996: [AIRFLOW-3151] Pin FlaskAppBuilder 1.11.1 in setup.py
feng-tao commented on issue #3996: [AIRFLOW-3151] Pin FlaskAppBuilder 1.11.1 in setup.py URL: https://github.com/apache/incubator-airflow/pull/3996#issuecomment-426820837 https://github.com/apache/incubator-airflow/pull/3937 should fix the issue. Besides we need the latest FAB(1.12) as it contains an important security fix. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3151) FlaskAppBuilder 1.12.0 is installed by default and breaks www_rbac UI.
[ https://issues.apache.org/jira/browse/AIRFLOW-3151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637526#comment-16637526 ] ASF GitHub Bot commented on AIRFLOW-3151: - oliviersm199 opened a new pull request #3996: [AIRFLOW-3151] Pin FlaskAppBuilder 1.11.1 in setup.py URL: https://github.com/apache/incubator-airflow/pull/3996 ### Jira My PR addresses the following [AIRFLOW-3151](https://issues.apache.org/jira/browse/AIRFLOW-3151) ### Description The PR pins the version of FlaskAppBuilder to 1.11.1 since the new release 1.12.0 breaks the www_rbac UI. ### Tests Doesn't require additional tests because not adding any code to source, just pinning flaskappbuilder like flask-admin is pinned. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > FlaskAppBuilder 1.12.0 is installed by default and breaks www_rbac UI. > -- > > Key: AIRFLOW-3151 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3151 > Project: Apache Airflow > Issue Type: Bug >Reporter: Olivier Simard-Morissette >Priority: Major > Attachments: Screen Shot 2018-10-03 at 5.11.49 PM.png > > > A new version of FlaskAppBuilder was released 12 hours ago or so. The > setup.py in v1-10-stable pins the version of flask-appbuilder to be greater > than 1.11.1. Version 1.12.0 breaks the new UI (screenshot attached). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] oliviersm199 opened a new pull request #3996: [AIRFLOW-3151] Pin FlaskAppBuilder 1.11.1 in setup.py
oliviersm199 opened a new pull request #3996: [AIRFLOW-3151] Pin FlaskAppBuilder 1.11.1 in setup.py URL: https://github.com/apache/incubator-airflow/pull/3996 ### Jira My PR addresses the following [AIRFLOW-3151](https://issues.apache.org/jira/browse/AIRFLOW-3151) ### Description The PR pins the version of FlaskAppBuilder to 1.11.1 since the new release 1.12.0 breaks the www_rbac UI. ### Tests Doesn't require additional tests because not adding any code to source, just pinning flaskappbuilder like flask-admin is pinned. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (AIRFLOW-3142) Fix deprecated calls to BigQueryBaseCursor.run_query
[ https://issues.apache.org/jira/browse/AIRFLOW-3142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Shao closed AIRFLOW-3142. --- Resolution: Won't Fix > Fix deprecated calls to BigQueryBaseCursor.run_query > > > Key: AIRFLOW-3142 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3142 > Project: Apache Airflow > Issue Type: Bug >Reporter: Jason Shao >Assignee: Jason Shao >Priority: Minor > > BigQueryOperator.execute and BigQueryCursor.execute use `bql` instead of `sql` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] jason-udacity commented on issue #3987: [AIRFLOW-3142] Fix deprecated calls to BigQueryBaseCursor.run_query
jason-udacity commented on issue #3987: [AIRFLOW-3142] Fix deprecated calls to BigQueryBaseCursor.run_query URL: https://github.com/apache/incubator-airflow/pull/3987#issuecomment-426790228 @kaxil makes sense. I wasn't sure if there would be a 1.11/1.10.1 release first. thanks for reviewing. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #3937: [AIRFLOW-3103][AIRFLOW-3147] Update flask-appbuilder
codecov-io edited a comment on issue #3937: [AIRFLOW-3103][AIRFLOW-3147] Update flask-appbuilder URL: https://github.com/apache/incubator-airflow/pull/3937#issuecomment-426722645 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3937?src=pr=h1) Report > Merging [#3937](https://codecov.io/gh/apache/incubator-airflow/pull/3937?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/45ed3cec826073a8316dea83035d7da4a2ff0f2e?src=pr=desc) will **decrease** coverage by `60.26%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3937?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#3937 +/- ## === - Coverage 75.49% 15.22% -60.27% === Files 199 199 Lines 1594715947 === - Hits12039 2428 -9611 - Misses 390813519 +9611 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3937?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/www/utils.py](https://codecov.io/gh/apache/incubator-airflow/pull/3937/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdXRpbHMucHk=) | `0% <ø> (-89.35%)` | :arrow_down: | | [airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3937/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `0% <ø> (-68.86%)` | :arrow_down: | | [airflow/www\_rbac/decorators.py](https://codecov.io/gh/apache/incubator-airflow/pull/3937/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy9kZWNvcmF0b3JzLnB5) | `0% <0%> (-74%)` | :arrow_down: | | [airflow/default\_login.py](https://codecov.io/gh/apache/incubator-airflow/pull/3937/diff?src=pr=tree#diff-YWlyZmxvdy9kZWZhdWx0X2xvZ2luLnB5) | `0% <0%> (-58.98%)` | :arrow_down: | | [airflow/www\_rbac/security.py](https://codecov.io/gh/apache/incubator-airflow/pull/3937/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy9zZWN1cml0eS5weQ==) | `0% <0%> (-91.28%)` | :arrow_down: | | [airflow/www\_rbac/validators.py](https://codecov.io/gh/apache/incubator-airflow/pull/3937/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92YWxpZGF0b3JzLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/utils/log/es\_task\_handler.py](https://codecov.io/gh/apache/incubator-airflow/pull/3937/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZXNfdGFza19oYW5kbGVyLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/example\_dags/subdags/subdag.py](https://codecov.io/gh/apache/incubator-airflow/pull/3937/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3Mvc3ViZGFncy9zdWJkYWcucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/www/validators.py](https://codecov.io/gh/apache/incubator-airflow/pull/3937/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmFsaWRhdG9ycy5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/example\_dags/example\_docker\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/3937/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | ... and [155 more](https://codecov.io/gh/apache/incubator-airflow/pull/3937/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3937?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3937?src=pr=footer). Last update [45ed3ce...7bdea18](https://codecov.io/gh/apache/incubator-airflow/pull/3937?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #3937: [AIRFLOW-3103][AIRFLOW-3147] Update flask-appbuilder
codecov-io edited a comment on issue #3937: [AIRFLOW-3103][AIRFLOW-3147] Update flask-appbuilder URL: https://github.com/apache/incubator-airflow/pull/3937#issuecomment-426722645 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3937?src=pr=h1) Report > Merging [#3937](https://codecov.io/gh/apache/incubator-airflow/pull/3937?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/45ed3cec826073a8316dea83035d7da4a2ff0f2e?src=pr=desc) will **decrease** coverage by `60.26%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3937?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#3937 +/- ## === - Coverage 75.49% 15.22% -60.27% === Files 199 199 Lines 1594715947 === - Hits12039 2428 -9611 - Misses 390813519 +9611 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3937?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/www/utils.py](https://codecov.io/gh/apache/incubator-airflow/pull/3937/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdXRpbHMucHk=) | `0% <ø> (-89.35%)` | :arrow_down: | | [airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3937/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `0% <ø> (-68.86%)` | :arrow_down: | | [airflow/www\_rbac/decorators.py](https://codecov.io/gh/apache/incubator-airflow/pull/3937/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy9kZWNvcmF0b3JzLnB5) | `0% <0%> (-74%)` | :arrow_down: | | [airflow/default\_login.py](https://codecov.io/gh/apache/incubator-airflow/pull/3937/diff?src=pr=tree#diff-YWlyZmxvdy9kZWZhdWx0X2xvZ2luLnB5) | `0% <0%> (-58.98%)` | :arrow_down: | | [airflow/www\_rbac/security.py](https://codecov.io/gh/apache/incubator-airflow/pull/3937/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy9zZWN1cml0eS5weQ==) | `0% <0%> (-91.28%)` | :arrow_down: | | [airflow/www\_rbac/validators.py](https://codecov.io/gh/apache/incubator-airflow/pull/3937/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92YWxpZGF0b3JzLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/utils/log/es\_task\_handler.py](https://codecov.io/gh/apache/incubator-airflow/pull/3937/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZXNfdGFza19oYW5kbGVyLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/example\_dags/subdags/subdag.py](https://codecov.io/gh/apache/incubator-airflow/pull/3937/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3Mvc3ViZGFncy9zdWJkYWcucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/www/validators.py](https://codecov.io/gh/apache/incubator-airflow/pull/3937/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmFsaWRhdG9ycy5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/example\_dags/example\_docker\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/3937/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | ... and [155 more](https://codecov.io/gh/apache/incubator-airflow/pull/3937/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3937?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3937?src=pr=footer). Last update [45ed3ce...7bdea18](https://codecov.io/gh/apache/incubator-airflow/pull/3937?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3142) Fix deprecated calls to BigQueryBaseCursor.run_query
[ https://issues.apache.org/jira/browse/AIRFLOW-3142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637453#comment-16637453 ] ASF GitHub Bot commented on AIRFLOW-3142: - kaxil closed pull request #3987: [AIRFLOW-3142] Fix deprecated calls to BigQueryBaseCursor.run_query URL: https://github.com/apache/incubator-airflow/pull/3987 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/airflow/contrib/hooks/bigquery_hook.py b/airflow/contrib/hooks/bigquery_hook.py index dd77df1283..f861397d9b 100644 --- a/airflow/contrib/hooks/bigquery_hook.py +++ b/airflow/contrib/hooks/bigquery_hook.py @@ -1487,7 +1487,7 @@ def execute(self, operation, parameters=None): """ sql = _bind_parameters(operation, parameters) if parameters else operation -self.job_id = self.run_query(sql) +self.job_id = self.run_query(sql=sql) def executemany(self, operation, seq_of_parameters): """ diff --git a/airflow/contrib/operators/bigquery_operator.py b/airflow/contrib/operators/bigquery_operator.py index 9386e57c07..e1c3d2bae6 100644 --- a/airflow/contrib/operators/bigquery_operator.py +++ b/airflow/contrib/operators/bigquery_operator.py @@ -181,7 +181,7 @@ def execute(self, context): conn = hook.get_conn() self.bq_cursor = conn.cursor() self.bq_cursor.run_query( -self.sql, +sql=self.sql, destination_dataset_table=self.destination_dataset_table, write_disposition=self.write_disposition, allow_large_results=self.allow_large_results, This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Fix deprecated calls to BigQueryBaseCursor.run_query > > > Key: AIRFLOW-3142 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3142 > Project: Apache Airflow > Issue Type: Bug >Reporter: Jason Shao >Assignee: Jason Shao >Priority: Minor > > BigQueryOperator.execute and BigQueryCursor.execute use `bql` instead of `sql` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] kaxil closed pull request #3987: [AIRFLOW-3142] Fix deprecated calls to BigQueryBaseCursor.run_query
kaxil closed pull request #3987: [AIRFLOW-3142] Fix deprecated calls to BigQueryBaseCursor.run_query URL: https://github.com/apache/incubator-airflow/pull/3987 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/airflow/contrib/hooks/bigquery_hook.py b/airflow/contrib/hooks/bigquery_hook.py index dd77df1283..f861397d9b 100644 --- a/airflow/contrib/hooks/bigquery_hook.py +++ b/airflow/contrib/hooks/bigquery_hook.py @@ -1487,7 +1487,7 @@ def execute(self, operation, parameters=None): """ sql = _bind_parameters(operation, parameters) if parameters else operation -self.job_id = self.run_query(sql) +self.job_id = self.run_query(sql=sql) def executemany(self, operation, seq_of_parameters): """ diff --git a/airflow/contrib/operators/bigquery_operator.py b/airflow/contrib/operators/bigquery_operator.py index 9386e57c07..e1c3d2bae6 100644 --- a/airflow/contrib/operators/bigquery_operator.py +++ b/airflow/contrib/operators/bigquery_operator.py @@ -181,7 +181,7 @@ def execute(self, context): conn = hook.get_conn() self.bq_cursor = conn.cursor() self.bq_cursor.run_query( -self.sql, +sql=self.sql, destination_dataset_table=self.destination_dataset_table, write_disposition=self.write_disposition, allow_large_results=self.allow_large_results, This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil commented on issue #3987: [AIRFLOW-3142] Fix deprecated calls to BigQueryBaseCursor.run_query
kaxil commented on issue #3987: [AIRFLOW-3142] Fix deprecated calls to BigQueryBaseCursor.run_query URL: https://github.com/apache/incubator-airflow/pull/3987#issuecomment-426777445 Thanks for this PR but we plan to deprecate it in Airflow 2.0 and it will be a part of major changes going in to GCP related Operators. Hence, I am closing this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-3150) Make execution_date a template field in TriggerDagRunOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kyle Hamlin updated AIRFLOW-3150: - Labels: easy-fix (was: ) > Make execution_date a template field in TriggerDagRunOperator > - > > Key: AIRFLOW-3150 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3150 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Reporter: Kyle Hamlin >Priority: Minor > Labels: easy-fix > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3150) Make execution_date a template field in TriggerDagRunOperator
Kyle Hamlin created AIRFLOW-3150: Summary: Make execution_date a template field in TriggerDagRunOperator Key: AIRFLOW-3150 URL: https://issues.apache.org/jira/browse/AIRFLOW-3150 Project: Apache Airflow Issue Type: Improvement Components: operators Reporter: Kyle Hamlin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] codecov-io commented on issue #3937: [AIRFLOW-3103][AIRFLOW-3147] Update flask-appbuilder
codecov-io commented on issue #3937: [AIRFLOW-3103][AIRFLOW-3147] Update flask-appbuilder URL: https://github.com/apache/incubator-airflow/pull/3937#issuecomment-426722645 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3937?src=pr=h1) Report > Merging [#3937](https://codecov.io/gh/apache/incubator-airflow/pull/3937?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/45ed3cec826073a8316dea83035d7da4a2ff0f2e?src=pr=desc) will **decrease** coverage by `60.26%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3937?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#3937 +/- ## === - Coverage 75.49% 15.22% -60.27% === Files 199 199 Lines 1594715947 === - Hits12039 2428 -9611 - Misses 390813519 +9611 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3937?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/www/utils.py](https://codecov.io/gh/apache/incubator-airflow/pull/3937/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdXRpbHMucHk=) | `0% <ø> (-89.35%)` | :arrow_down: | | [airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3937/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `0% <ø> (-68.86%)` | :arrow_down: | | [airflow/www\_rbac/decorators.py](https://codecov.io/gh/apache/incubator-airflow/pull/3937/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy9kZWNvcmF0b3JzLnB5) | `0% <0%> (-74%)` | :arrow_down: | | [airflow/default\_login.py](https://codecov.io/gh/apache/incubator-airflow/pull/3937/diff?src=pr=tree#diff-YWlyZmxvdy9kZWZhdWx0X2xvZ2luLnB5) | `0% <0%> (-58.98%)` | :arrow_down: | | [airflow/www\_rbac/security.py](https://codecov.io/gh/apache/incubator-airflow/pull/3937/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy9zZWN1cml0eS5weQ==) | `0% <0%> (-91.28%)` | :arrow_down: | | [airflow/www\_rbac/validators.py](https://codecov.io/gh/apache/incubator-airflow/pull/3937/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92YWxpZGF0b3JzLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/utils/log/es\_task\_handler.py](https://codecov.io/gh/apache/incubator-airflow/pull/3937/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZXNfdGFza19oYW5kbGVyLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/example\_dags/subdags/subdag.py](https://codecov.io/gh/apache/incubator-airflow/pull/3937/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3Mvc3ViZGFncy9zdWJkYWcucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/www/validators.py](https://codecov.io/gh/apache/incubator-airflow/pull/3937/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmFsaWRhdG9ycy5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/example\_dags/example\_docker\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/3937/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | ... and [155 more](https://codecov.io/gh/apache/incubator-airflow/pull/3937/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3937?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3937?src=pr=footer). Last update [45ed3ce...3fb7ae1](https://codecov.io/gh/apache/incubator-airflow/pull/3937?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-3149) GCP dataproc cluster creation should have the option to delete an ERROR cluster
Aaron Dossett created AIRFLOW-3149: -- Summary: GCP dataproc cluster creation should have the option to delete an ERROR cluster Key: AIRFLOW-3149 URL: https://issues.apache.org/jira/browse/AIRFLOW-3149 Project: Apache Airflow Issue Type: Improvement Components: gcp Affects Versions: 1.10.0 Reporter: Aaron Dossett Assignee: Aaron Dossett We sometimes encounter issues where a dataproc cluster creation ends up in ERROR state. That is, the cluster “exists” but in the state of ERROR[1] (not just that the cluster creation API call failed). This makes retries impossible since the cluster name already exists subsequent retried creations are guaranteed to fail. A `delete_cluster_on_error` parameter should be added to the `DataprocClusterCreateOperator` operator that controls whether or not an attempt to delete an ERROR cluster is made. [1] - I’ve seen that happen in two ways 1) a purely transient error from GCP `Internal server error` or the like 2) when the request is rejected because it would exceed the project quota. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] feng-tao commented on issue #3937: [AIRFLOW-3103] Update flask-login.
feng-tao commented on issue #3937: [AIRFLOW-3103] Update flask-login. URL: https://github.com/apache/incubator-airflow/pull/3937#issuecomment-426709131 Cool. Great to see this pr :) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb commented on issue #3937: [AIRFLOW-3103] Update flask-login.
ashb commented on issue #3937: [AIRFLOW-3103] Update flask-login.
URL:
https://github.com/apache/incubator-airflow/pull/3937#issuecomment-426708423
You need this too:
```
diff --git a/tests/www_rbac/test_security.py
b/tests/www_rbac/test_security.py
index 67ea5b30..6e0b5726 100644
--- a/tests/www_rbac/test_security.py
+++ b/tests/www_rbac/test_security.py
@@ -109,7 +109,7 @@ class TestSecurity(unittest.TestCase):
def test_get_user_roles(self):
user = mock.MagicMock()
-user.is_anonymous.return_value = False
+user.is_anonymous = False
roles = self.appbuilder.sm.find_role('Admin')
user.roles = roles
self.assertEqual(self.security_manager.get_user_roles(user), roles)
@@ -144,7 +144,7 @@ class TestSecurity(unittest.TestCase):
self.security_manager.init_role(role_name, role_vms, role_perms)
role = self.security_manager.find_role(role_name)
user.roles = [role]
-user.is_anonymous.return_value = False
+user.is_anonymous = False
mock_get_all_permissions_views.return_value = {('can_dag_read',
'dag_id')}
mock_get_user_roles.return_value = [role]
@@ -154,7 +154,7 @@ class TestSecurity(unittest.TestCase):
@mock.patch('airflow.www_rbac.security.AirflowSecurityManager._has_view_access')
def test_has_access(self, mock_has_view_access):
user = mock.MagicMock()
-user.is_anonymous.return_value = False
+user.is_anonymous = False
mock_has_view_access.return_value = True
self.assertTrue(self.security_manager.has_access('perm', 'view',
user))
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ashb commented on a change in pull request #3937: [AIRFLOW-3103] Update flask-login.
ashb commented on a change in pull request #3937: [AIRFLOW-3103] Update
flask-login.
URL: https://github.com/apache/incubator-airflow/pull/3937#discussion_r222380402
##
File path: tests/www/test_utils.py
##
@@ -143,7 +143,7 @@ def some_func():
self.assertEqual(fake_username, kwargs['owner'])
mocked_session_instance.add.assert_called_once()
-@mock.patch("flask_login._get_user")
+@mock.patch("flask_login.utils._get_user")
Review comment:
Thanks -I see what I did wrong and came to fix it to find you'd done it
already :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ashb commented on a change in pull request #3937: [AIRFLOW-3103] Update flask-login.
ashb commented on a change in pull request #3937: [AIRFLOW-3103] Update flask-login. URL: https://github.com/apache/incubator-airflow/pull/3937#discussion_r222380107 ## File path: setup.py ## @@ -298,10 +298,10 @@ def do_setup(): 'croniter>=0.3.17, <0.4', 'dill>=0.2.2, <0.3', 'flask>=0.12.4, <0.13', -'flask-appbuilder>=1.11.1, <2.0.0', +'flask-appbuilder>=1.12, <2.0.0', Review comment: Could you add `[AIRFLOW-3147]` in to your commit message too please? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb edited a comment on issue #3937: [AIRFLOW-3103] Update flask-login.
ashb edited a comment on issue #3937: [AIRFLOW-3103] Update flask-login. URL: https://github.com/apache/incubator-airflow/pull/3937#issuecomment-426670833 @jmcarp Looks like something else needs an update: ``` AttributeError: does not have the attribute '_get_user' ``` Or I can pick this up if you don't have time. Turns out it wasn't that big a change needed. And since you ticked the "allow maintainters to update this branch" I could fix it for you. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb edited a comment on issue #3937: [AIRFLOW-3103] Update flask-login.
ashb edited a comment on issue #3937: [AIRFLOW-3103] Update flask-login. URL: https://github.com/apache/incubator-airflow/pull/3937#issuecomment-426670833 @jmcarp Looks like something else needs an update: ``` AttributeError: does not have the attribute '_get_user' ``` (Or I can pick this up if you don't have time) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3148) Argument "parameters" in RedshiftToS3Transfer is not necessary
[
https://issues.apache.org/jira/browse/AIRFLOW-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637063#comment-16637063
]
ASF GitHub Bot commented on AIRFLOW-3148:
-
XD-DENG opened a new pull request #3995: [AIRFLOW-3148] Remove unnecessary arg
"parameters" in RedshiftToS3Transfer
URL: https://github.com/apache/incubator-airflow/pull/3995
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3148
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
"Parameters" are used to help render the SQL command. But in this operator,
only schema and table are needed. There is no SQL command to render.
By checking the code, we can also find argument "parameters" is never really
used.
(Fix a minor issue in the docstring as well)
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Argument "parameters" in RedshiftToS3Transfer is not necessary
> --
>
> Key: AIRFLOW-3148
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3148
> Project: Apache Airflow
> Issue Type: Improvement
> Components: operators
>Affects Versions: 1.10.0
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Minor
>
> Argument "parameters" in RedshiftToS3Transfer is not necessary.
>
> Parameters are used to help render the SQL command. But in this operator,
> only schema and table are needed. There is no SQL command to render.
>
> Actually by checking the code, we can also realise argument "parameters" is
> never really used.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] XD-DENG opened a new pull request #3995: [AIRFLOW-3148] Remove unnecessary arg "parameters" in RedshiftToS3Transfer
XD-DENG opened a new pull request #3995: [AIRFLOW-3148] Remove unnecessary arg
"parameters" in RedshiftToS3Transfer
URL: https://github.com/apache/incubator-airflow/pull/3995
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3148
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
"Parameters" are used to help render the SQL command. But in this operator,
only schema and table are needed. There is no SQL command to render.
By checking the code, we can also find argument "parameters" is never really
used.
(Fix a minor issue in the docstring as well)
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-3148) Argument "parameters" in RedshiftToS3Transfer is not necessary
Xiaodong DENG created AIRFLOW-3148: -- Summary: Argument "parameters" in RedshiftToS3Transfer is not necessary Key: AIRFLOW-3148 URL: https://issues.apache.org/jira/browse/AIRFLOW-3148 Project: Apache Airflow Issue Type: Improvement Components: operators Affects Versions: 1.10.0 Reporter: Xiaodong DENG Assignee: Xiaodong DENG Argument "parameters" in RedshiftToS3Transfer is not necessary. Parameters are used to help render the SQL command. But in this operator, only schema and table are needed. There is no SQL command to render. Actually by checking the code, we can also realise argument "parameters" is never really used. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ashb commented on issue #3937: [AIRFLOW-3103] Update flask-login.
ashb commented on issue #3937: [AIRFLOW-3103] Update flask-login. URL: https://github.com/apache/incubator-airflow/pull/3937#issuecomment-426670833 @jmcarp Looks like something else needs an update: ``` AttributeError: does not have the attribute '_get_user' ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sethwoodworth commented on issue #3918: [AIRFLOW-3088] Include slack-compatible emoji image.
sethwoodworth commented on issue #3918: [AIRFLOW-3088] Include slack-compatible emoji image. URL: https://github.com/apache/incubator-airflow/pull/3918#issuecomment-426650027 @r39132 Renamed and rebased This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb commented on issue #3937: [AIRFLOW-3103] Update flask-login.
ashb commented on issue #3937: [AIRFLOW-3103] Update flask-login. URL: https://github.com/apache/incubator-airflow/pull/3937#issuecomment-426649250 FAB has been released, am re-running the tests to see if they pass. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-3147) Update Flask-AppBuilder version
Ash Berlin-Taylor created AIRFLOW-3147: -- Summary: Update Flask-AppBuilder version Key: AIRFLOW-3147 URL: https://issues.apache.org/jira/browse/AIRFLOW-3147 Project: Apache Airflow Issue Type: New Feature Reporter: Ash Berlin-Taylor 1.12.0 is out which has some features we want. We should upgrade the version we require -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ashb closed pull request #3990: [AIRFLOW-2951] Update dag_run table end_date when state change
ashb closed pull request #3990: [AIRFLOW-2951] Update dag_run table end_date
when state change
URL: https://github.com/apache/incubator-airflow/pull/3990
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/models.py b/airflow/models.py
index 8fc259d1b5..428923ff9e 100755
--- a/airflow/models.py
+++ b/airflow/models.py
@@ -4845,6 +4845,8 @@ def get_state(self):
def set_state(self, state):
if self._state != state:
self._state = state
+self.end_date = timezone.utcnow() if self._state in
State.finished() else None
+
if self.dag_id is not None:
# FIXME: Due to the scoped_session factor we we don't get a
clean
# session here, so something really weird goes on:
@@ -5068,7 +5070,7 @@ def update_state(self, session=None):
if (not unfinished_tasks and
any(r.state in (State.FAILED, State.UPSTREAM_FAILED) for r
in roots)):
self.log.info('Marking run %s failed', self)
-self.state = State.FAILED
+self.set_state(State.FAILED)
dag.handle_callback(self, success=False, reason='task_failure',
session=session)
@@ -5076,20 +5078,20 @@ def update_state(self, session=None):
elif not unfinished_tasks and all(r.state in (State.SUCCESS,
State.SKIPPED)
for r in roots):
self.log.info('Marking run %s successful', self)
-self.state = State.SUCCESS
+self.set_state(State.SUCCESS)
dag.handle_callback(self, success=True, reason='success',
session=session)
# if *all tasks* are deadlocked, the run failed
elif (unfinished_tasks and none_depends_on_past and
none_task_concurrency and no_dependencies_met):
self.log.info('Deadlock; marking run %s failed', self)
-self.state = State.FAILED
+self.set_state(State.FAILED)
dag.handle_callback(self, success=False,
reason='all_tasks_deadlocked',
session=session)
# finally, if the roots aren't done, the dag is still running
else:
-self.state = State.RUNNING
+self.set_state(State.RUNNING)
# todo: determine we want to use with_for_update to make sure to lock
the run
session.merge(self)
diff --git a/tests/models.py b/tests/models.py
index 60aee3c84f..55fa41bd90 100644
--- a/tests/models.py
+++ b/tests/models.py
@@ -915,6 +915,124 @@ def on_failure_callable(context):
updated_dag_state = dag_run.update_state()
self.assertEqual(State.FAILED, updated_dag_state)
+def test_dagrun_set_state_end_date(self):
+session = settings.Session()
+
+dag = DAG(
+'test_dagrun_set_state_end_date',
+start_date=DEFAULT_DATE,
+default_args={'owner': 'owner1'})
+
+dag.clear()
+
+now = timezone.utcnow()
+dr = dag.create_dagrun(run_id='test_dagrun_set_state_end_date',
+ state=State.RUNNING,
+ execution_date=now,
+ start_date=now)
+
+# Initial end_date should be NULL
+# State.SUCCESS and State.FAILED are all ending state and should set
end_date
+# State.RUNNING set end_date back to NULL
+session.add(dr)
+session.commit()
+self.assertIsNone(dr.end_date)
+
+dr.set_state(State.SUCCESS)
+session.merge(dr)
+session.commit()
+
+dr_database = session.query(DagRun).filter(
+DagRun.run_id == 'test_dagrun_set_state_end_date'
+).one()
+self.assertIsNotNone(dr_database.end_date)
+self.assertEqual(dr.end_date, dr_database.end_date)
+
+dr.set_state(State.RUNNING)
+session.merge(dr)
+session.commit()
+
+dr_database = session.query(DagRun).filter(
+DagRun.run_id == 'test_dagrun_set_state_end_date'
+).one()
+
+self.assertIsNone(dr_database.end_date)
+
+dr.set_state(State.FAILED)
+session.merge(dr)
+session.commit()
+dr_database = session.query(DagRun).filter(
+DagRun.run_id == 'test_dagrun_set_state_end_date'
+).one()
+
+self.assertIsNotNone(dr_database.end_date)
+self.assertEqual(dr.end_date, dr_database.end_date)
+
+def test_dagrun_update_state_end_date(self):
+session = settings.Session()
+
+dag = DAG(
+'test_dagrun_update_state_end_date',
+
[jira] [Commented] (AIRFLOW-2951) dag_run end_date Null after a dag is finished
[
https://issues.apache.org/jira/browse/AIRFLOW-2951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636958#comment-16636958
]
ASF GitHub Bot commented on AIRFLOW-2951:
-
ashb closed pull request #3990: [AIRFLOW-2951] Update dag_run table end_date
when state change
URL: https://github.com/apache/incubator-airflow/pull/3990
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/models.py b/airflow/models.py
index 8fc259d1b5..428923ff9e 100755
--- a/airflow/models.py
+++ b/airflow/models.py
@@ -4845,6 +4845,8 @@ def get_state(self):
def set_state(self, state):
if self._state != state:
self._state = state
+self.end_date = timezone.utcnow() if self._state in
State.finished() else None
+
if self.dag_id is not None:
# FIXME: Due to the scoped_session factor we we don't get a
clean
# session here, so something really weird goes on:
@@ -5068,7 +5070,7 @@ def update_state(self, session=None):
if (not unfinished_tasks and
any(r.state in (State.FAILED, State.UPSTREAM_FAILED) for r
in roots)):
self.log.info('Marking run %s failed', self)
-self.state = State.FAILED
+self.set_state(State.FAILED)
dag.handle_callback(self, success=False, reason='task_failure',
session=session)
@@ -5076,20 +5078,20 @@ def update_state(self, session=None):
elif not unfinished_tasks and all(r.state in (State.SUCCESS,
State.SKIPPED)
for r in roots):
self.log.info('Marking run %s successful', self)
-self.state = State.SUCCESS
+self.set_state(State.SUCCESS)
dag.handle_callback(self, success=True, reason='success',
session=session)
# if *all tasks* are deadlocked, the run failed
elif (unfinished_tasks and none_depends_on_past and
none_task_concurrency and no_dependencies_met):
self.log.info('Deadlock; marking run %s failed', self)
-self.state = State.FAILED
+self.set_state(State.FAILED)
dag.handle_callback(self, success=False,
reason='all_tasks_deadlocked',
session=session)
# finally, if the roots aren't done, the dag is still running
else:
-self.state = State.RUNNING
+self.set_state(State.RUNNING)
# todo: determine we want to use with_for_update to make sure to lock
the run
session.merge(self)
diff --git a/tests/models.py b/tests/models.py
index 60aee3c84f..55fa41bd90 100644
--- a/tests/models.py
+++ b/tests/models.py
@@ -915,6 +915,124 @@ def on_failure_callable(context):
updated_dag_state = dag_run.update_state()
self.assertEqual(State.FAILED, updated_dag_state)
+def test_dagrun_set_state_end_date(self):
+session = settings.Session()
+
+dag = DAG(
+'test_dagrun_set_state_end_date',
+start_date=DEFAULT_DATE,
+default_args={'owner': 'owner1'})
+
+dag.clear()
+
+now = timezone.utcnow()
+dr = dag.create_dagrun(run_id='test_dagrun_set_state_end_date',
+ state=State.RUNNING,
+ execution_date=now,
+ start_date=now)
+
+# Initial end_date should be NULL
+# State.SUCCESS and State.FAILED are all ending state and should set
end_date
+# State.RUNNING set end_date back to NULL
+session.add(dr)
+session.commit()
+self.assertIsNone(dr.end_date)
+
+dr.set_state(State.SUCCESS)
+session.merge(dr)
+session.commit()
+
+dr_database = session.query(DagRun).filter(
+DagRun.run_id == 'test_dagrun_set_state_end_date'
+).one()
+self.assertIsNotNone(dr_database.end_date)
+self.assertEqual(dr.end_date, dr_database.end_date)
+
+dr.set_state(State.RUNNING)
+session.merge(dr)
+session.commit()
+
+dr_database = session.query(DagRun).filter(
+DagRun.run_id == 'test_dagrun_set_state_end_date'
+).one()
+
+self.assertIsNone(dr_database.end_date)
+
+dr.set_state(State.FAILED)
+session.merge(dr)
+session.commit()
+dr_database = session.query(DagRun).filter(
+DagRun.run_id == 'test_dagrun_set_state_end_date'
+).one()
+
+self.assertIsNotNone(dr_database.end_date)
+
[GitHub] ron819 commented on issue #2799: airflow uses OAuth Provider keycloak
ron819 commented on issue #2799: airflow uses OAuth Provider keycloak URL: https://github.com/apache/incubator-airflow/pull/2799#issuecomment-426630882 @fisher-monkey @Fokko is there any progress with this? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ron819 commented on issue #2233: [AIRFLOW-1098] Fix issue in setting parent_dag when loading dags
ron819 commented on issue #2233: [AIRFLOW-1098] Fix issue in setting parent_dag when loading dags URL: https://github.com/apache/incubator-airflow/pull/2233#issuecomment-426622739 @xiaomj is this still relevant? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3071) Unable to clear Val of Variable from the UI
[
https://issues.apache.org/jira/browse/AIRFLOW-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636898#comment-16636898
]
ASF GitHub Bot commented on AIRFLOW-3071:
-
ron819 closed pull request #3948: [AIRFLOW-3071] clear Val of Variable from the
UI
URL: https://github.com/apache/incubator-airflow/pull/3948
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/www/views.py b/airflow/www/views.py
index d9078caa39..0cd17b0623 100644
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -2022,11 +2022,10 @@ def variables(self, form):
try:
if request.method == 'POST':
data = request.json
-if data:
-with create_session() as session:
-var = models.Variable(key=form, val=json.dumps(data))
-session.add(var)
-session.commit()
+with create_session() as session:
+var = models.Variable(key=form, val=json.dumps(data))
+session.add(var)
+session.commit()
return ""
else:
return self.render(
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Unable to clear Val of Variable from the UI
> ---
>
> Key: AIRFLOW-3071
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3071
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.10.0
>Reporter: jack
>Assignee: jack
>Priority: Minor
> Labels: easyfix
>
> This is quite annoying bug.
>
> Reproduce steps:
> # Create a Variable.
> # Give the Variable a Val & save it.
> # Click edit Variable. You will see the Key with Red {color:#FF}*{color}
> and the value that you entered.
> # Remove the Val (leave the field blank) and click save.
> # No errors will appear. However if you will re-enter to the Variable you
> will see that the Blank value was not saved.
>
> Please allow to remove Val. This is also the intend behavior because the Val
> has no {color:#FF}*{color} near it.
> The current work around is to delete the Variable and re-create it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] ron819 closed pull request #3948: [AIRFLOW-3071] clear Val of Variable from the UI
ron819 closed pull request #3948: [AIRFLOW-3071] clear Val of Variable from the UI URL: https://github.com/apache/incubator-airflow/pull/3948 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/airflow/www/views.py b/airflow/www/views.py index d9078caa39..0cd17b0623 100644 --- a/airflow/www/views.py +++ b/airflow/www/views.py @@ -2022,11 +2022,10 @@ def variables(self, form): try: if request.method == 'POST': data = request.json -if data: -with create_session() as session: -var = models.Variable(key=form, val=json.dumps(data)) -session.add(var) -session.commit() +with create_session() as session: +var = models.Variable(key=form, val=json.dumps(data)) +session.add(var) +session.commit() return "" else: return self.render( This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] phani8996 commented on a change in pull request #3989: [AIRFLOW-1945] Autoscale celery workers for airflow added
phani8996 commented on a change in pull request #3989: [AIRFLOW-1945] Autoscale celery workers for airflow added URL: https://github.com/apache/incubator-airflow/pull/3989#discussion_r79356 ## File path: airflow/config_templates/default_airflow.cfg ## @@ -349,6 +349,14 @@ celery_app_name = airflow.executors.celery_executor # your worker box and the nature of your tasks worker_concurrency = 16 +# The minimum and maximum concurrency that will be used when starting workers with the +# "airflow worker" command. This defines the number of task instances that +# a worker will take, minimum numbers are created in beginning and scaled up and down +# based on number of queued tasks. pick these numbers based on resources on +# worker box and the nature of the task. If autoscale option is available worker_concurrency +# will be ignored +worker_autoscale = 12,16 Review comment: I agree on not using default values but we don't have a provision to use shell functions here so that during execution we can get number of cores and all. What can be done in these scenarios. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] simonvanderveldt edited a comment on issue #3504: [AIRFLOW-2310]: Add AWS Glue Job Compatibility to Airflow
simonvanderveldt edited a comment on issue #3504: [AIRFLOW-2310]: Add AWS Glue Job Compatibility to Airflow URL: https://github.com/apache/incubator-airflow/pull/3504#issuecomment-426541089 @OElesin Have you been able to make any progress on this? There's one duplicate file in there `airflow/contrib/aws_glue_job_hook.py` vs `airflow/contrib/hook/aws_glue_job_hook.py`. [edit] From looking at the PR it seems the code as is will both create a job (using boto3's `create_job()`) as well as run the job (using boto3's `start_job_run()`). IMHO at the very least these should be separate things so one can run jobs without creating them. To expand on this, we create the jobs using a normal development workflow/CI and we just want to be able to run them from Airflow. IMHO it's not Airflow's responsibility to create resources, it should only invoke them. FYI All the unrelated changes are because you merged `master` into your branch instead of rebasing your branch on top of `master` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] simonvanderveldt edited a comment on issue #3504: [AIRFLOW-2310]: Add AWS Glue Job Compatibility to Airflow
simonvanderveldt edited a comment on issue #3504: [AIRFLOW-2310]: Add AWS Glue Job Compatibility to Airflow URL: https://github.com/apache/incubator-airflow/pull/3504#issuecomment-426541089 @OElesin Have you been able to make any progress on this? There's one duplicate file in there `airflow/contrib/aws_glue_job_hook.py` vs `airflow/contrib/hook/aws_glue_job_hook.py`. [edit] From looking at the PR it seems it will both create a job (using boto3's `create_job()`) as well as run the job (using boto3's `start_job_run()`). IMHO at the very least these should be separate things so one can run jobs without creating them. To expand on this, we create the jobs using a normal development workflow/CI and we just want to be able to run them from Airflow. IMHO it's not Airflow's responsibility to create resources, it should only invoke them. FYI All the unrelated changes are because you merged `master` into your branch instead of rebasing your branch on top of `master` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] simonvanderveldt edited a comment on issue #3504: [AIRFLOW-2310]: Add AWS Glue Job Compatibility to Airflow
simonvanderveldt edited a comment on issue #3504: [AIRFLOW-2310]: Add AWS Glue Job Compatibility to Airflow URL: https://github.com/apache/incubator-airflow/pull/3504#issuecomment-426541089 @OElesin Have you been able to make any progress on this? There's one duplicate file in there `airflow/contrib/aws_glue_job_hook.py` vs `airflow/contrib/hook/aws_glue_job_hook.py`. [edit] From looking at the PR it seems it will both create a job (using boto3's `create_job()`) as well as run the job (using boto3's `start_job_run()`). IMHO at the very least these should be separate things so one can run jobs without creating them. To expand on this, we create the jobs using a normal development workflow/CI and we just want to be able to run them from Airflow. IMHO it's not Airflow's responsibility to create resources, it should only invoke them. FYI All the unrelated changes are because you merged `master` into your branch instead of rebasing your branch on top of `master` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] simonvanderveldt edited a comment on issue #3504: [AIRFLOW-2310]: Add AWS Glue Job Compatibility to Airflow
simonvanderveldt edited a comment on issue #3504: [AIRFLOW-2310]: Add AWS Glue Job Compatibility to Airflow URL: https://github.com/apache/incubator-airflow/pull/3504#issuecomment-426541089 @OElesin Have you been able to make any progress on this? There's one duplicate file in there `airflow/contrib/aws_glue_job_hook.py` vs `airflow/contrib/hook/aws_glue_job_hook.py`. [edit] From looking at the PR it seems it will both create a job (using boto3's `create_job()`) as well as run the job (using boto3's `start_job_run()`). IMHO at the very least these should be separate things so one can run jobs without creating them. To expand on this, we create the jobs using a normal development workflow/CI and we just want to be able to run them from Airflow. FYI All the unrelated changes are because you merged `master` into your branch instead of rebasing your branch on top of `master` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-3146) TimeDeltaSensor - consider also end time stamp
jack created AIRFLOW-3146:
-
Summary: TimeDeltaSensor - consider also end time stamp
Key: AIRFLOW-3146
URL: https://issues.apache.org/jira/browse/AIRFLOW-3146
Project: Apache Airflow
Issue Type: Wish
Reporter: jack
Currently the TimeDeltaSensor takes these arguments:
{color:#4070a0} :param delta: time length to wait after execution_date
before succeeding{color}{color:#404040}
{color}{color:#4070a0} :type delta: datetime.timedelta{color}{color:#404040}
{color}{color:#4070a0} """{color}{color:#404040}
{color}{color:#55}@apply_defaults{color}{color:#404040}
{color}{color:#007020}def{color}
{color:#06287e}__init__{color}{color:#404040}({color}{color:#007020}self{color}{color:#404040},{color}
{color:#404040}delta{color}{color:#404040},{color}
{color:#66}*{color}{color:#404040}args{color}{color:#404040},{color}
{color:#66}**{color}{color:#404040}kwargs{color}{color:#404040}):{color}{color:#404040}
{color}{color:#007020}super{color}{color:#404040}({color}{color:#404040}TimeDeltaSensor{color}{color:#404040},{color}
{color:#007020}self{color}{color:#404040}){color}{color:#66}.{color}{color:#06287e}__init__{color}{color:#404040}({color}{color:#66}*{color}{color:#404040}args{color}{color:#404040},{color}
{color:#66}**{color}{color:#404040}kwargs{color}{color:#404040}){color}{color:#404040}
{color}{color:#007020}self{color}{color:#66}.{color}{color:#404040}delta{color}
{color:#66}={color} {color:#404040}delta{color}
*The problem:*
The operator assumes that the user would like to wait execution_date + delta.
This isn't always the case, The user might want to wait delta after the task
has finished running.
Lets take this example:
{color:#4070a0} 2016-01-01 can only start running on 2016-01-02. The
timedelta here{color}{color:#404040}
{color}{color:#4070a0} represents the time after the execution period has
closed.{color}
Now, say the task ran for +4 hours.+ So if we set
{code:java}
delta = 5 minutes {code}
it actually doesn't wait at all. The sensor is useless in that senario. What
the user may be intended is :
execution_date + duration to complete task + delta.
Or in simple words: Timestamp of execution end + delta.
*My suggestion:*
Add another Boolean parameter which will choose if the delta is from
execution_date or execution_date + duration of task
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] codecov-io commented on issue #3993: [AIRFLOW-1354] Fix pool_link() return value
codecov-io commented on issue #3993: [AIRFLOW-1354] Fix pool_link() return value URL: https://github.com/apache/incubator-airflow/pull/3993#issuecomment-426569292 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3993?src=pr=h1) Report > Merging [#3993](https://codecov.io/gh/apache/incubator-airflow/pull/3993?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/45ed3cec826073a8316dea83035d7da4a2ff0f2e?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3993?src=pr=tree) ```diff @@Coverage Diff @@ ## master#3993 +/- ## == - Coverage 75.49% 75.48% -0.01% == Files 199 199 Lines 1594715947 == - Hits1203912038 -1 - Misses 3908 3909 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3993?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3993/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `68.85% <100%> (ø)` | :arrow_up: | | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/3993/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `91.62% <0%> (-0.05%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3993?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3993?src=pr=footer). Last update [45ed3ce...eb5bfa3](https://codecov.io/gh/apache/incubator-airflow/pull/3993?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #3987: [AIRFLOW-3142] Fix deprecated calls to BigQueryBaseCursor.run_query
codecov-io commented on issue #3987: [AIRFLOW-3142] Fix deprecated calls to BigQueryBaseCursor.run_query URL: https://github.com/apache/incubator-airflow/pull/3987#issuecomment-426568474 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3987?src=pr=h1) Report > Merging [#3987](https://codecov.io/gh/apache/incubator-airflow/pull/3987?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/45ed3cec826073a8316dea83035d7da4a2ff0f2e?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3987?src=pr=tree) ```diff @@Coverage Diff @@ ## master#3987 +/- ## == - Coverage 75.49% 75.48% -0.01% == Files 199 199 Lines 1594715947 == - Hits1203912038 -1 - Misses 3908 3909 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3987?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/3987/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `91.62% <0%> (-0.05%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3987?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3987?src=pr=footer). Last update [45ed3ce...e5bd2f1](https://codecov.io/gh/apache/incubator-airflow/pull/3987?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] KevinYang21 commented on issue #3990: [AIRFLOW-2951] Update dag_run table end_date when state change
KevinYang21 commented on issue #3990: [AIRFLOW-2951] Update dag_run table end_date when state change URL: https://github.com/apache/incubator-airflow/pull/3990#issuecomment-426568248 :+1: LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-2800) Remove airflow/ low-hanging linting errors
[ https://issues.apache.org/jira/browse/AIRFLOW-2800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-2800: Fix Version/s: (was: 2.0.0) 1.10.1 > Remove airflow/ low-hanging linting errors > -- > > Key: AIRFLOW-2800 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2800 > Project: Apache Airflow > Issue Type: Bug >Reporter: Andy Cooper >Assignee: Andy Cooper >Priority: Major > Fix For: 1.10.1 > > > Removing low hanging linting errors from airflow directory > Focuses on > * E226 > * W291 > as well as *some* E501 (line too long) where it did not risk reducing > readability -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-1104) Concurrency check in scheduler should count queued tasks as well as running
[ https://issues.apache.org/jira/browse/AIRFLOW-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-1104: Fix Version/s: (was: 2.0.0) 1.10.1 > Concurrency check in scheduler should count queued tasks as well as running > --- > > Key: AIRFLOW-1104 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1104 > Project: Apache Airflow > Issue Type: Bug > Environment: see https://github.com/apache/incubator-airflow/pull/2221 > "Tasks with the QUEUED state should also be counted below, but for now we > cannot count them. This is because there is no guarantee that queued tasks in > failed dagruns will or will not eventually run and queued tasks that will > never run will consume slots and can stall a DAG. Once we can guarantee that > all queued tasks in failed dagruns will never run (e.g. make sure that all > running/newly queued TIs have running dagruns), then we can include QUEUED > tasks here, with the constraint that they are in running dagruns." >Reporter: Alex Guziel >Priority: Minor > Fix For: 1.10.1 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3144) Validate kerberos keytab on startup
[
https://issues.apache.org/jira/browse/AIRFLOW-3144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636630#comment-16636630
]
Ash Berlin-Taylor commented on AIRFLOW-3144:
Sounds useful - is there an function in the kerberos library we are using that
we could use instead of shelling out?
> Validate kerberos keytab on startup
> ---
>
> Key: AIRFLOW-3144
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3144
> Project: Apache Airflow
> Issue Type: Improvement
> Components: authentication
>Reporter: Kris Wilson
>Priority: Minor
>
> at Twitter, we recently ran into an issue where an Airflow user was passing
> the wrong secrets file as their kerberos service principal keytab. Airflow
> happily accepted this file (which contained plain old ascii text) as a keytab
> and then broke at runtime with the following opaque log message:
>
> {code:java}
> [2018-10-01 23:45:14,976] ERROR in kerberos_ldap: Kerberos initialization
> error for HTTP@$REDACTED: ('Cannot get sequence cursor from keytab', 2){code}
>
> this made the problem unclear. rather than blindly accept any old file as a
> keytab, it would be awesome if Airflow could run a validation step against
> the file to confirm it's validity on startup by shelling out to either
> `klist` or `kutil` (or using some equivalent lib).
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2674) BashOperator eats stdout and stderr logs
[ https://issues.apache.org/jira/browse/AIRFLOW-2674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor updated AIRFLOW-2674: --- Fix Version/s: 1.10.1 Issue Type: Bug (was: Improvement) > BashOperator eats stdout and stderr logs > > > Key: AIRFLOW-2674 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2674 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Reporter: Tim Swast >Priority: Minor > Fix For: 1.10.1 > > > I've noticed that when I use the BashOperator, I do not see output from the > bash processes in the task logs or even my machine's logs. This makes it > difficult-to-impossible debug problems in a BashOperator task. > See [related StackOverflow question "Airflow BashOperator log doesn't contain > full output"|https://stackoverflow.com/q/43400302/101923]. > Possibly related to https://issues.apache.org/jira/browse/AIRFLOW-1733 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ashb commented on issue #3990: [AIRFLOW-2951] Update dag_run table end_date when state change
ashb commented on issue #3990: [AIRFLOW-2951] Update dag_run table end_date when state change URL: https://github.com/apache/incubator-airflow/pull/3990#issuecomment-426560565 Thanks @YingboWang This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb commented on a change in pull request #3994: [AIRFLOW-3136] Add retry_number to TaskInstance Key property to avoid race condition
ashb commented on a change in pull request #3994: [AIRFLOW-3136] Add retry_number to TaskInstance Key property to avoid race condition URL: https://github.com/apache/incubator-airflow/pull/3994#discussion_r31277 ## File path: airflow/models.py ## @@ -1230,7 +1230,7 @@ def key(self): """ Returns a tuple that identifies the task instance uniquely """ -return self.dag_id, self.task_id, self.execution_date +return self.dag_id, self.task_id, self.execution_date, self.try_number Review comment: Right now the PK columns of the TI table is (self.dag_id, self.task_id, self.execution_date) and try_number is adjusted in place. This change might be a good idea but I think it is more complex than just this -- for instance it probably makes sense to store each individual try in the task instance table, but that is a much bigger change. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb commented on a change in pull request #3989: [AIRFLOW-1945] Autoscale celery workers for airflow added
ashb commented on a change in pull request #3989: [AIRFLOW-1945] Autoscale celery workers for airflow added URL: https://github.com/apache/incubator-airflow/pull/3989#discussion_r29910 ## File path: airflow/config_templates/default_airflow.cfg ## @@ -349,6 +349,14 @@ celery_app_name = airflow.executors.celery_executor # your worker box and the nature of your tasks worker_concurrency = 16 +# The minimum and maximum concurrency that will be used when starting workers with the +# "airflow worker" command. This defines the number of task instances that +# a worker will take, minimum numbers are created in beginning and scaled up and down +# based on number of queued tasks. pick these numbers based on resources on +# worker box and the nature of the task. If autoscale option is available worker_concurrency +# will be ignored +worker_autoscale = 12,16 Review comment: A link to the section of the celery docs about this in the comment would help too. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb commented on a change in pull request #3989: [AIRFLOW-1945] Autoscale celery workers for airflow added
ashb commented on a change in pull request #3989: [AIRFLOW-1945] Autoscale celery workers for airflow added URL: https://github.com/apache/incubator-airflow/pull/3989#discussion_r29674 ## File path: airflow/config_templates/default_airflow.cfg ## @@ -349,6 +349,14 @@ celery_app_name = airflow.executors.celery_executor # your worker box and the nature of your tasks worker_concurrency = 16 +# The minimum and maximum concurrency that will be used when starting workers with the +# "airflow worker" command. This defines the number of task instances that +# a worker will take, minimum numbers are created in beginning and scaled up and down +# based on number of queued tasks. pick these numbers based on resources on +# worker box and the nature of the task. If autoscale option is available worker_concurrency +# will be ignored +worker_autoscale = 12,16 Review comment: We probably shouldn't set a default value for this and worker_concurrency. Please comment this one out. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-2844) Airflow Logs BrokenPipeException
[
https://issues.apache.org/jira/browse/AIRFLOW-2844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636603#comment-16636603
]
jack commented on AIRFLOW-2844:
---
if print("something") causes issues and Airflow recommended method is to use
logger. I would recommend to highlight it in the documentation and maybe
provide a warning in the UI when it detects print("something") asking the user
to change it to logger.
> Airflow Logs BrokenPipeException
>
>
> Key: AIRFLOW-2844
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2844
> Project: Apache Airflow
> Issue Type: Bug
> Components: logging
>Affects Versions: 1.9.0
>Reporter: Kyle Bridenstine
>Priority: Critical
>
> I'm using a clustered Airflow environment where I have four AWS ec2-instances
> for the servers.
> *ec2-instances*
> - Server 1: Webserver, Scheduler, Redis Queue, PostgreSQL Database
> - Server 2: Webserver
> - Server 3: Worker
> - Server 4: Worker
> My setup has been working perfectly fine for three months now but
> sporadically about once a week I get a Broken Pipe Exception when Airflow is
> attempting to log something.
> {code:java}
> Log file isn't local.
> Fetching here:
> http://ip-1-2-3-4:8793/log/foobar/task_1/2018-07-13T00:00:00/1.log
> [2018-07-16 00:00:15,521] {cli.py:374} INFO - Running on host ip-1-2-3-4
> [2018-07-16 00:00:15,698] {models.py:1197} INFO - Dependencies all met for
>
> [2018-07-16 00:00:15,710] {models.py:1197} INFO - Dependencies all met for
>
> [2018-07-16 00:00:15,710] {models.py:1407} INFO -
>
> Starting attempt 1 of 1
>
> [2018-07-16 00:00:15,719] {models.py:1428} INFO - Executing
> on 2018-07-13 00:00:00
> [2018-07-16 00:00:15,720] {base_task_runner.py:115} INFO - Running: ['bash',
> '-c', 'airflow run foobar task_1 2018-07-13T00:00:00 --job_id 1320 --raw -sd
> DAGS_FOLDER/datalake_digitalplatform_arl_workflow_schedule_test_2.py']
> [2018-07-16 00:00:16,532] {base_task_runner.py:98} INFO - Subtask:
> [2018-07-16 00:00:16,532] {configuration.py:206} WARNING - section/key
> [celery/celery_ssl_active] not found in config
> [2018-07-16 00:00:16,532] {base_task_runner.py:98} INFO - Subtask:
> [2018-07-16 00:00:16,532] {default_celery.py:41} WARNING - Celery Executor
> will run without SSL
> [2018-07-16 00:00:16,534] {base_task_runner.py:98} INFO - Subtask:
> [2018-07-16 00:00:16,533] {_init_.py:45} INFO - Using executor CeleryExecutor
> [2018-07-16 00:00:16,597] {base_task_runner.py:98} INFO - Subtask:
> [2018-07-16 00:00:16,597] {models.py:189} INFO - Filling up the DagBag from
> /home/ec2-user/airflow/dags/datalake_digitalplatform_arl_workflow_schedule_test_2.py
> [2018-07-16 00:00:16,768] {cli.py:374} INFO - Running on host ip-1-2-3-4
> [2018-07-16 00:16:24,931] {logging_mixin.py:84} WARNING - — Logging error —
> [2018-07-16 00:16:24,931] {logging_mixin.py:84} WARNING - Traceback (most
> recent call last):
> [2018-07-16 00:16:24,931] {logging_mixin.py:84} WARNING - File
> "/usr/lib64/python3.6/logging/_init_.py", line 996, in emit
> self.flush()
> [2018-07-16 00:16:24,932] {logging_mixin.py:84} WARNING - File
> "/usr/lib64/python3.6/logging/_init_.py", line 976, in flush
> self.stream.flush()
> [2018-07-16 00:16:24,932] {logging_mixin.py:84} WARNING - BrokenPipeError:
> [Errno 32] Broken pipe
> [2018-07-16 00:16:24,932] {logging_mixin.py:84} WARNING - Call stack:
> [2018-07-16 00:16:24,933] {logging_mixin.py:84} WARNING - File
> "/usr/bin/airflow", line 27, in
> args.func(args)
> [2018-07-16 00:16:24,934] {logging_mixin.py:84} WARNING - File
> "/usr/local/lib/python3.6/site-packages/airflow/bin/cli.py", line 392, in run
> pool=args.pool,
> [2018-07-16 00:16:24,934] {logging_mixin.py:84} WARNING - File
> "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 50, in
> wrapper
> result = func(*args, **kwargs)
> [2018-07-16 00:16:24,934] {logging_mixin.py:84} WARNING - File
> "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 1488, in
> _run_raw_task
> result = task_copy.execute(context=context)
> [2018-07-16 00:16:24,934] {logging_mixin.py:84} WARNING - File
> "/usr/local/lib/python3.6/site-packages/airflow/operators/sensors.py", line
> 78, in execute
> while not self.poke(context):
> [2018-07-16 00:16:24,934] {logging_mixin.py:84} WARNING - File
> "/home/ec2-user/airflow/plugins/custom_plugins.py", line 35, in poke
> directory = os.listdir(full_path)
> [2018-07-16 00:16:24,934] {logging_mixin.py:84} WARNING - File
> "/usr/local/lib/python3.6/site-packages/airflow/utils/timeout.py", line 36,
> in handle_timeout
> self.log.error("Process timed out")
> [2018-07-16 00:16:24,934] {logging_mixin.py:84}
[jira] [Commented] (AIRFLOW-2674) BashOperator eats stdout and stderr logs
[ https://issues.apache.org/jira/browse/AIRFLOW-2674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636592#comment-16636592 ] jack commented on AIRFLOW-2674: --- I think it should be tagged as BUG not as improvement. > BashOperator eats stdout and stderr logs > > > Key: AIRFLOW-2674 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2674 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Reporter: Tim Swast >Priority: Minor > > I've noticed that when I use the BashOperator, I do not see output from the > bash processes in the task logs or even my machine's logs. This makes it > difficult-to-impossible debug problems in a BashOperator task. > See [related StackOverflow question "Airflow BashOperator log doesn't contain > full output"|https://stackoverflow.com/q/43400302/101923]. > Possibly related to https://issues.apache.org/jira/browse/AIRFLOW-1733 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-1354) www/views.py pool_link produces wrong link
[
https://issues.apache.org/jira/browse/AIRFLOW-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Cederstrand updated AIRFLOW-1354:
--
Affects Version/s: (was: 1.8.0)
1.10.0
> www/views.py pool_link produces wrong link
> --
>
> Key: AIRFLOW-1354
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1354
> Project: Apache Airflow
> Issue Type: Bug
> Components: webapp
>Affects Versions: 1.10.0
>Reporter: Erik Cederstrand
>Priority: Minor
>
> The pool_link() function in www/views.py which produces the link on the pool
> names in the [/admin/pool/|http://green.nzcorp.net:7105/admin/pool/] list
> view builds a link to
> {code:none}
> '/admin/taskinstance/?flt1_pool_equals=' + m.pool{code}
> but should return
> {code:none}
> '/admin/pool/?flt1_pool_equals=' + m.pool{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-1354) www/views.py pool_link produces wrong link
[
https://issues.apache.org/jira/browse/AIRFLOW-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Cederstrand updated AIRFLOW-1354:
--
Description:
The pool_link() function in www/views.py which produces the link on the pool
names in the [/admin/pool/|http://green.nzcorp.net:7105/admin/pool/] list view
builds a link to
{code:none}
'/admin/taskinstance/?flt1_pool_equals=' + m.pool{code}
but should return
{code:none}
'/admin/pool/?flt1_pool_equals=' + m.pool{code}
was:
The pool_link() function in www/views.py builds a link to
{code:none}'/admin/taskinstance/?flt1_pool_equals=' + m.pool{code}
but should probably return
{code:none}'/admin/pool/?flt1_pool_equals=' + m.pool{code}
> www/views.py pool_link produces wrong link
> --
>
> Key: AIRFLOW-1354
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1354
> Project: Apache Airflow
> Issue Type: Bug
> Components: webapp
>Affects Versions: 1.8.0
>Reporter: Erik Cederstrand
>Priority: Minor
>
> The pool_link() function in www/views.py which produces the link on the pool
> names in the [/admin/pool/|http://green.nzcorp.net:7105/admin/pool/] list
> view builds a link to
> {code:none}
> '/admin/taskinstance/?flt1_pool_equals=' + m.pool{code}
> but should return
> {code:none}
> '/admin/pool/?flt1_pool_equals=' + m.pool{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (AIRFLOW-3134) Add New Operator - MySQLToS3TransformOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-3134 started by Stefano Francavilla. > Add New Operator - MySQLToS3TransformOperator > - > > Key: AIRFLOW-3134 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3134 > Project: Apache Airflow > Issue Type: New Feature > Components: operators >Affects Versions: 1.10.0 >Reporter: Stefano Francavilla >Assignee: Stefano Francavilla >Priority: Minor > Labels: MissingFeature, operators > > Taking inspiration from the [S3Transform > Operator|https://github.com/apache/incubator-airflow/blob/master/airflow/operators/s3_file_transform_operator.py] > and from an use case I was working in the past weeks, I was wondering if it > would be useful to add a new operator: "MySQLToS3TransformOperator". > The operator allows to transfer (transformed) data resulting from a SELECT > statement to an S3 Bucket. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-949) kill_process_tree does not kill the root process
[ https://issues.apache.org/jira/browse/AIRFLOW-949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Cederstrand closed AIRFLOW-949. Resolution: Duplicate > kill_process_tree does not kill the root process > > > Key: AIRFLOW-949 > URL: https://issues.apache.org/jira/browse/AIRFLOW-949 > Project: Apache Airflow > Issue Type: Bug > Components: utils >Affects Versions: 1.8.0rc4 >Reporter: Erik Cederstrand >Priority: Major > Labels: patch > Attachments: helpers.patch > > > The kill_process_tree() function in airflow/utils/helper.py does not attempt > to kill the root process. Since there's also a kill_descendant_processes() > function, I assume that was the intent. > Also, according to the comments, the intent is to send first SIGTERM, and > then SIGKILL, to decendant processes. But in fact, SIGTERM is sent twice. > The attached patch fixes both problems. > This was found while investigating why the airflow worker would not kill > certain jobs that had crashed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-3134) Add New Operator - MySQLToS3TransformOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefano Francavilla reassigned AIRFLOW-3134: Assignee: Stefano Francavilla > Add New Operator - MySQLToS3TransformOperator > - > > Key: AIRFLOW-3134 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3134 > Project: Apache Airflow > Issue Type: New Feature > Components: operators >Affects Versions: 1.10.0 >Reporter: Stefano Francavilla >Assignee: Stefano Francavilla >Priority: Minor > Labels: MissingFeature, operators > > Taking inspiration from the [S3Transform > Operator|https://github.com/apache/incubator-airflow/blob/master/airflow/operators/s3_file_transform_operator.py] > and from an use case I was working in the past weeks, I was wondering if it > would be useful to add a new operator: "MySQLToS3TransformOperator". > The operator allows to transfer (transformed) data resulting from a SELECT > statement to an S3 Bucket. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-949) kill_process_tree does not kill the root process
[ https://issues.apache.org/jira/browse/AIRFLOW-949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636570#comment-16636570 ] Erik Cederstrand commented on AIRFLOW-949: -- Closing as this seems fixed by AIRFLOW-1109 > kill_process_tree does not kill the root process > > > Key: AIRFLOW-949 > URL: https://issues.apache.org/jira/browse/AIRFLOW-949 > Project: Apache Airflow > Issue Type: Bug > Components: utils >Affects Versions: 1.8.0rc4 >Reporter: Erik Cederstrand >Priority: Major > Labels: patch > Attachments: helpers.patch > > > The kill_process_tree() function in airflow/utils/helper.py does not attempt > to kill the root process. Since there's also a kill_descendant_processes() > function, I assume that was the intent. > Also, according to the comments, the intent is to send first SIGTERM, and > then SIGKILL, to decendant processes. But in fact, SIGTERM is sent twice. > The attached patch fixes both problems. > This was found while investigating why the airflow worker would not kill > certain jobs that had crashed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] codecov-io edited a comment on issue #3989: [AIRFLOW-1945] Autoscale celery workers for airflow added
codecov-io edited a comment on issue #3989: [AIRFLOW-1945] Autoscale celery workers for airflow added URL: https://github.com/apache/incubator-airflow/pull/3989#issuecomment-426543786 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3989?src=pr=h1) Report > Merging [#3989](https://codecov.io/gh/apache/incubator-airflow/pull/3989?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/45ed3cec826073a8316dea83035d7da4a2ff0f2e?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3989?src=pr=tree) ```diff @@Coverage Diff @@ ## master#3989 +/- ## == - Coverage 75.49% 75.48% -0.01% == Files 199 199 Lines 1594715947 == - Hits1203912038 -1 - Misses 3908 3909 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3989?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/bin/cli.py](https://codecov.io/gh/apache/incubator-airflow/pull/3989/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `64.43% <ø> (ø)` | :arrow_up: | | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/3989/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `91.62% <0%> (-0.05%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3989?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3989?src=pr=footer). Last update [45ed3ce...1b496c4](https://codecov.io/gh/apache/incubator-airflow/pull/3989?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #3989: [AIRFLOW-1945] Autoscale celery workers for airflow added
codecov-io commented on issue #3989: [AIRFLOW-1945] Autoscale celery workers for airflow added URL: https://github.com/apache/incubator-airflow/pull/3989#issuecomment-426543786 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3989?src=pr=h1) Report > Merging [#3989](https://codecov.io/gh/apache/incubator-airflow/pull/3989?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/45ed3cec826073a8316dea83035d7da4a2ff0f2e?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3989?src=pr=tree) ```diff @@Coverage Diff @@ ## master#3989 +/- ## == - Coverage 75.49% 75.48% -0.01% == Files 199 199 Lines 1594715947 == - Hits1203912038 -1 - Misses 3908 3909 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3989?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/bin/cli.py](https://codecov.io/gh/apache/incubator-airflow/pull/3989/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `64.43% <ø> (ø)` | :arrow_up: | | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/3989/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `91.62% <0%> (-0.05%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3989?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3989?src=pr=footer). Last update [45ed3ce...1b496c4](https://codecov.io/gh/apache/incubator-airflow/pull/3989?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] simonvanderveldt commented on issue #3504: [AIRFLOW-2310]: Add AWS Glue Job Compatibility to Airflow
simonvanderveldt commented on issue #3504: [AIRFLOW-2310]: Add AWS Glue Job Compatibility to Airflow URL: https://github.com/apache/incubator-airflow/pull/3504#issuecomment-426541089 @OElesin Have you been able to make any progress on this? There's one duplicate file in there `airflow/contrib/aws_glue_job_hook.py` vs `airflow/contrib/hook/aws_glue_job_hook.py`. FYI All the unrelated changes are because you merged `master` into your branch instead of rebasing your branch on top of `master` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-3145) reduce airflow dag scheduling latency in production
jack created AIRFLOW-3145:
-
Summary: reduce airflow dag scheduling latency in production
Key: AIRFLOW-3145
URL: https://issues.apache.org/jira/browse/AIRFLOW-3145
Project: Apache Airflow
Issue Type: Task
Affects Versions: 1.10.1
Reporter: jack
The docs says:
* {{}}
{code:java}
max_threads: Scheduler will spawn multiple threads in parallel to schedule
dags. This is controlled by max_threads with default value of 2. User should
increase this value to a larger value(e.g numbers of cpus where scheduler runs
- 1) in production.{code}
[https://airflow.apache.org/faq.html#how-can-my-airflow-dag-run-faster]
However in the Aiflow.cfg the comment is
{code:java}
#The scheduler can run multiple threads in parallel to schedule dags.
# This defines how many threads will run.
max_threads = 2{code}
May I suggest that the
{code:java}
User should increase this value to a larger value(e.g numbers of cpus where
scheduler runs - 1) in production.{code}
will also be written in the comment at Aiflow.cfg
Not everyone read the docs from top to bottom this could be easily missed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3136) Scheduler Failing the Task retries run while processing Executor Events
[
https://issues.apache.org/jira/browse/AIRFLOW-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636546#comment-16636546
]
ASF GitHub Bot commented on AIRFLOW-3136:
-
vardancse opened a new pull request #3994: [AIRFLOW-3136] Add retry_number to
TaskInstance Key property to avoid race condition
URL: https://github.com/apache/incubator-airflow/pull/3994
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following
[AIRFLOW-3136](https://issues.apache.org/jira/browse/AIRFLOW-3136) issues and
references them in the PR title.
### Description
- [ ] Here are some details about my PR.
We were seeing an intermittent issue where executor reports task instance
finished while task says it's in queue state, it was due to a race condition
between scheduler which was clearing event_buffer in _process_executor_events
method in jobs.py executor was about to put next_retry task's status as running
which was failed in previous try. So, we thought to add retry_number as the
member of TaskInstance key property.
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Scheduler Failing the Task retries run while processing Executor Events
> ---
>
> Key: AIRFLOW-3136
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3136
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 1.9.0
>Reporter: raman
>Priority: Major
>
> Following behaviour is observed with Airflow 1.9 with LocalExecutor mode
>
> Airflow scheduler processes the executor events in
> "_process_executor_events(self, simple_dag_bag, session=None)" function of
> jobs.py.
> The events are identified by key which is composed of dag id, task id,
> execution date. So all retries of a task have the same key.
> If task retry interval is very small like 30 seconds than scheduler might
> schedule the next retry run while the previous task run result is still in
> the executor event queue.
> Current task run might be in queued state while scheduler is processing the
> executor's previous events Which might make scheduler to fail the current run
> because of following code in the jobs.py file
> def _process_executor_events(self, simple_dag_bag, session=None):
> """
> Respond to executor events.
> """
> # TODO: this shares quite a lot of code with _manage_executor_state
> TI = models.TaskInstance
> *for key, state in
> list(self.executor.get_event_buffer(simple_dag_bag.dag_ids)*
> *.items()):*
> dag_id, task_id, execution_date = key
> self.log.info(
> "Executor reports %s.%s execution_date=%s as %s",
> dag_id, task_id, execution_date, state
> )
> if state == State.FAILED or state == State.SUCCESS:
> qry = session.query(TI).filter(TI.dag_id == dag_id,
> TI.task_id == task_id,
> TI.execution_date == execution_date)
> ti = qry.first()
> if not ti:
> self.log.warning("TaskInstance %s went missing from the database", ti)
> continue
> TODO: should we fail RUNNING as well, as we do in Backfills?
> *if ti.state == State.QUEUED:*
> msg = ("Executor reports task instance %s finished (%s) "
> "although the task says its %s. Was the task "
> "killed externally?".format(ti, state, ti.state))
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1354) www/views.py pool_link produces wrong link
[
https://issues.apache.org/jira/browse/AIRFLOW-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636545#comment-16636545
]
ASF GitHub Bot commented on AIRFLOW-1354:
-
ecederstrand opened a new pull request #3993: [AIRFLOW-1354] Fix pool_link()
return value
URL: https://github.com/apache/incubator-airflow/pull/3993
The URL had a copy-paste error
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> www/views.py pool_link produces wrong link
> --
>
> Key: AIRFLOW-1354
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1354
> Project: Apache Airflow
> Issue Type: Bug
> Components: webapp
>Affects Versions: 1.8.0
>Reporter: Erik Cederstrand
>Priority: Minor
>
> The pool_link() function in www/views.py builds a link to
> {code:none}'/admin/taskinstance/?flt1_pool_equals=' + m.pool{code}
> but should probably return
> {code:none}'/admin/pool/?flt1_pool_equals=' + m.pool{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] vardancse opened a new pull request #3994: [AIRFLOW-3136] Add retry_number to TaskInstance Key property to avoid race condition
vardancse opened a new pull request #3994: [AIRFLOW-3136] Add retry_number to
TaskInstance Key property to avoid race condition
URL: https://github.com/apache/incubator-airflow/pull/3994
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following
[AIRFLOW-3136](https://issues.apache.org/jira/browse/AIRFLOW-3136) issues and
references them in the PR title.
### Description
- [ ] Here are some details about my PR.
We were seeing an intermittent issue where executor reports task instance
finished while task says it's in queue state, it was due to a race condition
between scheduler which was clearing event_buffer in _process_executor_events
method in jobs.py executor was about to put next_retry task's status as running
which was failed in previous try. So, we thought to add retry_number as the
member of TaskInstance key property.
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ecederstrand opened a new pull request #3993: [AIRFLOW-1354] Fix pool_link() return value
ecederstrand opened a new pull request #3993: [AIRFLOW-1354] Fix pool_link()
return value
URL: https://github.com/apache/incubator-airflow/pull/3993
The URL had a copy-paste error
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
