[GitHub] [airflow] codecov-io edited a comment on issue #6764: [AIRFLOW-6206] Move and rename AWS batch operator [AIP-21]
codecov-io edited a comment on issue #6764: [AIRFLOW-6206] Move and rename AWS batch operator [AIP-21] URL: https://github.com/apache/airflow/pull/6764#issuecomment-563485064 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6764?src=pr=h1) Report > Merging [#6764](https://codecov.io/gh/apache/airflow/pull/6764?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/69ef1c68c262d9678366e42d4e8e1426363a7f9e?src=pr=desc) will **decrease** coverage by `0.13%`. > The diff coverage is `96.09%`. [](https://codecov.io/gh/apache/airflow/pull/6764?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6764 +/- ## == - Coverage 84.62% 84.48% -0.14% == Files 673 677 +4 Lines 3827939076 +797 == + Hits3239233015 +623 - Misses 5887 6061 +174 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6764?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/operators/awsbatch\_operator.py](https://codecov.io/gh/apache/airflow/pull/6764/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9hd3NiYXRjaF9vcGVyYXRvci5weQ==) | `100% <100%> (+4.16%)` | :arrow_up: | | [airflow/providers/amazon/aws/operators/batch.py](https://codecov.io/gh/apache/airflow/pull/6764/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9vcGVyYXRvcnMvYmF0Y2gucHk=) | `95.83% <95.83%> (ø)` | | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6764/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6764/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6764/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6764/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6764/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.2% <0%> (-20.52%)` | :arrow_down: | | [airflow/contrib/operators/jira\_operator.py](https://codecov.io/gh/apache/airflow/pull/6764/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9qaXJhX29wZXJhdG9yLnB5) | `71.87% <0%> (-4.8%)` | :arrow_down: | | [airflow/contrib/hooks/jira\_hook.py](https://codecov.io/gh/apache/airflow/pull/6764/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2ppcmFfaG9vay5weQ==) | `75.67% <0%> (-4.33%)` | :arrow_down: | | [airflow/contrib/sensors/jira\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6764/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvamlyYV9zZW5zb3IucHk=) | `56.89% <0%> (-3.11%)` | :arrow_down: | | ... and [8 more](https://codecov.io/gh/apache/airflow/pull/6764/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6764?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6764?src=pr=footer). Last update [69ef1c6...c9dac4c](https://codecov.io/gh/apache/airflow/pull/6764?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dstandish edited a comment on issue #6376: [WIP] [AIRFLOW-5705] Add creds backend classes including AWS SSM
dstandish edited a comment on issue #6376: [WIP] [AIRFLOW-5705] Add creds backend classes including AWS SSM URL: https://github.com/apache/airflow/pull/6376#issuecomment-565763910 @mik-laj @ashb what do you think about this one? now that #6440 (cloud sql dynamic conn generation fix) is merged, there's nothing really standing in the way of this one, except for considerations of whether this is right structure. Some example use cases... 1. temporary dev / test cluster: with astronomer, it's easy to spin up new clusters. so for testing something, or special backfill job we can easily create temporary dev cluster. with this change, we would not have to worry about getting connections into cluster. with one env var, or appropriate IAM, the connection info would be available. 2. sharing creds with developers i assume it is common for developers to have connections store in a file like `connections.env.sh` and then sourced in `.bashrc`. Maybe they _also_ have to have them in `.env` file if they also use docker. maybe other folks manage `airflow connections -a` scripts. this change could eliminate all of that by centralizing creds storage and allowing developers to pull from the same source. depending on the service, creds could potentially be versioned. creds changes would be synchronized for everyone. do you think this needs an AIP? if so can i have access to create (apache jira username is dstandish)? do you have any feedback on current structure? i understand that exactly _how_ we do this is probably more controversial than the general idea of opening up creds sourcing. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dstandish commented on issue #6376: [WIP] [AIRFLOW-5705] Add creds backend classes including AWS SSM
dstandish commented on issue #6376: [WIP] [AIRFLOW-5705] Add creds backend classes including AWS SSM URL: https://github.com/apache/airflow/pull/6376#issuecomment-565763910 @mik-laj @ashb what do you think about this one? now that #6440 (cloud sql dynamic conn generation fix) is merged, there's nothing really standing in the way of this one, except for considerations of whether this is right structure. Some example use cases... 1. temporary dev / test cluster: with astronomer, it's easy to spin up new clusters. so for testing something, or special backfill job we can easily create temporary dev cluster. with this change, we would not have to worry about getting connections into cluster. with one env var, or appropriate IAM, they connection info would be available. 2. sharing creds with developers i assume it is common for developers to have connections store in a file like `connections.env.sh` and then sourced in `.bashrc`. Maybe they _also_ have to have them in `.env` file if they also use docker. maybe other folks manage `airflow connections -a` scripts. this change could eliminate all of that by centralizing creds storage and allowing developers to pull from the same source. depending on the service, creds could potentially be versioned. creds changes would be synchronized for everyone. do you think this needs an AIP? if so can i have access to create (apache jira username is dstandish)? do you have any feedback on current structure? i understand that exactly _how_ we do this is probably more controversial than the general idea of opening up creds sourcing. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dstandish edited a comment on issue #6817: [AIRFLOW-6254] obscure conn extra in logs
dstandish edited a comment on issue #6817: [AIRFLOW-6254] obscure conn extra in
logs
URL: https://github.com/apache/airflow/pull/6817#issuecomment-565762773
@potiuk @ashb i suppose we could iterate `extra_dejson` and just obscure the
values? i.e. so at least keys would be displayed... not sure it adds
worthwhile benefit if the values are obscured anyway.
i.e. `{k: 'XX' for k, _ in extra_dejson.items()}`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] dstandish edited a comment on issue #6817: [AIRFLOW-6254] obscure conn extra in logs
dstandish edited a comment on issue #6817: [AIRFLOW-6254] obscure conn extra in
logs
URL: https://github.com/apache/airflow/pull/6817#issuecomment-565762773
@potiuk @ashb i suppose we could iterate `extra_dejson` and just obscure the
values? i.e. so at least keys would be displayed... not sure it adds
worthwhile benefit if the values are obscured anyway.
i.e. `{k, 'XX' for k, _ in extra_dejson.items()}`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] dstandish edited a comment on issue #6817: [AIRFLOW-6254] obscure conn extra in logs
dstandish edited a comment on issue #6817: [AIRFLOW-6254] obscure conn extra in
logs
URL: https://github.com/apache/airflow/pull/6817#issuecomment-565762773
@potiuk @ashb i suppose we could iterate `extra_dejson` and just obscure the
values? i.e. so at least keys would be displayed... not sure it's worth it
though
i.e. `{k, 'XX' for k, _ in extra_dejson.items()}`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] dstandish edited a comment on issue #6817: [AIRFLOW-6254] obscure conn extra in logs
dstandish edited a comment on issue #6817: [AIRFLOW-6254] obscure conn extra in logs URL: https://github.com/apache/airflow/pull/6817#issuecomment-565762773 @potiuk @ashb i suppose we could iterate `extra_dejson` and just obscure the values? i.e. so at least keys would be displayed... not sure it's worth it though This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dstandish commented on issue #6817: [AIRFLOW-6254] obscure conn extra in logs
dstandish commented on issue #6817: [AIRFLOW-6254] obscure conn extra in logs URL: https://github.com/apache/airflow/pull/6817#issuecomment-565762773 @potiuk @ashb i suppose we could iterate `extra_dejson` and just obscure the keys? not sure it's worth it though This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dggs123 opened a new pull request #6821: [AIRFLOW-5616] Refactor: replace pyhive with presto official client i…
dggs123 opened a new pull request #6821: [AIRFLOW-5616] Refactor: replace
pyhive with presto official client i…
URL: https://github.com/apache/airflow/pull/6821
…n presto hook
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-5616) PrestoHook to use prestodb
[
https://issues.apache.org/jira/browse/AIRFLOW-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16996575#comment-16996575
]
ASF GitHub Bot commented on AIRFLOW-5616:
-
dggs123 commented on pull request #6821: [AIRFLOW-5616] Refactor: replace
pyhive with presto official client i…
URL: https://github.com/apache/airflow/pull/6821
…n presto hook
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> PrestoHook to use prestodb
> --
>
> Key: AIRFLOW-5616
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5616
> Project: Apache Airflow
> Issue Type: Improvement
> Components: hooks
>Affects Versions: 1.10.5
>Reporter: Alexandre Brilhante
>Assignee: Gaurav Sehgal
>Priority: Minor
>
> PrestoHook currently uses PyHive which doesn't support transactions whereas
> prestodb
> ([https://github.com/prestodb/presto-python-client)|https://github.com/prestodb/presto-python-client]
> does. I think it would more flexible to use prestodb as client. I can work
> on a PR.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #6820: [AIRFLOW-6255] Redis Hook Refactor
codecov-io edited a comment on issue #6820: [AIRFLOW-6255] Redis Hook Refactor URL: https://github.com/apache/airflow/pull/6820#issuecomment-565753224 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6820?src=pr=h1) Report > Merging [#6820](https://codecov.io/gh/apache/airflow/pull/6820?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8107651f8f578cc98801f37ae50ebc6be9e9d035?src=pr=desc) will **decrease** coverage by `0.27%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6820?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6820 +/- ## == - Coverage 84.59% 84.32% -0.28% == Files 676 676 Lines 3833138352 +21 == - Hits3242832339 -89 - Misses 5903 6013 +110 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6820?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/sensors/redis\_key\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvcmVkaXNfa2V5X3NlbnNvci5weQ==) | `100% <100%> (ø)` | :arrow_up: | | [...irflow/contrib/operators/redis\_publish\_operator.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9yZWRpc19wdWJsaXNoX29wZXJhdG9yLnB5) | `100% <100%> (ø)` | :arrow_up: | | [airflow/contrib/hooks/redis\_hook.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3JlZGlzX2hvb2sucHk=) | `100% <100%> (ø)` | :arrow_up: | | [airflow/contrib/sensors/redis\_pub\_sub\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvcmVkaXNfcHViX3N1Yl9zZW5zb3IucHk=) | `100% <100%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.2% <0%> (-20.52%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6820?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6820?src=pr=footer). Last update [8107651...1c809b8](https://codecov.io/gh/apache/airflow/pull/6820?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6820: [AIRFLOW-6255] Redis Hook Refactor
codecov-io edited a comment on issue #6820: [AIRFLOW-6255] Redis Hook Refactor URL: https://github.com/apache/airflow/pull/6820#issuecomment-565753224 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6820?src=pr=h1) Report > Merging [#6820](https://codecov.io/gh/apache/airflow/pull/6820?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8107651f8f578cc98801f37ae50ebc6be9e9d035?src=pr=desc) will **decrease** coverage by `0.27%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6820?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6820 +/- ## == - Coverage 84.59% 84.32% -0.28% == Files 676 676 Lines 3833138352 +21 == - Hits3242832339 -89 - Misses 5903 6013 +110 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6820?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/sensors/redis\_key\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvcmVkaXNfa2V5X3NlbnNvci5weQ==) | `100% <100%> (ø)` | :arrow_up: | | [...irflow/contrib/operators/redis\_publish\_operator.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9yZWRpc19wdWJsaXNoX29wZXJhdG9yLnB5) | `100% <100%> (ø)` | :arrow_up: | | [airflow/contrib/hooks/redis\_hook.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3JlZGlzX2hvb2sucHk=) | `100% <100%> (ø)` | :arrow_up: | | [airflow/contrib/sensors/redis\_pub\_sub\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvcmVkaXNfcHViX3N1Yl9zZW5zb3IucHk=) | `100% <100%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.2% <0%> (-20.52%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6820?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6820?src=pr=footer). Last update [8107651...1c809b8](https://codecov.io/gh/apache/airflow/pull/6820?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6820: [AIRFLOW-6255] Redis Hook Refactor
codecov-io commented on issue #6820: [AIRFLOW-6255] Redis Hook Refactor URL: https://github.com/apache/airflow/pull/6820#issuecomment-565753224 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6820?src=pr=h1) Report > Merging [#6820](https://codecov.io/gh/apache/airflow/pull/6820?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8107651f8f578cc98801f37ae50ebc6be9e9d035?src=pr=desc) will **decrease** coverage by `0.27%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6820?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6820 +/- ## == - Coverage 84.59% 84.32% -0.28% == Files 676 676 Lines 3833138352 +21 == - Hits3242832339 -89 - Misses 5903 6013 +110 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6820?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/sensors/redis\_key\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvcmVkaXNfa2V5X3NlbnNvci5weQ==) | `100% <100%> (ø)` | :arrow_up: | | [...irflow/contrib/operators/redis\_publish\_operator.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9yZWRpc19wdWJsaXNoX29wZXJhdG9yLnB5) | `100% <100%> (ø)` | :arrow_up: | | [airflow/contrib/hooks/redis\_hook.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3JlZGlzX2hvb2sucHk=) | `100% <100%> (ø)` | :arrow_up: | | [airflow/contrib/sensors/redis\_pub\_sub\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvcmVkaXNfcHViX3N1Yl9zZW5zb3IucHk=) | `100% <100%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.2% <0%> (-20.52%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6820?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6820?src=pr=footer). Last update [8107651...1c809b8](https://codecov.io/gh/apache/airflow/pull/6820?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6643: [AIRFLOW-6040] Fix KubernetesJobWatcher Read time out error
codecov-io edited a comment on issue #6643: [AIRFLOW-6040] Fix KubernetesJobWatcher Read time out error URL: https://github.com/apache/airflow/pull/6643#issuecomment-558187895 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6643?src=pr=h1) Report > Merging [#6643](https://codecov.io/gh/apache/airflow/pull/6643?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8107651f8f578cc98801f37ae50ebc6be9e9d035?src=pr=desc) will **decrease** coverage by `0.29%`. > The diff coverage is `9.09%`. [](https://codecov.io/gh/apache/airflow/pull/6643?src=pr=tree) ```diff @@ Coverage Diff@@ ## master #6643 +/- ## - Coverage 84.59% 84.3% -0.3% Files 676 676 Lines 38331 38336 +5 - Hits32428 32319-109 - Misses 59036017+114 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6643?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==) | `56.89% <9.09%> (-0.41%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.2% <0%> (-20.52%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6643?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6643?src=pr=footer). Last update [8107651...72799cb](https://codecov.io/gh/apache/airflow/pull/6643?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6643: [AIRFLOW-6040] Fix KubernetesJobWatcher Read time out error
codecov-io edited a comment on issue #6643: [AIRFLOW-6040] Fix KubernetesJobWatcher Read time out error URL: https://github.com/apache/airflow/pull/6643#issuecomment-558187895 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6643?src=pr=h1) Report > Merging [#6643](https://codecov.io/gh/apache/airflow/pull/6643?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8107651f8f578cc98801f37ae50ebc6be9e9d035?src=pr=desc) will **decrease** coverage by `0.29%`. > The diff coverage is `9.09%`. [](https://codecov.io/gh/apache/airflow/pull/6643?src=pr=tree) ```diff @@ Coverage Diff@@ ## master #6643 +/- ## - Coverage 84.59% 84.3% -0.3% Files 676 676 Lines 38331 38336 +5 - Hits32428 32319-109 - Misses 59036017+114 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6643?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==) | `56.89% <9.09%> (-0.41%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.2% <0%> (-20.52%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6643?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6643?src=pr=footer). Last update [8107651...72799cb](https://codecov.io/gh/apache/airflow/pull/6643?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request #6820: [AIRFLOW-6255] Redis Hook Refactor
nuclearpinguin commented on a change in pull request #6820: [AIRFLOW-6255]
Redis Hook Refactor
URL: https://github.com/apache/airflow/pull/6820#discussion_r357936721
##
File path: airflow/contrib/hooks/redis_hook.py
##
@@ -66,3 +66,47 @@ def get_conn(self):
db=self.db)
return self.redis
+
+def set_key_value(self, k, v):
+"""
+Create redis key ${k} with value ${v}
+"""
+client = self.get_conn()
+
+return client.set(k, v)
+
+def get_key(self, k):
+""" Get value of key ${key} stored in Redis """
+client = self.get_conn()
+
+return client.get(k)
+
+def delete_key(self, k):
+""" Delete key ${k} from redis """
+client = self.get_conn()
+
+return client.delete(k)
+
+def publish_message(self, channel, message):
+""" Publish message ${message} to channel ${channel}"""
+client = self.get_conn()
+
+client.publish(channel=channel, message=message)
+
+def check_if_key_exists(self, key):
+""" Verify if key ${k} exists in Redis """
+client = self.get_conn()
+
+return client.exists(key)
+
+def create_pubsub(self):
+""" Create pubsub object which can subscribe to Redis channel """
+client = self.get_conn()
+
+return client.pubsub()
Review comment:
Here, adding type annotation will be really helpful because user could
easily go to Pubsub object.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request #6820: [AIRFLOW-6255] Redis Hook Refactor
nuclearpinguin commented on a change in pull request #6820: [AIRFLOW-6255] Redis Hook Refactor URL: https://github.com/apache/airflow/pull/6820#discussion_r357936828 ## File path: airflow/contrib/sensors/redis_pub_sub_sensor.py ## @@ -40,7 +40,7 @@ def __init__(self, channels, redis_conn_id, *args, **kwargs): super().__init__(*args, **kwargs) self.channels = channels self.redis_conn_id = redis_conn_id -self.pubsub = RedisHook(redis_conn_id=self.redis_conn_id).get_conn().pubsub() +self.pubsub = RedisHook(redis_conn_id=self.redis_conn_id).create_pubsub() Review comment: It would be nice to move this invocation to `poke` method. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request #6820: [AIRFLOW-6255] Redis Hook Refactor
nuclearpinguin commented on a change in pull request #6820: [AIRFLOW-6255]
Redis Hook Refactor
URL: https://github.com/apache/airflow/pull/6820#discussion_r357936768
##
File path: airflow/contrib/operators/redis_publish_operator.py
##
@@ -61,6 +61,6 @@ def execute(self, context):
self.log.info('Sending messsage %s to Redis on channel %s',
self.message, self.channel)
-result = redis_hook.get_conn().publish(channel=self.channel,
message=self.message)
+result = redis_hook.publish_message(channel=self.channel,
message=self.message)
Review comment:
The result will always be `None` because the hook method does not return
anything
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request #6820: [AIRFLOW-6255] Redis Hook Refactor
nuclearpinguin commented on a change in pull request #6820: [AIRFLOW-6255] Redis Hook Refactor URL: https://github.com/apache/airflow/pull/6820#discussion_r357936601 ## File path: airflow/contrib/hooks/redis_hook.py ## @@ -66,3 +66,47 @@ def get_conn(self): db=self.db) return self.redis + +def set_key_value(self, k, v): Review comment: ```suggestion def set_key_value(self, key: str, v: value: Any): ``` Using meaningful names is appreciated. Airflow also encourages adding type annotations ;) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request #6820: [AIRFLOW-6255] Redis Hook Refactor
nuclearpinguin commented on a change in pull request #6820: [AIRFLOW-6255]
Redis Hook Refactor
URL: https://github.com/apache/airflow/pull/6820#discussion_r357936630
##
File path: airflow/contrib/hooks/redis_hook.py
##
@@ -66,3 +66,47 @@ def get_conn(self):
db=self.db)
return self.redis
+
+def set_key_value(self, k, v):
+"""
+Create redis key ${k} with value ${v}
+"""
+client = self.get_conn()
+
+return client.set(k, v)
+
+def get_key(self, k):
+""" Get value of key ${key} stored in Redis """
+client = self.get_conn()
+
+return client.get(k)
+
+def delete_key(self, k):
+""" Delete key ${k} from redis """
+client = self.get_conn()
+
+return client.delete(k)
+
+def publish_message(self, channel, message):
+""" Publish message ${message} to channel ${channel}"""
+client = self.get_conn()
+
+client.publish(channel=channel, message=message)
Review comment:
Does this method returns True/False? If yes it would be great to return this
information from hook method.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6643: [AIRFLOW-6040] Fix KubernetesJobWatcher Read time out error
codecov-io edited a comment on issue #6643: [AIRFLOW-6040] Fix KubernetesJobWatcher Read time out error URL: https://github.com/apache/airflow/pull/6643#issuecomment-558187895 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6643?src=pr=h1) Report > Merging [#6643](https://codecov.io/gh/apache/airflow/pull/6643?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8107651f8f578cc98801f37ae50ebc6be9e9d035?src=pr=desc) will **decrease** coverage by `0.29%`. > The diff coverage is `9.09%`. [](https://codecov.io/gh/apache/airflow/pull/6643?src=pr=tree) ```diff @@ Coverage Diff@@ ## master #6643 +/- ## - Coverage 84.59% 84.3% -0.3% Files 676 676 Lines 38331 38336 +5 - Hits32428 32319-109 - Misses 59036017+114 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6643?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==) | `56.89% <9.09%> (-0.41%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.2% <0%> (-20.52%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6643?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6643?src=pr=footer). Last update [8107651...72799cb](https://codecov.io/gh/apache/airflow/pull/6643?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6643: [AIRFLOW-6040] Fix KubernetesJobWatcher Read time out error
codecov-io edited a comment on issue #6643: [AIRFLOW-6040] Fix KubernetesJobWatcher Read time out error URL: https://github.com/apache/airflow/pull/6643#issuecomment-558187895 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6643?src=pr=h1) Report > Merging [#6643](https://codecov.io/gh/apache/airflow/pull/6643?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8107651f8f578cc98801f37ae50ebc6be9e9d035?src=pr=desc) will **decrease** coverage by `0.29%`. > The diff coverage is `9.09%`. [](https://codecov.io/gh/apache/airflow/pull/6643?src=pr=tree) ```diff @@ Coverage Diff@@ ## master #6643 +/- ## - Coverage 84.59% 84.3% -0.3% Files 676 676 Lines 38331 38336 +5 - Hits32428 32319-109 - Misses 59036017+114 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6643?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==) | `56.89% <9.09%> (-0.41%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.2% <0%> (-20.52%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6643?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6643?src=pr=footer). Last update [8107651...72799cb](https://codecov.io/gh/apache/airflow/pull/6643?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6255) Redis Hook - refactor & add most important methods
[
https://issues.apache.org/jira/browse/AIRFLOW-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16996551#comment-16996551
]
ASF GitHub Bot commented on AIRFLOW-6255:
-
mgorsk1 commented on pull request #6820: [AIRFLOW-6255] Redis Hook Refactor
URL: https://github.com/apache/airflow/pull/6820
Make sure you have checked _all_ steps below.
### Jira
- [V] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [V] Here are some details about my PR, including screenshots of any UI
changes:
This PR aims to refactor Redis Hook and all classes (sensors/operators)
which are using it. So far every component used Redis Hook by interfacing
directly with redis client. This change adds dedicated hook methods to interact
with Redis and moves away from interfacing with redis client directly.
### Tests
- [V] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
I've extended tests/contrib/hooks/test_redis_hook.py and refactored
tests/contrib/operators/test_redis_pub_sub_operator.py,
tests/contrib/sensors/test_redis_sensor.py and
tests/contrib/sensors/test_redis_pub_sub_sensor.py to align with added hook
methods.
### Commits
- [V] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [V] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Redis Hook - refactor & add most important methods
> --
>
> Key: AIRFLOW-6255
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6255
> Project: Apache Airflow
> Issue Type: Improvement
> Components: hooks
>Affects Versions: 1.10.6
>Reporter: Mariusz Górski
>Assignee: Mariusz Górski
>Priority: Minor
> Labels: hooks, redis, refactor
>
> Currently Redis Hook consists of get_conn func only and every operation using
> it is calling Redis client methods directly. This Issue aims to extend Redis
> Hook so it provides all necessary methods wrapping Redis functionality used
> so far.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] mgorsk1 opened a new pull request #6820: [AIRFLOW-6255] Redis Hook Refactor
mgorsk1 opened a new pull request #6820: [AIRFLOW-6255] Redis Hook Refactor
URL: https://github.com/apache/airflow/pull/6820
Make sure you have checked _all_ steps below.
### Jira
- [V] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [V] Here are some details about my PR, including screenshots of any UI
changes:
This PR aims to refactor Redis Hook and all classes (sensors/operators)
which are using it. So far every component used Redis Hook by interfacing
directly with redis client. This change adds dedicated hook methods to interact
with Redis and moves away from interfacing with redis client directly.
### Tests
- [V] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
I've extended tests/contrib/hooks/test_redis_hook.py and refactored
tests/contrib/operators/test_redis_pub_sub_operator.py,
tests/contrib/sensors/test_redis_sensor.py and
tests/contrib/sensors/test_redis_pub_sub_sensor.py to align with added hook
methods.
### Commits
- [V] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [V] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request #6811: [WIP][AIRFLOW-6245] Add custom waiters for AWS batch jobs
nuclearpinguin commented on a change in pull request #6811: [WIP][AIRFLOW-6245]
Add custom waiters for AWS batch jobs
URL: https://github.com/apache/airflow/pull/6811#discussion_r357934353
##
File path: tests/providers/amazon/aws/operators/test_batch.py
##
@@ -119,52 +122,52 @@ def test_execute_with_failures(self):
parameters=None,
)
-def test_wait_end_tasks(self):
-client_mock = mock.Mock()
-self.batch.jobId = JOB_ID
-self.batch.client = client_mock
-
-self.batch._wait_for_task_ended()
-
-
client_mock.get_waiter.assert_called_once_with("job_execution_complete")
-
client_mock.get_waiter.return_value.wait.assert_called_once_with(jobs=[JOB_ID])
-self.assertEqual(sys.maxsize,
client_mock.get_waiter.return_value.config.max_attempts)
-
-@mock.patch("airflow.contrib.operators.awsbatch_operator.randint")
+# def test_wait_end_tasks(self):
+# import sys
+# client_mock = mock.Mock()
+# self.batch.jobId = JOB_ID
+# self.batch.client = client_mock
+#
+# self.batch._wait_for_task_ended()
+#
+#
client_mock.get_waiter.assert_called_once_with("job_execution_complete")
+#
client_mock.get_waiter.return_value.wait.assert_called_once_with(jobs=[JOB_ID])
+# self.assertEqual(sys.maxsize,
client_mock.get_waiter.return_value.config.max_attempts)
Review comment:
Why is this test commented?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request #6811: [WIP][AIRFLOW-6245] Add custom waiters for AWS batch jobs
nuclearpinguin commented on a change in pull request #6811: [WIP][AIRFLOW-6245]
Add custom waiters for AWS batch jobs
URL: https://github.com/apache/airflow/pull/6811#discussion_r357934353
##
File path: tests/providers/amazon/aws/operators/test_batch.py
##
@@ -119,52 +122,52 @@ def test_execute_with_failures(self):
parameters=None,
)
-def test_wait_end_tasks(self):
-client_mock = mock.Mock()
-self.batch.jobId = JOB_ID
-self.batch.client = client_mock
-
-self.batch._wait_for_task_ended()
-
-
client_mock.get_waiter.assert_called_once_with("job_execution_complete")
-
client_mock.get_waiter.return_value.wait.assert_called_once_with(jobs=[JOB_ID])
-self.assertEqual(sys.maxsize,
client_mock.get_waiter.return_value.config.max_attempts)
-
-@mock.patch("airflow.contrib.operators.awsbatch_operator.randint")
+# def test_wait_end_tasks(self):
+# import sys
+# client_mock = mock.Mock()
+# self.batch.jobId = JOB_ID
+# self.batch.client = client_mock
+#
+# self.batch._wait_for_task_ended()
+#
+#
client_mock.get_waiter.assert_called_once_with("job_execution_complete")
+#
client_mock.get_waiter.return_value.wait.assert_called_once_with(jobs=[JOB_ID])
+# self.assertEqual(sys.maxsize,
client_mock.get_waiter.return_value.config.max_attempts)
Review comment:
Why this test is commented?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-5727) Incorrect SqoopOperator --connect param with jdbc connection
[ https://issues.apache.org/jira/browse/AIRFLOW-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16996504#comment-16996504 ] ASF subversion and git services commented on AIRFLOW-5727: -- Commit 55ae4eca3474817304fb503e4f6af65cc4fb477f in airflow's branch refs/heads/v1-10-test from Mindaugas [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=55ae4ec ] [AIRFLOW-5727] SqoopHook: Build --connect parameter only if port/schema are defined (#6397) (cherry picked from commit 4649563b56da1da95df7b659f3c3959cbf9a4e71) > Incorrect SqoopOperator --connect param with jdbc connection > > > Key: AIRFLOW-5727 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5727 > Project: Apache Airflow > Issue Type: Bug > Components: contrib >Affects Versions: 1.10.5 >Reporter: Mindaugas Pocius >Priority: Minor > Fix For: 1.10.7 > > > When using JDBC Connection in SqoopOperator it's --connect param is > incorrectly set. I.E.: > _jdbc:oracle:thin:@//:/:*None/None*_. This happens, since > JDBC Connection only has url set and no port or schema in DB. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5798) Set default to None for external_task_id in ExternalTaskSensor
[ https://issues.apache.org/jira/browse/AIRFLOW-5798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16996505#comment-16996505 ] ASF subversion and git services commented on AIRFLOW-5798: -- Commit c1e9868055036d8bca12ca9c10e8344292cdb0c7 in airflow's branch refs/heads/v1-10-test from Vladimir Gavrilenko [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=c1e9868 ] [AIRFLOW-5798] Set default ExternalTaskSensor.external_task_id (#6431) - fix type annotation for ExternalTaskSensor (cherry picked from commit 3fb9d4935f5d30feb4c077f17fdbda04bc195e90) > Set default to None for external_task_id in ExternalTaskSensor > -- > > Key: AIRFLOW-5798 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5798 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Affects Versions: 1.10.5 >Reporter: Felix Uellendall >Priority: Major > Fix For: 1.10.7 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5695) Cannot run a task from UI if its state is None
[ https://issues.apache.org/jira/browse/AIRFLOW-5695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16996503#comment-16996503 ] ASF subversion and git services commented on AIRFLOW-5695: -- Commit 47cf28564f1fefec4da1b2f207e360aacc30cea3 in airflow's branch refs/heads/v1-10-test from Ping Zhang [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=47cf285 ] [AIRFLOW-5695] use RUNNING_DEPS to check run from UI (#6367) (cherry picked from commit 3d3cc41678297ac113287760ed5fe9968a96f1d9) > Cannot run a task from UI if its state is None > -- > > Key: AIRFLOW-5695 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5695 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: 1.10.4 >Reporter: Ping Zhang >Priority: Major > Fix For: 1.10.7 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5731) Make output format from list commands configurable
[
https://issues.apache.org/jira/browse/AIRFLOW-5731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor updated AIRFLOW-5731:
---
Fix Version/s: (was: 1.10.7)
Not pulling in to 1.10.7 - code has changed too much to make this worth the
hassle of cherry picking.
> Make output format from list commands configurable
> --
>
> Key: AIRFLOW-5731
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5731
> Project: Apache Airflow
> Issue Type: Improvement
> Components: cli
>Affects Versions: 1.10.5
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Major
> Fix For: 2.0.0
>
>
> Currently, list commands such as {{airflow connections list}} and {{airflow
> users list}} display prettily formatted tables, but they are not so suitable
> for parsing and processing. It'd be useful if users can specify output
> format, e.g. csv or tsv.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5640) BaseOperator email parameter is wrongly typed and not documented
[
https://issues.apache.org/jira/browse/AIRFLOW-5640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16996497#comment-16996497
]
ASF subversion and git services commented on AIRFLOW-5640:
--
Commit 88088e64d9625e38bb78cee1309384e8f0ac562a in airflow's branch
refs/heads/v1-10-test from Martin Winkel
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=88088e6 ]
[AIRFLOW-5640] Document and test `email` parameters of BaseOperator (#6315)
* Refactored get_email_address_list to have a better
separation between string handling and other iterables.
* Explicitely casting get_email_address_list argument
to a list in case the argument was an iterable. This
enables direct support for tuples, sets or the like.
* Fixed type annotation of email parameter of
BaseOperator to show that iterables are directly
supported.
* Added docstring entries for email, email_on_retry,
email_on_failure and queue in BaseOperator.
(cherry picked from commit 9ec562f88ef8e690f0b17526878b46847f0422e7)
> BaseOperator email parameter is wrongly typed and not documented
>
>
> Key: AIRFLOW-5640
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5640
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: 1.10.5
>Reporter: Cedrik Neumann
>Assignee: Martin Winkel
>Priority: Major
> Fix For: 1.10.7
>
>
> The {{email}} field is not documented in BaseOperator and furthermore the
> type annotation {{str}} is wrong
> [here|https://github.com/apache/airflow/blob/master/airflow/models/baseoperator.py#L273].
> The method {{get_email_address_list}} clearly accepts lists of strings as
> well as comma and semicolon delimited lists:
> [here|https://github.com/apache/airflow/blob/88989200a66291580088188f06a6db503ac823e2/airflow/utils/email.py#L123]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5643) S3Hook logic is duplicated and could be clearer
[ https://issues.apache.org/jira/browse/AIRFLOW-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16996496#comment-16996496 ] ASF subversion and git services commented on AIRFLOW-5643: -- Commit f0e00dd552354903b3001531f950848419685636 in airflow's branch refs/heads/v1-10-test from Louis Guitton [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=f0e00dd ] [AIRFLOW-5643] Reduce duplicated logic in S3Hook (#6313) S3Hook.load_bytes is duplicating the logic of S3Hook.load_file_obj. Instead, we should stay consistent : S3Hook.load_string is already delegating the logic to S3Hook.load_bytes, so we can use the same approach to delegate to S3Hook.load_file_obj (cherry picked from commit ac42428bf530c259ab1b0dca08458c1ebf49b04a) > S3Hook logic is duplicated and could be clearer > --- > > Key: AIRFLOW-5643 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5643 > Project: Apache Airflow > Issue Type: Improvement > Components: aws, hooks >Affects Versions: 1.10.5 >Reporter: Louis Guitton >Assignee: Louis Guitton >Priority: Trivial > Fix For: 1.10.7 > > > S3Hook.load_bytes is duplicating the logic of S3Hook.load_file_obj > [https://github.com/apache/airflow/blob/master/airflow/hooks/S3_hook.py#L464-L539|https://github.com/apache/airflow/blob/master/airflow/hooks/S3_hook.py#L465-L539] > Instead, we should stay consistent : S3Hook.load_string is already delegating > the logic to S3Hook.load_bytes, so we can use the same approach to delegate > to S3Hook.load_file_obj -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5339) Wait forever spark hook if python version is 3
[
https://issues.apache.org/jira/browse/AIRFLOW-5339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor updated AIRFLOW-5339:
---
Fix Version/s: (was: 1.10.7)
2.0.0
Marking this as resolved for 2.0.0 only as no one has confirmed how this change
will behave with python 2.
> Wait forever spark hook if python version is 3
> --
>
> Key: AIRFLOW-5339
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5339
> Project: Apache Airflow
> Issue Type: Bug
> Components: hooks
>Affects Versions: 1.10.0, 1.10.1, 1.10.2, 1.10.3, 1.10.4
>Reporter: Juhong Jung
>Assignee: Juhong Jung
>Priority: Major
> Fix For: 2.0.0
>
>
> I found spark hook is not terminating forever.
> Because Python 3 *{{subprocess.Popen}}* return {{*bytes*}},
> *{{iter(sp.stdout.readlin, '')}}* pattern codes are not work.
> I will submit PR.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] ashb merged pull request #6815: Bump npm from 6.4.1 to 6.13.4 in /airflow/www
ashb merged pull request #6815: Bump npm from 6.4.1 to 6.13.4 in /airflow/www URL: https://github.com/apache/airflow/pull/6815 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on issue #6801: [AIRFLOW-XXX] Drive _cmd config options by env var
ashb commented on issue #6801: [AIRFLOW-XXX] Drive _cmd config options by env var URL: https://github.com/apache/airflow/pull/6801#issuecomment-565734274 This looks like it's causing failures in the rest of the config tests -- can you check on what's going on. You'll likely have to run them locally as the test output doesn't include the full error for some reason. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5646) ADD GcpSpeechToTextRecognizeLongRunningSpeechOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-5646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16996465#comment-16996465 ] ASF GitHub Bot commented on AIRFLOW-5646: - stale[bot] commented on pull request #6318: [AIRFLOW-5646] Add operator of "speech to text" for long running URL: https://github.com/apache/airflow/pull/6318 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > ADD GcpSpeechToTextRecognizeLongRunningSpeechOperator > - > > Key: AIRFLOW-5646 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5646 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, hooks, operators >Affects Versions: 1.10.5 >Reporter: taira yoshiki >Assignee: taira yoshiki >Priority: Minor > > Hello, > I want to add an operator corresponding to the following API. > [https://speech.googleapis.com/v1/speech:longrunningrecognize] > Currently there is only an Operator to call the following API. > [https://speech.googleapis.com/v1/speech:recognize] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] stale[bot] closed pull request #6318: [AIRFLOW-5646] Add operator of "speech to text" for long running
stale[bot] closed pull request #6318: [AIRFLOW-5646] Add operator of "speech to text" for long running URL: https://github.com/apache/airflow/pull/6318 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6799: [AIRFLOW-XXX] KubernetesPodOperator: fix _extract_xcom for larger json files
codecov-io edited a comment on issue #6799: [AIRFLOW-XXX] KubernetesPodOperator: fix _extract_xcom for larger json files URL: https://github.com/apache/airflow/pull/6799#issuecomment-565150765 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6799?src=pr=h1) Report > Merging [#6799](https://codecov.io/gh/apache/airflow/pull/6799?src=pr=desc) into [v1-10-test](https://codecov.io/gh/apache/airflow/commit/11f4f64db7de1123df2abbe18b7b92fb5824e294?src=pr=desc) will **decrease** coverage by `0.07%`. > The diff coverage is `80.38%`. [](https://codecov.io/gh/apache/airflow/pull/6799?src=pr=tree) ```diff @@ Coverage Diff @@ ## v1-10-test#6799 +/- ## == - Coverage81.2% 81.13% -0.08% == Files 523 519 -4 Lines 3548635347 -139 == - Hits2881728678 -139 Misses 6669 6669 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6799?src=pr=tree) | Coverage Δ | | |---|---|---| | [...ow/contrib/example\_dags/example\_qubole\_operator.py](https://codecov.io/gh/apache/airflow/pull/6799/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3F1Ym9sZV9vcGVyYXRvci5weQ==) | `90% <ø> (ø)` | :arrow_up: | | [airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/6799/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5) | `77.94% <ø> (ø)` | :arrow_up: | | [airflow/contrib/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6799/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2t1YmVybmV0ZXMvcG9kX2xhdW5jaGVyLnB5) | `92.48% <ø> (ø)` | :arrow_up: | | [airflow/utils/log/file\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/6799/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZmlsZV90YXNrX2hhbmRsZXIucHk=) | `88.76% <0%> (ø)` | :arrow_up: | | [airflow/serialization/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/6799/diff?src=pr=tree#diff-YWlyZmxvdy9zZXJpYWxpemF0aW9uL19faW5pdF9fLnB5) | `100% <100%> (ø)` | :arrow_up: | | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/6799/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `94.77% <100%> (-0.43%)` | :arrow_down: | | [airflow/www\_rbac/app.py](https://codecov.io/gh/apache/airflow/pull/6799/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy9hcHAucHk=) | `96.58% <100%> (ø)` | :arrow_up: | | [airflow/models/pool.py](https://codecov.io/gh/apache/airflow/pull/6799/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvcG9vbC5weQ==) | `97.36% <100%> (ø)` | :arrow_up: | | [airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/6799/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=) | `100% <100%> (ø)` | :arrow_up: | | [airflow/sentry.py](https://codecov.io/gh/apache/airflow/pull/6799/diff?src=pr=tree#diff-YWlyZmxvdy9zZW50cnkucHk=) | `89.04% <100%> (ø)` | :arrow_up: | | ... and [106 more](https://codecov.io/gh/apache/airflow/pull/6799/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6799?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6799?src=pr=footer). Last update [11f4f64...f2b76e7](https://codecov.io/gh/apache/airflow/pull/6799?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow-site] nuclearpinguin commented on issue #219: Add blog with Airflow Survey results
nuclearpinguin commented on issue #219: Add blog with Airflow Survey results URL: https://github.com/apache/airflow-site/pull/219#issuecomment-565724983 @mik-laj can you please take a look This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6779: [AIRFLOW-6222] http hook should log response text
codecov-io edited a comment on issue #6779: [AIRFLOW-6222] http hook should log response text URL: https://github.com/apache/airflow/pull/6779#issuecomment-565712291 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6779?src=pr=h1) Report > Merging [#6779](https://codecov.io/gh/apache/airflow/pull/6779?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/cfa773828ca5025943fac6eadc90694bb26edc5c?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6779?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6779 +/- ## == - Coverage 84.59% 84.31% -0.29% == Files 676 676 Lines 3833138330 -1 == - Hits3242832318 -110 - Misses 5903 6012 +109 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6779?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/hooks/http\_hook.py](https://codecov.io/gh/apache/airflow/pull/6779/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9odHRwX2hvb2sucHk=) | `95.45% <100%> (-0.07%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6779/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6779/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6779/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6779/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6779/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.2% <0%> (-20.52%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6779/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.59% <0%> (-0.29%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6779/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.8% <0%> (+0.38%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6779?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6779?src=pr=footer). Last update [cfa7738...16bb352](https://codecov.io/gh/apache/airflow/pull/6779?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6779: [AIRFLOW-6222] http hook should log response text
codecov-io commented on issue #6779: [AIRFLOW-6222] http hook should log response text URL: https://github.com/apache/airflow/pull/6779#issuecomment-565712291 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6779?src=pr=h1) Report > Merging [#6779](https://codecov.io/gh/apache/airflow/pull/6779?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/cfa773828ca5025943fac6eadc90694bb26edc5c?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6779?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6779 +/- ## == - Coverage 84.59% 84.31% -0.29% == Files 676 676 Lines 3833138330 -1 == - Hits3242832318 -110 - Misses 5903 6012 +109 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6779?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/hooks/http\_hook.py](https://codecov.io/gh/apache/airflow/pull/6779/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9odHRwX2hvb2sucHk=) | `95.45% <100%> (-0.07%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6779/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6779/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6779/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6779/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6779/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.2% <0%> (-20.52%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6779/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.59% <0%> (-0.29%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6779/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.8% <0%> (+0.38%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6779?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6779?src=pr=footer). Last update [cfa7738...16bb352](https://codecov.io/gh/apache/airflow/pull/6779?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] stale[bot] closed pull request #5779: [AIRLFOW-5163] Default do_xcom_push to False
stale[bot] closed pull request #5779: [AIRLFOW-5163] Default do_xcom_push to False URL: https://github.com/apache/airflow/pull/5779 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6257) Docs: improving Variable sections
[
https://issues.apache.org/jira/browse/AIRFLOW-6257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16996314#comment-16996314
]
ASF GitHub Bot commented on AIRFLOW-6257:
-
SolbiatiAlessandro commented on pull request #6819: [AIRFLOW-6257] Docs:
improving Variable sections
URL: https://github.com/apache/airflow/pull/6819
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Docs: improving Variable sections
> -
>
> Key: AIRFLOW-6257
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6257
> Project: Apache Airflow
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 1.10.6
>Reporter: Alessandro Solbiati
>Assignee: Alessandro Solbiati
>Priority: Major
>
> Adding details on how to set variables in code, as it's often a question
> found on SOF and not answered in Airflow docs
> [https://stackoverflow.com/questions/54045048/with-code-how-do-you-update-an-airflow-variable]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] SolbiatiAlessandro opened a new pull request #6819: [AIRFLOW-6257] Docs: improving Variable sections
SolbiatiAlessandro opened a new pull request #6819: [AIRFLOW-6257] Docs:
improving Variable sections
URL: https://github.com/apache/airflow/pull/6819
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-6257) Docs: improving Variable sections
Alessandro Solbiati created AIRFLOW-6257: Summary: Docs: improving Variable sections Key: AIRFLOW-6257 URL: https://issues.apache.org/jira/browse/AIRFLOW-6257 Project: Apache Airflow Issue Type: Improvement Components: documentation Affects Versions: 1.10.6 Reporter: Alessandro Solbiati Adding details on how to set variables in code, as it's often a question found on SOF and not answered in Airflow docs [https://stackoverflow.com/questions/54045048/with-code-how-do-you-update-an-airflow-variable] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-6257) Docs: improving Variable sections
[ https://issues.apache.org/jira/browse/AIRFLOW-6257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alessandro Solbiati reassigned AIRFLOW-6257: Assignee: Alessandro Solbiati > Docs: improving Variable sections > - > > Key: AIRFLOW-6257 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6257 > Project: Apache Airflow > Issue Type: Improvement > Components: documentation >Affects Versions: 1.10.6 >Reporter: Alessandro Solbiati >Assignee: Alessandro Solbiati >Priority: Major > > Adding details on how to set variables in code, as it's often a question > found on SOF and not answered in Airflow docs > [https://stackoverflow.com/questions/54045048/with-code-how-do-you-update-an-airflow-variable] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] mik-laj commented on issue #6818: [AIRFLOW-6256] BaseJob was not imported hence not deleted on reset
mik-laj commented on issue #6818: [AIRFLOW-6256] BaseJob was not imported hence not deleted on reset URL: https://github.com/apache/airflow/pull/6818#issuecomment-565702958 Can you provide steps to reproduce the problem? The current issues and PR descriptions are not very descriptive. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6818: [AIRFLOW-6256] BaseJob was not imported hence not deleted on reset
codecov-io commented on issue #6818: [AIRFLOW-6256] BaseJob was not imported hence not deleted on reset URL: https://github.com/apache/airflow/pull/6818#issuecomment-565702528 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6818?src=pr=h1) Report > Merging [#6818](https://codecov.io/gh/apache/airflow/pull/6818?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/27785f47bae1f0336dbae21fa5718b3f309d2e9e?src=pr=desc) will **decrease** coverage by `0.29%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6818?src=pr=tree) ```diff @@Coverage Diff@@ ## master#6818 +/- ## = - Coverage84.6% 84.31% -0.3% = Files 676 676 Lines 3832938330 +1 = - Hits3243032318-112 - Misses 5899 6012+113 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6818?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/6818/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `100% <100%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6818/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6818/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6818/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6818/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6818/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.2% <0%> (-20.52%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6818/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.42% <0%> (-0.58%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6818?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6818?src=pr=footer). Last update [27785f4...8c7d25d](https://codecov.io/gh/apache/airflow/pull/6818?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6818: [AIRFLOW-6256] BaseJob was not imported hence not deleted on reset
codecov-io edited a comment on issue #6818: [AIRFLOW-6256] BaseJob was not imported hence not deleted on reset URL: https://github.com/apache/airflow/pull/6818#issuecomment-565702528 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6818?src=pr=h1) Report > Merging [#6818](https://codecov.io/gh/apache/airflow/pull/6818?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/27785f47bae1f0336dbae21fa5718b3f309d2e9e?src=pr=desc) will **decrease** coverage by `0.29%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6818?src=pr=tree) ```diff @@Coverage Diff@@ ## master#6818 +/- ## = - Coverage84.6% 84.31% -0.3% = Files 676 676 Lines 3832938330 +1 = - Hits3243032318-112 - Misses 5899 6012+113 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6818?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/6818/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `100% <100%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6818/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6818/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6818/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6818/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6818/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.2% <0%> (-20.52%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6818/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.42% <0%> (-0.58%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6818?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6818?src=pr=footer). Last update [27785f4...8c7d25d](https://codecov.io/gh/apache/airflow/pull/6818?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6376: [WIP] [AIRFLOW-5705] Add creds backend classes including AWS SSM
codecov-io edited a comment on issue #6376: [WIP] [AIRFLOW-5705] Add creds backend classes including AWS SSM URL: https://github.com/apache/airflow/pull/6376#issuecomment-544833958 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6376?src=pr=h1) Report > Merging [#6376](https://codecov.io/gh/apache/airflow/pull/6376?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/5b461d626a6f352c0c3031e7b793d56b1f7260fa?src=pr=desc) will **decrease** coverage by `0.3%`. > The diff coverage is `84.09%`. [](https://codecov.io/gh/apache/airflow/pull/6376?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6376 +/- ## == - Coverage 84.59% 84.29% -0.31% == Files 676 680 +4 Lines 3831838485 +167 == + Hits3241732441 +24 - Misses 5901 6044 +143 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6376?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6376/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `93.23% <100%> (+0.12%)` | :arrow_up: | | [airflow/creds/env\_vars.py](https://codecov.io/gh/apache/airflow/pull/6376/diff?src=pr=tree#diff-YWlyZmxvdy9jcmVkcy9lbnZfdmFycy5weQ==) | `100% <100%> (ø)` | | | [airflow/hooks/base\_hook.py](https://codecov.io/gh/apache/airflow/pull/6376/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9iYXNlX2hvb2sucHk=) | `86.2% <100%> (-5.8%)` | :arrow_down: | | [airflow/creds/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/6376/diff?src=pr=tree#diff-YWlyZmxvdy9jcmVkcy9fX2luaXRfXy5weQ==) | `100% <100%> (ø)` | | | [airflow/creds/metastore.py](https://codecov.io/gh/apache/airflow/pull/6376/diff?src=pr=tree#diff-YWlyZmxvdy9jcmVkcy9tZXRhc3RvcmUucHk=) | `100% <100%> (ø)` | | | [airflow/creds/aws\_ssm.py](https://codecov.io/gh/apache/airflow/pull/6376/diff?src=pr=tree#diff-YWlyZmxvdy9jcmVkcy9hd3Nfc3NtLnB5) | `63.15% <63.15%> (ø)` | | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6376/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6376/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6376/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6376/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | ... and [7 more](https://codecov.io/gh/apache/airflow/pull/6376/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6376?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6376?src=pr=footer). Last update [5b461d6...e0529d8](https://codecov.io/gh/apache/airflow/pull/6376?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #6817: [AIRFLOW-6254] obscure conn extra in logs
potiuk commented on issue #6817: [AIRFLOW-6254] obscure conn extra in logs URL: https://github.com/apache/airflow/pull/6817#issuecomment-565702196 Ah. I haven't seen your comment @ashb and merged it first. I think it's dangerous to try to find the fields inside. Each connection has its own private schema and for the sake of security I think it's better to obfuscate it all. There is low risk with not showing it - user can always check connection table or variable if you are not sure if all is ok. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6226) warning.catch_warning should not be used in our code
[ https://issues.apache.org/jira/browse/AIRFLOW-6226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16996263#comment-16996263 ] ASF subversion and git services commented on AIRFLOW-6226: -- Commit cfa773828ca5025943fac6eadc90694bb26edc5c in airflow's branch refs/heads/master from Jarek Potiuk [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=cfa7738 ] [AIRFLOW-6226] Always reset warnings in tests (#6785) > warning.catch_warning should not be used in our code > - > > Key: AIRFLOW-6226 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6226 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0, 1.10.6 >Reporter: Jarek Potiuk >Priority: Major > > Sometime we use warning.catch_warnings in our code directly. > As explained in [https://blog.ionelmc.ro/2013/06/26/testing-python-warnings/] > warnings are cached in "__warningregistry__" and if warning is emitted, it is > not emited for the second time. > > Therefore warning.catch_warnings should never be used directly in our test > code. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6226) warning.catch_warning should not be used in our code
[ https://issues.apache.org/jira/browse/AIRFLOW-6226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-6226. --- Fix Version/s: 1.10.7 Resolution: Fixed > warning.catch_warning should not be used in our code > - > > Key: AIRFLOW-6226 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6226 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0, 1.10.6 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 1.10.7 > > > Sometime we use warning.catch_warnings in our code directly. > As explained in [https://blog.ionelmc.ro/2013/06/26/testing-python-warnings/] > warnings are cached in "__warningregistry__" and if warning is emitted, it is > not emited for the second time. > > Therefore warning.catch_warnings should never be used directly in our test > code. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6226) warning.catch_warning should not be used in our code
[ https://issues.apache.org/jira/browse/AIRFLOW-6226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16996262#comment-16996262 ] ASF GitHub Bot commented on AIRFLOW-6226: - potiuk commented on pull request #6785: [AIRFLOW-6226] Always reset warnings in tests URL: https://github.com/apache/airflow/pull/6785 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > warning.catch_warning should not be used in our code > - > > Key: AIRFLOW-6226 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6226 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0, 1.10.6 >Reporter: Jarek Potiuk >Priority: Major > > Sometime we use warning.catch_warnings in our code directly. > As explained in [https://blog.ionelmc.ro/2013/06/26/testing-python-warnings/] > warnings are cached in "__warningregistry__" and if warning is emitted, it is > not emited for the second time. > > Therefore warning.catch_warnings should never be used directly in our test > code. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk merged pull request #6785: [AIRFLOW-6226] Always reset warnings in tests
potiuk merged pull request #6785: [AIRFLOW-6226] Always reset warnings in tests URL: https://github.com/apache/airflow/pull/6785 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-6091) BigQueryCursor doesn't properly clean buffer, page_token and all_pages_loaded before starting .execute() method
[ https://issues.apache.org/jira/browse/AIRFLOW-6091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-6091. --- Fix Version/s: 1.10.7 Resolution: Fixed Also cherry-picked to 1.10.7 > BigQueryCursor doesn't properly clean buffer, page_token and all_pages_loaded > before starting .execute() method > --- > > Key: AIRFLOW-6091 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6091 > Project: Apache Airflow > Issue Type: Bug > Components: hooks >Affects Versions: 1.10.6 >Reporter: Tomasz Żukowski >Assignee: Tomasz Żukowski >Priority: Major > Fix For: 1.10.7 > > > There are two cases when it might be dangerous: > * If you run cursor.execute() and cursor.fetchall() and then once again > cursor.execute() for another query cursor.fetchall() will return empty list > because will be still set to True > * If you run cursor.execute() and cursor.fetchone() or fetchmany() and don't > read whole results and then run cursor.execute() with different query next > call to next will return data from old job which will be still stored in > buffer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-4824) MySqlHook needs to override DbApiHook.get_uri to pull in extra for charset=utf-8 during create_engine
[ https://issues.apache.org/jira/browse/AIRFLOW-4824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-4824. --- Fix Version/s: 1.10.7 Resolution: Fixed > MySqlHook needs to override DbApiHook.get_uri to pull in extra for > charset=utf-8 during create_engine > - > > Key: AIRFLOW-4824 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4824 > Project: Apache Airflow > Issue Type: Bug > Components: hooks >Affects Versions: 1.10.3 >Reporter: lolcode >Assignee: Tomasz Żukowski >Priority: Minor > Fix For: 1.10.7 > > > When using the engine from a MySQLHook in other code (such as Pandas) the > engine returned from the create_engine function is missing the charset=utf-8 > This issue was reported here: > [https://stackoverflow.com/questions/46084744/how-to-explicitly-declare-charset-utf8-for-airflow-connections] > conn = MySqlHook(mysql_conn_id='conn_id') > engine = conn.get_sqlalchemy_engine() > I can see that the code in function *get_uri* in dbapi_hook.py does not use > the charset = utf8 information from the extra section and that mysql_hook.py > does not override the function. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4824) MySqlHook needs to override DbApiHook.get_uri to pull in extra for charset=utf-8 during create_engine
[ https://issues.apache.org/jira/browse/AIRFLOW-4824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16996260#comment-16996260 ] Jarek Potiuk commented on AIRFLOW-4824: --- Cherry-picked to 1.10.7 :) > MySqlHook needs to override DbApiHook.get_uri to pull in extra for > charset=utf-8 during create_engine > - > > Key: AIRFLOW-4824 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4824 > Project: Apache Airflow > Issue Type: Bug > Components: hooks >Affects Versions: 1.10.3 >Reporter: lolcode >Assignee: Tomasz Żukowski >Priority: Minor > Fix For: 1.10.7 > > > When using the engine from a MySQLHook in other code (such as Pandas) the > engine returned from the create_engine function is missing the charset=utf-8 > This issue was reported here: > [https://stackoverflow.com/questions/46084744/how-to-explicitly-declare-charset-utf8-for-airflow-connections] > conn = MySqlHook(mysql_conn_id='conn_id') > engine = conn.get_sqlalchemy_engine() > I can see that the code in function *get_uri* in dbapi_hook.py does not use > the charset = utf8 information from the extra section and that mysql_hook.py > does not override the function. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6254) obscure conn extra in logs
[
https://issues.apache.org/jira/browse/AIRFLOW-6254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16996258#comment-16996258
]
ASF subversion and git services commented on AIRFLOW-6254:
--
Commit fdfdeeedba61f6b658436777a50d6130256abdd5 in airflow's branch
refs/heads/v1-10-test from dstandish
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=fdfdeee ]
[AIRFLOW-6254] obscure conn extra in logs (#6817)
(cherry picked from commit c6425a4be9c17bdcb4cbb134c3f19f1cff1c2e38)
> obscure conn extra in logs
> --
>
> Key: AIRFLOW-6254
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6254
> Project: Apache Airflow
> Issue Type: Bug
> Components: core
>Affects Versions: 1.10.6
>Reporter: Daniel Standish
>Assignee: Daniel Standish
>Priority: Major
> Fix For: 1.10.7
>
>
> When {{BaseHook.get_connection}} is called, it calls {{conn.log_info()}} on
> the returned {{conn}} object.
> This is prints to log the full contents of {{conn.extra}}.
> This is problematic because there can be sensitive information in
> {{conn.extra}}.
> The present change resolves this by adding method {{conn.log_info}} which
> obscures {{extra}}, and calling that in {{get_connection}} instead of
> {{debug_info}}.
> The {{debug_info}} method itself is left unchanged.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6254) obscure conn extra in logs
[
https://issues.apache.org/jira/browse/AIRFLOW-6254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jarek Potiuk resolved AIRFLOW-6254.
---
Resolution: Fixed
> obscure conn extra in logs
> --
>
> Key: AIRFLOW-6254
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6254
> Project: Apache Airflow
> Issue Type: Bug
> Components: core
>Affects Versions: 1.10.6
>Reporter: Daniel Standish
>Assignee: Daniel Standish
>Priority: Major
> Fix For: 1.10.7
>
>
> When {{BaseHook.get_connection}} is called, it calls {{conn.log_info()}} on
> the returned {{conn}} object.
> This is prints to log the full contents of {{conn.extra}}.
> This is problematic because there can be sensitive information in
> {{conn.extra}}.
> The present change resolves this by adding method {{conn.log_info}} which
> obscures {{extra}}, and calling that in {{get_connection}} instead of
> {{debug_info}}.
> The {{debug_info}} method itself is left unchanged.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6254) obscure conn extra in logs
[
https://issues.apache.org/jira/browse/AIRFLOW-6254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16996257#comment-16996257
]
ASF subversion and git services commented on AIRFLOW-6254:
--
Commit c6425a4be9c17bdcb4cbb134c3f19f1cff1c2e38 in airflow's branch
refs/heads/master from dstandish
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=c6425a4 ]
[AIRFLOW-6254] obscure conn extra in logs (#6817)
> obscure conn extra in logs
> --
>
> Key: AIRFLOW-6254
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6254
> Project: Apache Airflow
> Issue Type: Bug
> Components: core
>Affects Versions: 1.10.6
>Reporter: Daniel Standish
>Assignee: Daniel Standish
>Priority: Major
> Fix For: 1.10.7
>
>
> When {{BaseHook.get_connection}} is called, it calls {{conn.log_info()}} on
> the returned {{conn}} object.
> This is prints to log the full contents of {{conn.extra}}.
> This is problematic because there can be sensitive information in
> {{conn.extra}}.
> The present change resolves this by adding method {{conn.log_info}} which
> obscures {{extra}}, and calling that in {{get_connection}} instead of
> {{debug_info}}.
> The {{debug_info}} method itself is left unchanged.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4824) MySqlHook needs to override DbApiHook.get_uri to pull in extra for charset=utf-8 during create_engine
[ https://issues.apache.org/jira/browse/AIRFLOW-4824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16996255#comment-16996255 ] ASF subversion and git services commented on AIRFLOW-4824: -- Commit 21e9cadbe4e2681950d60abaf312bf1ffd835b79 in airflow's branch refs/heads/v1-10-test from zuku1985 [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=21e9cad ] [AIRFLOW-4824] Add charset handling for SqlAlchemy engine for MySqlHook (#6816) Airflow should handle various charsets for connections with MySQL dbs. This change allows to set charset in extra field of a connection when using SqlAlchemy engine. (cherry picked from commit 27785f47bae1f0336dbae21fa5718b3f309d2e9e) > MySqlHook needs to override DbApiHook.get_uri to pull in extra for > charset=utf-8 during create_engine > - > > Key: AIRFLOW-4824 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4824 > Project: Apache Airflow > Issue Type: Bug > Components: hooks >Affects Versions: 1.10.3 >Reporter: lolcode >Assignee: Tomasz Żukowski >Priority: Minor > > When using the engine from a MySQLHook in other code (such as Pandas) the > engine returned from the create_engine function is missing the charset=utf-8 > This issue was reported here: > [https://stackoverflow.com/questions/46084744/how-to-explicitly-declare-charset-utf8-for-airflow-connections] > conn = MySqlHook(mysql_conn_id='conn_id') > engine = conn.get_sqlalchemy_engine() > I can see that the code in function *get_uri* in dbapi_hook.py does not use > the charset = utf8 information from the extra section and that mysql_hook.py > does not override the function. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6091) BigQueryCursor doesn't properly clean buffer, page_token and all_pages_loaded before starting .execute() method
[ https://issues.apache.org/jira/browse/AIRFLOW-6091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16996254#comment-16996254 ] ASF subversion and git services commented on AIRFLOW-6091: -- Commit ee7d311867897feec490fcd2f9e1eb040c9db078 in airflow's branch refs/heads/v1-10-test from zuku1985 [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=ee7d311 ] [AIRFLOW-6091] Add flushing in execute method for BigQueryCursor (#6683) If you execute multiple queries results of old ones will be flushed allowing to read results of recent execute without any issues. (cherry picked from commit 0cf9598b0885cb94eb89c5eca8717f2245cb5de7) > BigQueryCursor doesn't properly clean buffer, page_token and all_pages_loaded > before starting .execute() method > --- > > Key: AIRFLOW-6091 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6091 > Project: Apache Airflow > Issue Type: Bug > Components: hooks >Affects Versions: 1.10.6 >Reporter: Tomasz Żukowski >Assignee: Tomasz Żukowski >Priority: Major > > There are two cases when it might be dangerous: > * If you run cursor.execute() and cursor.fetchall() and then once again > cursor.execute() for another query cursor.fetchall() will return empty list > because will be still set to True > * If you run cursor.execute() and cursor.fetchone() or fetchmany() and don't > read whole results and then run cursor.execute() with different query next > call to next will return data from old job which will be still stored in > buffer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk merged pull request #6817: [AIRFLOW-6254] obscure conn extra in logs
potiuk merged pull request #6817: [AIRFLOW-6254] obscure conn extra in logs URL: https://github.com/apache/airflow/pull/6817 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6254) obscure conn extra in logs
[
https://issues.apache.org/jira/browse/AIRFLOW-6254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16996256#comment-16996256
]
ASF GitHub Bot commented on AIRFLOW-6254:
-
potiuk commented on pull request #6817: [AIRFLOW-6254] obscure conn extra in
logs
URL: https://github.com/apache/airflow/pull/6817
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> obscure conn extra in logs
> --
>
> Key: AIRFLOW-6254
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6254
> Project: Apache Airflow
> Issue Type: Bug
> Components: core
>Affects Versions: 1.10.6
>Reporter: Daniel Standish
>Assignee: Daniel Standish
>Priority: Major
> Fix For: 1.10.7
>
>
> When {{BaseHook.get_connection}} is called, it calls {{conn.log_info()}} on
> the returned {{conn}} object.
> This is prints to log the full contents of {{conn.extra}}.
> This is problematic because there can be sensitive information in
> {{conn.extra}}.
> The present change resolves this by adding method {{conn.log_info}} which
> obscures {{extra}}, and calling that in {{get_connection}} instead of
> {{debug_info}}.
> The {{debug_info}} method itself is left unchanged.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6256) BaseJob table is not removed when clearing the DB
[ https://issues.apache.org/jira/browse/AIRFLOW-6256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16996253#comment-16996253 ] ASF subversion and git services commented on AIRFLOW-6256: -- Commit d68165612c60acbba246cc1f38459af5de38c6aa in airflow's branch refs/heads/v1-10-test from Jarek Potiuk [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=d681656 ] [AIRFLOW-6256] BaseJob was not imported hence not deleted on reset (cherry picked from commit 8c7d25d90c5efb988723ae0ab6e0b1965f1782a1) > BaseJob table is not removed when clearing the DB > - > > Key: AIRFLOW-6256 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6256 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 1.10.7 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk opened a new pull request #6818: [AIRFLOW-6256] BaseJob was not imported hence not deleted on reset
potiuk opened a new pull request #6818: [AIRFLOW-6256] BaseJob was not imported
hence not deleted on reset
URL: https://github.com/apache/airflow/pull/6818
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-6256
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-6256) BaseJob table is not removed when clearing the DB
[
https://issues.apache.org/jira/browse/AIRFLOW-6256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16996244#comment-16996244
]
ASF GitHub Bot commented on AIRFLOW-6256:
-
potiuk commented on pull request #6818: [AIRFLOW-6256] BaseJob was not imported
hence not deleted on reset
URL: https://github.com/apache/airflow/pull/6818
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-6256
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> BaseJob table is not removed when clearing the DB
> -
>
> Key: AIRFLOW-6256
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6256
> Project: Apache Airflow
> Issue Type: Bug
> Components: ci
>Affects Versions: 2.0.0
>Reporter: Jarek Potiuk
>Priority: Major
> Fix For: 1.10.7
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6256) BaseJob table is not removed when clearing the DB
[ https://issues.apache.org/jira/browse/AIRFLOW-6256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-6256: -- Affects Version/s: (was: 1.10.6) 2.0.0 > BaseJob table is not removed when clearing the DB > - > > Key: AIRFLOW-6256 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6256 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 1.10.7 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6256) BaseJob table is not removed when clearing the DB
Jarek Potiuk created AIRFLOW-6256: - Summary: BaseJob table is not removed when clearing the DB Key: AIRFLOW-6256 URL: https://issues.apache.org/jira/browse/AIRFLOW-6256 Project: Apache Airflow Issue Type: Bug Components: ci Affects Versions: 1.10.6 Reporter: Jarek Potiuk Fix For: 1.10.7 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io commented on issue #6785: [AIRFLOW-6226] Always reset warnings in tests
codecov-io commented on issue #6785: [AIRFLOW-6226] Always reset warnings in tests URL: https://github.com/apache/airflow/pull/6785#issuecomment-565695226 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6785?src=pr=h1) Report > Merging [#6785](https://codecov.io/gh/apache/airflow/pull/6785?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/5b461d626a6f352c0c3031e7b793d56b1f7260fa?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6785?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6785 +/- ## == - Coverage 84.59% 84.31% -0.29% == Files 676 676 Lines 3831838318 == - Hits3241732307 -110 - Misses 5901 6011 +110 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6785?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6785/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6785/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6785/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6785/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6785/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.2% <0%> (-20.52%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6785?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6785?src=pr=footer). Last update [5b461d6...7860d63](https://codecov.io/gh/apache/airflow/pull/6785?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-2325) Task logging with AWS Cloud watch
[ https://issues.apache.org/jira/browse/AIRFLOW-2325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16996233#comment-16996233 ] ASF GitHub Bot commented on AIRFLOW-2325: - stale[bot] commented on pull request #3229: [AIRFLOW-2325] Add cloudwatch task handler (IN PROGRESS) URL: https://github.com/apache/airflow/pull/3229 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Task logging with AWS Cloud watch > - > > Key: AIRFLOW-2325 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2325 > Project: Apache Airflow > Issue Type: New Feature > Components: logging >Reporter: Fang-Pen Lin >Priority: Minor > > In many cases, it's ideal to use remote logging while running Airflow in > production, as the worker could be easily scale down or scale up. Or the > worker is running in containers, where the local storage is not meant to be > there forever. In that case, the S3 task logging handler could be used > [https://github.com/apache/incubator-airflow/blob/master/airflow/utils/log/s3_task_handler.py] > However, it comes with drawback. S3 logging handler only uploads the log when > the task completed or failed. For long running tasks, it's hard to know > what's going on with the process until it finishes. > To make more real-time logging, I built a logging handler based on AWS > CloudWatch. It uses a third party python package `watchtower` > > [https://github.com/kislyuk/watchtower/tree/master/watchtower] > > I created a PR here [https://github.com/apache/incubator-airflow/pull/3229], > basically I just copy-pasted the code I wrote for my own project, it works > fine with 1.9 release, but never tested with master branch. Also, there is a > bug in watchtower causing task runner to hang forever when it completes. I > created an issue in their repo > [https://github.com/kislyuk/watchtower/issues/57] > And a PR for addressing that issue > [https://github.com/kislyuk/watchtower/pull/58] > > The PR is still far from ready to be reviewed, but I just want to get some > feedback before I spend more time on it. I would like to see if youguys want > this cloudwatch handler goes into the main repo, or do youguys prefer it to > be a standalone third-party module. If it's that case, I can close this > ticket and create a standalone repo on my own. If the PR is welcome, then I > can spend more time on polishing it based on your feedback, add tests / > documents and other stuff. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] stale[bot] closed pull request #3229: [AIRFLOW-2325] Add cloudwatch task handler (IN PROGRESS)
stale[bot] closed pull request #3229: [AIRFLOW-2325] Add cloudwatch task handler (IN PROGRESS) URL: https://github.com/apache/airflow/pull/3229 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
