[GitHub] [airflow] nanco2sensei closed pull request #6560: [AIRFLOW-XXX] add description for externally triggered DAGs to rest-api-ref.rst
nanco2sensei closed pull request #6560: [AIRFLOW-XXX] add description for externally triggered DAGs to rest-api-ref.rst URL: https://github.com/apache/airflow/pull/6560 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6582: [AIRFLOW-5928] Hive hooks load_file short circuit
codecov-io edited a comment on issue #6582: [AIRFLOW-5928] Hive hooks load_file short circuit URL: https://github.com/apache/airflow/pull/6582#issuecomment-553762142 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6582?src=pr=h1) Report > Merging [#6582](https://codecov.io/gh/apache/airflow/pull/6582?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/4c9198017d8bc2df85bd4e8cdeac770c0c2181e2?src=pr=desc) will **decrease** coverage by `0.32%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6582?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6582 +/- ## == - Coverage 83.83% 83.51% -0.33% == Files 645 645 Lines 3735837358 == - Hits3131831198 -120 - Misses 6040 6160 +120 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6582?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/6582/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5) | `77.6% <100%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6582/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6582/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6582/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6582/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6582/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `77.14% <0%> (-21.43%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6582/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (-3.63%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6582/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `58.15% <0%> (-0.17%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6582?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6582?src=pr=footer). Last update [4c91980...c2829e9](https://codecov.io/gh/apache/airflow/pull/6582?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6582: [AIRFLOW-5928] Hive hooks load_file short circuit
codecov-io commented on issue #6582: [AIRFLOW-5928] Hive hooks load_file short circuit URL: https://github.com/apache/airflow/pull/6582#issuecomment-553762142 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6582?src=pr=h1) Report > Merging [#6582](https://codecov.io/gh/apache/airflow/pull/6582?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/4c9198017d8bc2df85bd4e8cdeac770c0c2181e2?src=pr=desc) will **decrease** coverage by `0.32%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6582?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6582 +/- ## == - Coverage 83.83% 83.51% -0.33% == Files 645 645 Lines 3735837358 == - Hits3131831198 -120 - Misses 6040 6160 +120 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6582?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/6582/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5) | `77.6% <100%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6582/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6582/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6582/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6582/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6582/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `77.14% <0%> (-21.43%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6582/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (-3.63%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6582/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `58.15% <0%> (-0.17%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6582?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6582?src=pr=footer). Last update [4c91980...c2829e9](https://codecov.io/gh/apache/airflow/pull/6582?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5928) hive hooks load_file short circuit
[
https://issues.apache.org/jira/browse/AIRFLOW-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973958#comment-16973958
]
ASF GitHub Bot commented on AIRFLOW-5928:
-
zhongjiajie commented on pull request #6582: [AIRFLOW-5928] Hive hooks
load_file short circuit
URL: https://github.com/apache/airflow/pull/6582
If function load_file with parameter create and recreate are
set to False, hql = '' and should not call function
HiveCliHook.run_cli
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5928
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
If function load_file with parameter create and recreate are
set to False, hql = '' and should not call function
HiveCliHook.run_cli
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> hive hooks load_file short circuit
> --
>
> Key: AIRFLOW-5928
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5928
> Project: Apache Airflow
> Issue Type: Improvement
> Components: hooks
>Affects Versions: 1.10.6
>Reporter: zhongjiajie
>Assignee: zhongjiajie
>Priority: Major
>
> If function `load_file` with parameter `create` and `recreate` are `False`,
> `hql = ''` and should not call `HiveCliHook.run_cli`
> Due to `recreate` in two `if` statement, `HiveCliHook.run_cli` only need to
> in the last one.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] zhongjiajie opened a new pull request #6582: [AIRFLOW-5928] Hive hooks load_file short circuit
zhongjiajie opened a new pull request #6582: [AIRFLOW-5928] Hive hooks
load_file short circuit
URL: https://github.com/apache/airflow/pull/6582
If function load_file with parameter create and recreate are
set to False, hql = '' and should not call function
HiveCliHook.run_cli
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5928
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
If function load_file with parameter create and recreate are
set to False, hql = '' and should not call function
HiveCliHook.run_cli
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-5928) hive hooks load_file short circuit
zhongjiajie created AIRFLOW-5928: Summary: hive hooks load_file short circuit Key: AIRFLOW-5928 URL: https://issues.apache.org/jira/browse/AIRFLOW-5928 Project: Apache Airflow Issue Type: Improvement Components: hooks Affects Versions: 1.10.6 Reporter: zhongjiajie Assignee: zhongjiajie If function `load_file` with parameter `create` and `recreate` are `False`, `hql = ''` and should not call `HiveCliHook.run_cli` Due to `recreate` in two `if` statement, `HiveCliHook.run_cli` only need to in the last one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #6567: [AIRFLOW-5918][part of AIRFLOW-5893] Group tests for the Pools command
codecov-io edited a comment on issue #6567: [AIRFLOW-5918][part of AIRFLOW-5893] Group tests for the Pools command URL: https://github.com/apache/airflow/pull/6567#issuecomment-553704787 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6567?src=pr=h1) Report > Merging [#6567](https://codecov.io/gh/apache/airflow/pull/6567?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/2897624a510361948bb9e261dd2da7ab74271312?src=pr=desc) will **decrease** coverage by `0.12%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6567?src=pr=tree) ```diff @@Coverage Diff@@ ## master #6567 +/- ## = - Coverage 83.63% 83.5% -0.13% = Files 645 645 Lines 37291 37358 +67 = + Hits31187 31197 +10 - Misses 61046161 +57 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6567?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6567/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6567/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6567/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6567/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6567/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `77.14% <0%> (-21.43%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6567/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (-3.63%)` | :arrow_down: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6567/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.28% <0%> (-0.51%)` | :arrow_down: | | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/6567/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `90.85% <0%> (-0.02%)` | :arrow_down: | | [airflow/exceptions.py](https://codecov.io/gh/apache/airflow/pull/6567/diff?src=pr=tree#diff-YWlyZmxvdy9leGNlcHRpb25zLnB5) | `100% <0%> (ø)` | :arrow_up: | | [airflow/contrib/hooks/gcp\_dataproc\_hook.py](https://codecov.io/gh/apache/airflow/pull/6567/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djcF9kYXRhcHJvY19ob29rLnB5) | `100% <0%> (ø)` | :arrow_up: | | ... and [9 more](https://codecov.io/gh/apache/airflow/pull/6567/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6567?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6567?src=pr=footer). Last update [2897624...e2c72a9](https://codecov.io/gh/apache/airflow/pull/6567?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6567: [AIRFLOW-5918][part of AIRFLOW-5893] Group tests for the Pools command
codecov-io edited a comment on issue #6567: [AIRFLOW-5918][part of AIRFLOW-5893] Group tests for the Pools command URL: https://github.com/apache/airflow/pull/6567#issuecomment-553704787 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6567?src=pr=h1) Report > Merging [#6567](https://codecov.io/gh/apache/airflow/pull/6567?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/2897624a510361948bb9e261dd2da7ab74271312?src=pr=desc) will **decrease** coverage by `0.12%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6567?src=pr=tree) ```diff @@Coverage Diff@@ ## master #6567 +/- ## = - Coverage 83.63% 83.5% -0.13% = Files 645 645 Lines 37291 37358 +67 = + Hits31187 31197 +10 - Misses 61046161 +57 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6567?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6567/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6567/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6567/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6567/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6567/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `77.14% <0%> (-21.43%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6567/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (-3.63%)` | :arrow_down: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6567/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.28% <0%> (-0.51%)` | :arrow_down: | | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/6567/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `90.85% <0%> (-0.02%)` | :arrow_down: | | [airflow/exceptions.py](https://codecov.io/gh/apache/airflow/pull/6567/diff?src=pr=tree#diff-YWlyZmxvdy9leGNlcHRpb25zLnB5) | `100% <0%> (ø)` | :arrow_up: | | [airflow/contrib/hooks/gcp\_dataproc\_hook.py](https://codecov.io/gh/apache/airflow/pull/6567/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djcF9kYXRhcHJvY19ob29rLnB5) | `100% <0%> (ø)` | :arrow_up: | | ... and [9 more](https://codecov.io/gh/apache/airflow/pull/6567/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6567?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6567?src=pr=footer). Last update [2897624...e2c72a9](https://codecov.io/gh/apache/airflow/pull/6567?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6567: [AIRFLOW-5918][part of AIRFLOW-5893] Group tests for the Pools command
codecov-io commented on issue #6567: [AIRFLOW-5918][part of AIRFLOW-5893] Group tests for the Pools command URL: https://github.com/apache/airflow/pull/6567#issuecomment-553704787 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6567?src=pr=h1) Report > Merging [#6567](https://codecov.io/gh/apache/airflow/pull/6567?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/2897624a510361948bb9e261dd2da7ab74271312?src=pr=desc) will **decrease** coverage by `0.12%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6567?src=pr=tree) ```diff @@Coverage Diff@@ ## master #6567 +/- ## = - Coverage 83.63% 83.5% -0.13% = Files 645 645 Lines 37291 37358 +67 = + Hits31187 31197 +10 - Misses 61046161 +57 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6567?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6567/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6567/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6567/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6567/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6567/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `77.14% <0%> (-21.43%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6567/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (-3.63%)` | :arrow_down: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6567/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.28% <0%> (-0.51%)` | :arrow_down: | | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/6567/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `90.85% <0%> (-0.02%)` | :arrow_down: | | [airflow/exceptions.py](https://codecov.io/gh/apache/airflow/pull/6567/diff?src=pr=tree#diff-YWlyZmxvdy9leGNlcHRpb25zLnB5) | `100% <0%> (ø)` | :arrow_up: | | [airflow/contrib/hooks/gcp\_dataproc\_hook.py](https://codecov.io/gh/apache/airflow/pull/6567/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djcF9kYXRhcHJvY19ob29rLnB5) | `100% <0%> (ø)` | :arrow_up: | | ... and [9 more](https://codecov.io/gh/apache/airflow/pull/6567/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6567?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6567?src=pr=footer). Last update [2897624...e2c72a9](https://codecov.io/gh/apache/airflow/pull/6567?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-5927) Airflow cache import file or variable
[
https://issues.apache.org/jira/browse/AIRFLOW-5927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kasim updated AIRFLOW-5927:
---
Description:
I have a `config.py` pull configure from `Variable` and merge into default
config :
{code:java}
from datetime import datetime
from airflow.models import Variable
class Config:
version = "V21"
etl_feature_dir = f'/data/dm/sales_forecast/features/version={version}'
forecast_result_dir =
f'/data/dm/sales_forecast/results/fbprophet/version={version}'
forecast_model_dir =
f'/data/dm/sales_forecast/models/fbprophet/version={version}'
forecast_result_s3_dir =
f's3a://x/data/dm/sales_forecast/fbprophet/version={version}'
etl_dir = '/data/dm/sales_forecast/etl'
feature_current_path = 'hdfs://pupuxdc/test/MERGE_OUT'
dag_start_date = datetime(2019, 10, 25)
etl_start_time = "2019-06-01 00:00:00"
etl_end_time = " (execution_date +
macros.timedelta(days=8)).strftime('%Y-%m-%d 00:00:00') "
train_start_time = " (execution_date
macros.timedelta(days=90)).strftime('%Y%m-%d 00:00:00') "
train_end_time = " execution_date.strftime('%Y-%m-%d 00:00:00') "
predict_start_time = " execution_date.strftime('%Y-%m-%d 00:00:00') "
predict_end_time = " (execution_date +
macros.timedelta(days=7)).strftime('%Y-%m-%d 00:00:00') "
report_start_date = " (execution_date
macros.timedelta(days=6)).strftime('%Y%m-%d 00:00:00') "
report_end_date = " execution_date.strftime('%Y-%m-%d 00:00:00') "
sf_schedule_report = "30 8 * "
sf_schedule_etl = '30 1 * '
sf_schedule_main_flow = "45 2 * "
CONFIG_KEY = 'sf_config_%s' % Config.version
sf_config = Variable.get(CONFIG_KEY, deserialize_json=True, default_var={})
if sf_config:
for k, v in sf_config.items():
print(f'Overwrite {k} by {v}')
if hasattr(Config, k):
if k == 'dag_start_date':
setattr(Config, k, datetime.strptime(v, '%Y-%m-%d') )
elif v == 'None':
setattr(Config, k, None)
else:
setattr(Config, k, v)
{code}
And I have 5 dag file import this Config . they have some similar code like
{code:java}
from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.dummy_operator import DummyOperator
from airflow.operators.bash_operator import BashOperator
from airflow.operators.dagrun_operator import TriggerDagRunOperator
from airflow.models import Variable
from sf_dags_n.config import Config
default_args = {'owner': 'mithril','depends_on_past': False,
'email': ['mithril'],'email_on_failure': False,'email_on_retry': False,
'retries': 2,'schedule_interval': schedule_interval,}
{code}
{code:java}
dag = DAG('dm_sfx_etl_%s' % Config.version,
start_date=Config.dag_start_date,
default_args=default_args,
schedule_interval='20 1 * ',
user_defined_filters= { 'mod' : lambda s, d:s%d }
,
)
{code}
The stange thing is :
Change `sf_schedule_etl` in Variable took effect several times, but at some
point , I couldn't change it from variable any more, even I directly hard code
it :
```
dag = DAG('dm_sfx_etl_%s' % Config.version,
start_date=Config.dag_start_date ,
default_args=default_args,
schedule_interval='20 1* **',
user_defined_filters= \{ 'mod' : lambda s, d:s%d }
,
)
```
If such situation came, even delete dag file and delete from airflow webui
,didn't change `schedule_interval` .
PS: my dag file have running for some days, in these days ,I may add some
operator to it , or change some operator type, but it still fine . I think
there must be some cache in airflow lead to this problem.
was:
I have a `config.py` pull configure from `Variable` and merge into default
config :
{code:java}
from datetime import datetime
from airflow.models import Variable
class Config:
version = "V21"
etl_feature_dir = f'/data/dm/sales_forecast/features/version={version}'
forecast_result_dir =
f'/data/dm/sales_forecast/results/fbprophet/version={version}'
forecast_model_dir =
f'/data/dm/sales_forecast/models/fbprophet/version={version}'
forecast_result_s3_dir =
f's3a://x/data/dm/sales_forecast/fbprophet/version={version}'
etl_dir = '/data/dm/sales_forecast/etl'
feature_current_path = 'hdfs://pupuxdc/test/MERGE_OUT'
dag_start_date = datetime(2019, 10, 25)
etl_start_time = "2019-06-01 00:00:00"
etl_end_time = " (execution_date +
macros.timedelta(days=8)).strftime('%Y-%m-%d 00:00:00') "
train_start_time = " (execution_date
macros.timedelta(days=90)).strftime('%Y%m-%d 00:00:00') "
train_end_time = " execution_date.strftime('%Y-%m-%d 00:00:00') "
predict_start_time = " execution_date.strftime('%Y-%m-%d 00:00:00') "
[jira] [Updated] (AIRFLOW-5927) Airflow cache import file or variable
[
https://issues.apache.org/jira/browse/AIRFLOW-5927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kasim updated AIRFLOW-5927:
---
Description:
I have a `config.py` pull configure from `Variable` and merge into default
config :
{code:java}
from datetime import datetime
from airflow.models import Variable
class Config:
version = "V21"
etl_feature_dir = f'/data/dm/sales_forecast/features/version={version}'
forecast_result_dir =
f'/data/dm/sales_forecast/results/fbprophet/version={version}'
forecast_model_dir =
f'/data/dm/sales_forecast/models/fbprophet/version={version}'
forecast_result_s3_dir =
f's3a://x/data/dm/sales_forecast/fbprophet/version={version}'
etl_dir = '/data/dm/sales_forecast/etl'
feature_current_path = 'hdfs://pupuxdc/test/MERGE_OUT'
dag_start_date = datetime(2019, 10, 25)
etl_start_time = "2019-06-01 00:00:00"
etl_end_time = " (execution_date +
macros.timedelta(days=8)).strftime('%Y-%m-%d 00:00:00') "
train_start_time = " (execution_date
macros.timedelta(days=90)).strftime('%Y%m-%d 00:00:00') "
train_end_time = " execution_date.strftime('%Y-%m-%d 00:00:00') "
predict_start_time = " execution_date.strftime('%Y-%m-%d 00:00:00') "
predict_end_time = " (execution_date +
macros.timedelta(days=7)).strftime('%Y-%m-%d 00:00:00') "
report_start_date = " (execution_date
macros.timedelta(days=6)).strftime('%Y%m-%d 00:00:00') "
report_end_date = " execution_date.strftime('%Y-%m-%d 00:00:00') "
sf_schedule_report = "30 8 * "
sf_schedule_etl = '30 1 * '
sf_schedule_main_flow = "45 2 * "
CONFIG_KEY = 'sf_config_%s' % Config.version
sf_config = Variable.get(CONFIG_KEY, deserialize_json=True, default_var={})
if sf_config:
for k, v in sf_config.items():
print(f'Overwrite {k} by {v}')
if hasattr(Config, k):
if k == 'dag_start_date':
setattr(Config, k, datetime.strptime(v, '%Y-%m-%d') )
elif v == 'None':
setattr(Config, k, None)
else:
setattr(Config, k, v)
{code}
And I have 5 dag file import this Config . they have some similar code like
{code:java}
from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.dummy_operator import DummyOperator
from airflow.operators.bash_operator import BashOperator
from airflow.operators.dagrun_operator import TriggerDagRunOperator
from airflow.models import Variable
from sf_dags_n.config import Config
default_args = {'owner': 'mithril','depends_on_past': False,
'email': ['mithril'],'email_on_failure': False,'email_on_retry': False,
'retries': 2,'schedule_interval': schedule_interval,}
{code}
{code:java}
dag = DAG('dm_sfx_etl_%s' % Config.version,
start_date=datetime.strptime(Config.dag_start_date, '%Y-%m-%d') ,
default_args=default_args,
schedule_interval='20 1 * ',
user_defined_filters= { 'mod' : lambda s, d:s%d }
,
)
{code}
The stange thing is :
Change `sf_schedule_etl` in Variable took effect several times, but at some
point , I couldn't change it from variable any more, even I directly hard code
it :
```
dag = DAG('dm_sfx_etl_%s' % Config.version,
start_date=Config.dag_start_date ,
default_args=default_args,
schedule_interval='20 1** * ',
user_defined_filters= \{ 'mod' : lambda s, d:s%d }
,
)
```
If such situation came, even delete dag file and delete from airflow webui
,didn't change `schedule_interval` .
PS: my dag file have running for some days, in these days ,I may add some
operator to it , or change some operator type, but it still fine . I think
there must be some cache in airflow lead to this problem.
was:
I have a `config.py` pull configure from `Variable` and merge into default
config :
{code:java}
from datetime import datetime
from airflow.models import Variable
class Config:
version = "V21"
etl_feature_dir = f'/data/dm/sales_forecast/features/version={version}'
forecast_result_dir =
f'/data/dm/sales_forecast/results/fbprophet/version={version}'
forecast_model_dir =
f'/data/dm/sales_forecast/models/fbprophet/version={version}'
forecast_result_s3_dir =
f's3a://pupumall-dc-tmp/data/dm/sales_forecast/fbprophet/version={version}'
etl_dir = '/data/dm/sales_forecast/etl'
feature_current_path = 'hdfs://pupuxdc/test/MERGE_OUT'
dag_start_date = datetime(2019, 10, 25)
etl_start_time = "2019-06-01 00:00:00"
etl_end_time = " (execution_date +
macros.timedelta(days=8)).strftime('%Y-%m-%d 00:00:00') "
train_start_time = " (execution_date
macros.timedelta(days=90)).strftime('%Y%m-%d 00:00:00') "
train_end_time = " execution_date.strftime('%Y-%m-%d 00:00:00') "
predict_start_time = "
[jira] [Updated] (AIRFLOW-5927) Airflow cache import file or variable
[
https://issues.apache.org/jira/browse/AIRFLOW-5927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kasim updated AIRFLOW-5927:
---
Description:
I have a `config.py` pull configure from `Variable` and merge into default
config :
{code:java}
from datetime import datetime
from airflow.models import Variable
class Config:
version = "V21"
etl_feature_dir = f'/data/dm/sales_forecast/features/version={version}'
forecast_result_dir =
f'/data/dm/sales_forecast/results/fbprophet/version={version}'
forecast_model_dir =
f'/data/dm/sales_forecast/models/fbprophet/version={version}'
forecast_result_s3_dir =
f's3a://pupumall-dc-tmp/data/dm/sales_forecast/fbprophet/version={version}'
etl_dir = '/data/dm/sales_forecast/etl'
feature_current_path = 'hdfs://pupuxdc/test/MERGE_OUT'
dag_start_date = datetime(2019, 10, 25)
etl_start_time = "2019-06-01 00:00:00"
etl_end_time = " (execution_date +
macros.timedelta(days=8)).strftime('%Y-%m-%d 00:00:00') "
train_start_time = " (execution_date
macros.timedelta(days=90)).strftime('%Y%m-%d 00:00:00') "
train_end_time = " execution_date.strftime('%Y-%m-%d 00:00:00') "
predict_start_time = " execution_date.strftime('%Y-%m-%d 00:00:00') "
predict_end_time = " (execution_date +
macros.timedelta(days=7)).strftime('%Y-%m-%d 00:00:00') "
report_start_date = " (execution_date
macros.timedelta(days=6)).strftime('%Y%m-%d 00:00:00') "
report_end_date = " execution_date.strftime('%Y-%m-%d 00:00:00') "
sf_schedule_report = "30 8 * "
sf_schedule_etl = '30 1 * '
sf_schedule_main_flow = "45 2 * "
CONFIG_KEY = 'sf_config_%s' % Config.version
sf_config = Variable.get(CONFIG_KEY, deserialize_json=True, default_var={})
if sf_config:
for k, v in sf_config.items():
print(f'Overwrite {k} by {v}')
if hasattr(Config, k):
if k == 'dag_start_date':
setattr(Config, k, datetime.strptime(v, '%Y-%m-%d') )
elif v == 'None':
setattr(Config, k, None)
else:
setattr(Config, k, v)
{code}
And I have 5 dag file import this Config . they have some similar code like
{code:java}
from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.dummy_operator import DummyOperator
from airflow.operators.bash_operator import BashOperator
from airflow.operators.dagrun_operator import TriggerDagRunOperator
from airflow.models import Variable
from sf_dags_n.config import Config
default_args = {'owner': 'mithril','depends_on_past': False,
'email': ['mithril'],'email_on_failure': False,'email_on_retry': False,
'retries': 2,'schedule_interval': schedule_interval,}
{code}
{code:java}
dag = DAG('dm_sfx_etl_%s' % Config.version,
start_date=datetime.strptime(Config.dag_start_date, '%Y-%m-%d') ,
default_args=default_args,
schedule_interval='20 1 * ',
user_defined_filters= { 'mod' : lambda s, d:s%d }
,
)
{code}
The stange thing is :
Change `sf_schedule_etl` in Variable took effect several times, but at some
point , I couldn't change it from variable any more, even I directly hard code
it :
```
dag = DAG('dm_sfx_etl_%s' % Config.version,
start_date=Config.dag_start_date ,
default_args=default_args,
schedule_interval='20 1 * ** ',
user_defined_filters= \{ 'mod' : lambda s, d:s%d }
,
)
```
If such situation came, even delete dag file and delete from airflow webui
,didn't change `schedule_interval` .
PS: my dag file have running for some days, in these days ,I may add some
operator to it , or change some operator type, but it still fine . I think
there must be some cache in airflow lead to this problem.
was:
I have a `config.py` pull configure from `Variable` and merge into default
config :
```python
from datetime import datetime
from airflow.models import Variable
class Config:
version = "V21"
etl_feature_dir = f'/data/dm/sales_forecast/features/version=\{version}'
forecast_result_dir =
f'/data/dm/sales_forecast/results/fbprophet/version=\{version}'
forecast_model_dir =
f'/data/dm/sales_forecast/models/fbprophet/version=\{version}'
forecast_result_s3_dir =
f's3a://pupumall-dc-tmp/data/dm/sales_forecast/fbprophet/version=\{version}'
etl_dir = '/data/dm/sales_forecast/etl'
feature_current_path = 'hdfs://pupuxdc/test/MERGE_OUT'
dag_start_date = datetime(2019, 10, 25)
etl_start_time = "2019-06-01 00:00:00"
etl_end_time = "{{ (execution_date +
macros.timedelta(days=8)).strftime('%Y-%m-%d 00:00:00') }}"
train_start_time = "{{ (execution_date
-macros.timedelta(days=90)).strftime('%Y-%m-%d 00:00:00') }}"
train_end_time = "{{ execution_date.strftime('%Y-%m-%d 00:00:00') }}"
[jira] [Updated] (AIRFLOW-5927) Airflow cache import file or variable
[
https://issues.apache.org/jira/browse/AIRFLOW-5927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kasim updated AIRFLOW-5927:
---
Description:
I have a `config.py` pull configure from `Variable` and merge into default
config :
```python
from datetime import datetime
from airflow.models import Variable
class Config:
version = "V21"
etl_feature_dir = f'/data/dm/sales_forecast/features/version=\{version}'
forecast_result_dir =
f'/data/dm/sales_forecast/results/fbprophet/version=\{version}'
forecast_model_dir =
f'/data/dm/sales_forecast/models/fbprophet/version=\{version}'
forecast_result_s3_dir =
f's3a://pupumall-dc-tmp/data/dm/sales_forecast/fbprophet/version=\{version}'
etl_dir = '/data/dm/sales_forecast/etl'
feature_current_path = 'hdfs://pupuxdc/test/MERGE_OUT'
dag_start_date = datetime(2019, 10, 25)
etl_start_time = "2019-06-01 00:00:00"
etl_end_time = "{{ (execution_date +
macros.timedelta(days=8)).strftime('%Y-%m-%d 00:00:00') }}"
train_start_time = "{{ (execution_date
-macros.timedelta(days=90)).strftime('%Y-%m-%d 00:00:00') }}"
train_end_time = "{{ execution_date.strftime('%Y-%m-%d 00:00:00') }}"
predict_start_time = "{{ execution_date.strftime('%Y-%m-%d 00:00:00') }}"
predict_end_time = "{{ (execution_date +
macros.timedelta(days=7)).strftime('%Y-%m-%d 00:00:00') }}"
report_start_date = "{{ (execution_date
-macros.timedelta(days=6)).strftime('%Y-%m-%d 00:00:00') }}"
report_end_date = "{{ execution_date.strftime('%Y-%m-%d 00:00:00') }}"
sf_schedule_report = "30 8 ** *"
sf_schedule_etl = '30 1 ** *'
sf_schedule_main_flow = "45 2 ** *"
CONFIG_KEY = 'sf_config_%s' % Config.version
sf_config = Variable.get(CONFIG_KEY, deserialize_json=True, default_var={})
if sf_config:
for k, v in sf_config.items():
print(f'Overwrite \{k} by \{v}')
if hasattr(Config, k):
if k == 'dag_start_date':
setattr(Config, k, datetime.strptime(v, '%Y-%m-%d') )
elif v == 'None':
setattr(Config, k, None)
else:
setattr(Config, k, v)```
And I have 5 dag file import this Config . they have some similar code like
```python
from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.dummy_operator import DummyOperator
from airflow.operators.bash_operator import BashOperator
from airflow.operators.dagrun_operator import TriggerDagRunOperator
from airflow.models import Variable
from sf_dags_n.config import Config
default_args =
{ 'owner': 'mithril', 'depends_on_past': False, 'email':
['mithril'], 'email_on_failure': False, 'email_on_retry': False,
'retries': 2, }
dag = DAG('dm_sfx_etl_%s' % Config.version,
start_date=datetime.strptime(Config.dag_start_date, '%Y-%m-%d') ,
default_args=default_args,
schedule_interval=Config.sf_schedule_etl,
user_defined_filters=
{ 'mod' : lambda s, d:s%d }
,
)
# other codes
```
The stange thing is :
Change `sf_schedule_etl` in Variable took effect several times, but at some
point , I couldn't change it from variable any more, even I directly hard code
it :
```
dag = DAG('dm_sfx_etl_%s' % Config.version,
start_date=datetime.strptime(Config.dag_start_date, '%Y-%m-%d') ,
default_args=default_args,
schedule_interval='20 1 ** *',
user_defined_filters= \{ 'mod' : lambda s, d:s%d }
,
)
```
If such situation came, even delete dag file and delete from airflow webui
,didn't change `schedule_interval` .
PS: my dag file have running for some days, in these days ,I may add some
operator to it , or change some operator type, but it still fine . I think
there must be some cache in airflow lead to this problem.
was:
I have a `config.py` pull configure from `Variable` and merge into default
config :
```python
from datetime import datetime
from airflow.models import Variable
class Config:
version = "V21"
etl_feature_dir = f'/data/dm/sales_forecast/features/version=\{version}'
forecast_result_dir =
f'/data/dm/sales_forecast/results/fbprophet/version=\{version}'
forecast_model_dir =
f'/data/dm/sales_forecast/models/fbprophet/version=\{version}'
forecast_result_s3_dir =
f's3a://pupumall-dc-tmp/data/dm/sales_forecast/fbprophet/version=\{version}'
etl_dir = '/data/dm/sales_forecast/etl'
feature_current_path = 'hdfs://pupuxdc/test/MERGE_OUT'
dag_start_date = datetime(2019, 10, 25)
etl_start_time = "2019-06-01 00:00:00"
etl_end_time = "\{{ (execution_date +
macros.timedelta(days=8)).strftime('%Y-%m-%d 00:00:00') }}"
train_start_time = "\{{ (execution_date -
macros.timedelta(days=90)).strftime('%Y-%m-%d 00:00:00') }}"
train_end_time = "\{{
[GitHub] [airflow] mik-laj commented on issue #6487: [AIRFLOW-XXX] Alphabetical table and remove duplicate
mik-laj commented on issue #6487: [AIRFLOW-XXX] Alphabetical table and remove duplicate URL: https://github.com/apache/airflow/pull/6487#issuecomment-553690347 I thank you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] zhongjiajie commented on issue #6487: [AIRFLOW-XXX] Alphabetical table and remove duplicate
zhongjiajie commented on issue #6487: [AIRFLOW-XXX] Alphabetical table and remove duplicate URL: https://github.com/apache/airflow/pull/6487#issuecomment-553689896 @mik-laj Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-5927) Airflow cache import file or variable
kasim created AIRFLOW-5927:
--
Summary: Airflow cache import file or variable
Key: AIRFLOW-5927
URL: https://issues.apache.org/jira/browse/AIRFLOW-5927
Project: Apache Airflow
Issue Type: Bug
Components: DAG, database
Affects Versions: 1.10.3
Reporter: kasim
I have a `config.py` pull configure from `Variable` and merge into default
config :
```python
from datetime import datetime
from airflow.models import Variable
class Config:
version = "V21"
etl_feature_dir = f'/data/dm/sales_forecast/features/version=\{version}'
forecast_result_dir =
f'/data/dm/sales_forecast/results/fbprophet/version=\{version}'
forecast_model_dir =
f'/data/dm/sales_forecast/models/fbprophet/version=\{version}'
forecast_result_s3_dir =
f's3a://pupumall-dc-tmp/data/dm/sales_forecast/fbprophet/version=\{version}'
etl_dir = '/data/dm/sales_forecast/etl'
feature_current_path = 'hdfs://pupuxdc/test/MERGE_OUT'
dag_start_date = datetime(2019, 10, 25)
etl_start_time = "2019-06-01 00:00:00"
etl_end_time = "\{{ (execution_date +
macros.timedelta(days=8)).strftime('%Y-%m-%d 00:00:00') }}"
train_start_time = "\{{ (execution_date -
macros.timedelta(days=90)).strftime('%Y-%m-%d 00:00:00') }}"
train_end_time = "\{{ execution_date.strftime('%Y-%m-%d 00:00:00') }}"
predict_start_time = "\{{ execution_date.strftime('%Y-%m-%d 00:00:00') }}"
predict_end_time = "\{{ (execution_date +
macros.timedelta(days=7)).strftime('%Y-%m-%d 00:00:00') }}"
report_start_date = "\{{ (execution_date -
macros.timedelta(days=6)).strftime('%Y-%m-%d 00:00:00') }}"
report_end_date = "\{{ execution_date.strftime('%Y-%m-%d 00:00:00') }}"
sf_schedule_report = "30 8 * * *"
sf_schedule_etl = '30 1 * * *'
sf_schedule_main_flow = "45 2 * * *"
CONFIG_KEY = 'sf_config_%s' % Config.version
sf_config = Variable.get(CONFIG_KEY, deserialize_json=True, default_var={})
if sf_config:
for k, v in sf_config.items():
print(f'Overwrite \{k} by \{v}')
if hasattr(Config, k):
if k == 'dag_start_date':
print(datetime.strptime(v, '%Y-%m-%d'))
print(type(datetime.strptime(v, '%Y-%m-%d')))
setattr(Config, k, datetime.strptime(v, '%Y-%m-%d') )
print(Config.dag_start_date)
print(type(Config.dag_start_date))
if v == 'None':
setattr(Config, k, None)
else:
setattr(Config, k, v)```
And I have 5 dag file import this Config . they have some similar code like
```python
from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.dummy_operator import DummyOperator
from airflow.operators.bash_operator import BashOperator
from airflow.operators.dagrun_operator import TriggerDagRunOperator
from airflow.models import Variable
from sf_dags_n.config import Config
default_args = {
'owner': 'mithril',
'depends_on_past': False,
'email': ['mithril'],
'email_on_failure': False,
'email_on_retry': False,
'retries': 2,
}
print(Config.dag_start_date)
print(type(Config.dag_start_date))
dag = DAG('dm_sfx_etl_%s' % Config.version,
start_date=datetime.strptime(Config.dag_start_date, '%Y-%m-%d') ,
default_args=default_args,
schedule_interval=Config.sf_schedule_etl,
user_defined_filters={
'mod' : lambda s, d:s%d
},
)
# other codes
```
The stange thing is :
1. At first , in dag file which `from sf_dags_n.config import Config` ,
Config.dag_start_date was datetime type . But it became str several days ago,
I check it type in config.py and the dag file , Config.dag_start_date is
still datetime type , but in dag file is str . As I remenber, I set default
Config.dag_start_date with type str, but changed it when occured problem . Then
it was fine for some time before went wrong.
2. Change `sf_schedule_etl` in Variable took effect several times, but at some
point , I couldn't change it from variable any more, even I directly hard code
it :
```
dag = DAG('dm_sfx_etl_%s' % Config.version,
start_date=datetime.strptime(Config.dag_start_date, '%Y-%m-%d') ,
default_args=default_args,
schedule_interval='20 1 * * *',
user_defined_filters={
'mod' : lambda s, d:s%d
},
)
```
If such situation came, even delete dag file and delete from airflow webui
,didn't change `schedule_interval` .
PS: my dag file have running for some days, in these days ,I may add some
operator to it , or change some operator type, but it still fine . I think
there must be some cache in airflow lead to this problem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (AIRFLOW-4593) Memory leak in Airflow scheduler
[ https://issues.apache.org/jira/browse/AIRFLOW-4593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972109#comment-16972109 ] sison edited comment on AIRFLOW-4593 at 11/14/19 1:51 AM: -- I find this problem in version 1.10.6 . I run the scheduler in a kubenetes pod, the memory usage is growing about 100M per day And when I use the airflow 1.10.4 to run the scheduler, the problem has not gone away too. 2019.11.14，I find a interesting phenomenon，when I delete the scheduler logs, the scheduler memory usage decreased immediately. was (Author: sison): I find this problem in version 1.10.6 . I run the scheduler in a kubenetes pod, the memory usage is growing about 100M per day And when I use the airflow 1.10.4 to run the scheduler, the problem has not gone away too. > Memory leak in Airflow scheduler > > > Key: AIRFLOW-4593 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4593 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Affects Versions: 1.10.2 >Reporter: Nikhil SInghal >Priority: Major > Attachments: Screenshot 2019-05-30 at 3.19.33 PM.png > > > We are running Apache Airflow on Kubernetes. When I see my Grafana Dashboard > I see that the memory used is consistently increasing. Can anyone give me > some pointers for how can I debug this or with existing issues/solutions > related to this. > This is a sharp increase in Airflow scheduler and slow increase for worker > and webserver > !Screenshot 2019-05-30 at 3.19.33 PM.png|width=100%! > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5115) S3KeySensor template_fields for bucket_name & bucket_key do not support Jinja variables
[
https://issues.apache.org/jira/browse/AIRFLOW-5115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973822#comment-16973822
]
Juan Ramos Fuentes commented on AIRFLOW-5115:
-
After testing locally, I have confirmed that moving the input validation logic
to the `poke` method solves the issue. Do you have a PR open for this
[~dsynkov]? Let me know if I can help in any way. This will unblock me and my
team
> S3KeySensor template_fields for bucket_name & bucket_key do not support Jinja
> variables
> ---
>
> Key: AIRFLOW-5115
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5115
> Project: Apache Airflow
> Issue Type: Bug
> Components: aws
>Affects Versions: 1.9.0
>Reporter: Dmitriy Synkov
>Assignee: Dmitriy Synkov
>Priority: Minor
> Labels: easyfix, patch
> Fix For: 2.0.0
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> In all Airflow operators (which inherit form {{BaseOperator}}) there is a
> {{template_fields}} attribute defined as ["which fields will get
> jinjafied"|https://github.com/apache/airflow/blob/master/airflow/models/baseoperator.py#L218-L219]).
> For the {{S3KeySensor}} op in specific, these are {{template_fields =
> ('bucket_key', 'bucket_name')}}.
> The {{bucket_key}} kwarg, however, has some input validation in that the
> {{bucket_key}} needs to begin with the S3 protocol {{s3://}}; this exception
> is thrown by the
> [constructor|https://github.com/apache/airflow/blob/master/airflow/sensors/s3_key_sensor.py#L71-L74],
> which makes it impossible to use Jinja strings as an arg to {{bucket_key}},
> since these don't get rendered in the scope of the DAG {{*.py}} file itself.
> Below is an example; I'm using Airflow 1.9.0 with Python 3.5.3:
> Given the below DAG code, where "my_s3_key" is
> {{s3://bucket/prefix/object.txt:}}
> {code:java}
> dag = DAG('sample_dag', start_date=datetime(2019, 8, 1, 12, 15))
> s3_variable_sensor = S3KeySensor(
> task_id='s3_variable_sensor',
> bucket_key=Variable.get('my_s3_key'),
> dag=dag
> )
> s3_jinja_sensor = S3KeySensor(
> task_id='s3_jinja_sensor',
> bucket_key="{{ var.value.my_s3_key }}",
> dag=dag
> )
> {code}
> Executing the first task will run just fine while the next task will throw
> the following exception:
> {code:java}
> airflow.exceptions.AirflowException: Please provide a bucket_name.
> {code}
> This ticket is to propose a code change that will move input validation out
> of the constructor to allow for Jinja-templated strings to be passed into
> both {{bucket_name}} and {{bucket_key}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] mik-laj merged pull request #6487: [AIRFLOW-XXX] Alphabetical table and remove duplicate
mik-laj merged pull request #6487: [AIRFLOW-XXX] Alphabetical table and remove duplicate URL: https://github.com/apache/airflow/pull/6487 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6579: [AIRFLOW-5926] Fix race-condition in TestCliWebServer
mik-laj commented on a change in pull request #6579: [AIRFLOW-5926] Fix
race-condition in TestCliWebServer
URL: https://github.com/apache/airflow/pull/6579#discussion_r346068956
##
File path: tests/cli/test_cli.py
##
@@ -761,6 +761,31 @@ class TestCliWebServer(unittest.TestCase):
def setUpClass(cls):
cls.parser = cli.CLIFactory.get_parser()
+def setUp(self) -> None:
+self._check_processes()
+self._clean_pidfiles()
+
+def _check_processes(self):
+try:
+# Confirm that webserver hasn't been launched.
+# pgrep returns exit status 1 if no process matched.
+self.assertEqual(1, subprocess.Popen(["pgrep", "-f", "-c",
"airflow webserver"]).wait())
+self.assertEqual(1, subprocess.Popen(["pgrep", "-c",
"gunicorn"]).wait())
+except: # noqa: E722
+subprocess.Popen(["ps", "-ax"]).wait()
+raise
+
+def tearDown(self) -> None:
+self._check_processes()
+
+def _clean_pidfiles(self):
+pidfile_webserver = cli.setup_locations("webserver")[0]
+pidfile_monitor = cli.setup_locations("webserver-monitor")[0]
Review comment:
This can cause another race condition.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6539: [AIRFLOW-5783] AIP-21 Move aws redshift into providers structure
codecov-io commented on issue #6539: [AIRFLOW-5783] AIP-21 Move aws redshift into providers structure URL: https://github.com/apache/airflow/pull/6539#issuecomment-553662295 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6539?src=pr=h1) Report > Merging [#6539](https://codecov.io/gh/apache/airflow/pull/6539?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/d633d3ac44c395e6c43cd388f98fba1ce1c435a3?src=pr=desc) will **decrease** coverage by `0.9%`. > The diff coverage is `71.15%`. [](https://codecov.io/gh/apache/airflow/pull/6539?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6539 +/- ## == - Coverage 83.81% 82.91% -0.91% == Files 645 647 +2 Lines 3735637365 +9 == - Hits3130930980 -329 - Misses 6047 6385 +338 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6539?src=pr=tree) | Coverage Δ | | |---|---|---| | [...low/contrib/sensors/aws\_redshift\_cluster\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6539/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvYXdzX3JlZHNoaWZ0X2NsdXN0ZXJfc2Vuc29yLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/contrib/hooks/redshift\_hook.py](https://codecov.io/gh/apache/airflow/pull/6539/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3JlZHNoaWZ0X2hvb2sucHk=) | `0% <0%> (-75%)` | :arrow_down: | | [airflow/providers/amazon/aws/sensors/redshift.py](https://codecov.io/gh/apache/airflow/pull/6539/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9zZW5zb3JzL3JlZHNoaWZ0LnB5) | `100% <100%> (ø)` | | | [airflow/providers/amazon/aws/hooks/redshift.py](https://codecov.io/gh/apache/airflow/pull/6539/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9ob29rcy9yZWRzaGlmdC5weQ==) | `75.86% <75.86%> (ø)` | | | [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6539/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6539/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6539/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/6539/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6539/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6539/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | ... and [22 more](https://codecov.io/gh/apache/airflow/pull/6539/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6539?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6539?src=pr=footer). Last update [d633d3a...da6e397](https://codecov.io/gh/apache/airflow/pull/6539?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #6579: [AIRFLOW-5926] Fix race-condition in TestCliWebServer
mik-laj commented on issue #6579: [AIRFLOW-5926] Fix race-condition in TestCliWebServer URL: https://github.com/apache/airflow/pull/6579#issuecomment-553660837 @Kaxil I updated the description. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-5926) Fix race-condition in TestCliWebServer
[ https://issues.apache.org/jira/browse/AIRFLOW-5926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula updated AIRFLOW-5926: --- Description: The current tests work as follows. # The airflow command is started in a separate process, without waiting for it to finish. This separate process run the airflow process that creates the deamon process - airflow-webserver-monitor. The deamon process starts another process - gunicorn. # At the same time, the tests are waiting for the monitor process to start. Checks if the pid file for the monitor exists. # In the next step, the test checks that the airflow command has already ended *Bug* we should check the monitor process, not the command that starts it. # In the next step, the test checks if the gunicorn is already running # Tests sends a signal to the monitor to close processes - monitor and gunicorn. Summary: The test checks to see if there is a temporary process that starts the monitor instead of checking that the monitor is running. > Fix race-condition in TestCliWebServer > --- > > Key: AIRFLOW-5926 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5926 > Project: Apache Airflow > Issue Type: Bug > Components: cli >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Major > > The current tests work as follows. > # The airflow command is started in a separate process, without waiting for > it to finish. This separate process run the airflow process that creates the > deamon process - airflow-webserver-monitor. The deamon process starts another > process - gunicorn. > # At the same time, the tests are waiting for the monitor process to start. > Checks if the pid file for the monitor exists. > # In the next step, the test checks that the airflow command has already > ended > *Bug* we should check the monitor process, not the command that starts it. > # In the next step, the test checks if the gunicorn is already running > # Tests sends a signal to the monitor to close processes - monitor and > gunicorn. > Summary: > The test checks to see if there is a temporary process that starts the > monitor instead of checking that the monitor is running. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5115) S3KeySensor template_fields for bucket_name & bucket_key do not support Jinja variables
[
https://issues.apache.org/jira/browse/AIRFLOW-5115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973817#comment-16973817
]
Juan Ramos Fuentes commented on AIRFLOW-5115:
-
Currently running into this same issue when passing a Jinja template to the
bucket_key arg.
Running version 1.10.6
> S3KeySensor template_fields for bucket_name & bucket_key do not support Jinja
> variables
> ---
>
> Key: AIRFLOW-5115
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5115
> Project: Apache Airflow
> Issue Type: Bug
> Components: aws
>Affects Versions: 1.9.0
>Reporter: Dmitriy Synkov
>Assignee: Dmitriy Synkov
>Priority: Minor
> Labels: easyfix, patch
> Fix For: 2.0.0
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> In all Airflow operators (which inherit form {{BaseOperator}}) there is a
> {{template_fields}} attribute defined as ["which fields will get
> jinjafied"|https://github.com/apache/airflow/blob/master/airflow/models/baseoperator.py#L218-L219]).
> For the {{S3KeySensor}} op in specific, these are {{template_fields =
> ('bucket_key', 'bucket_name')}}.
> The {{bucket_key}} kwarg, however, has some input validation in that the
> {{bucket_key}} needs to begin with the S3 protocol {{s3://}}; this exception
> is thrown by the
> [constructor|https://github.com/apache/airflow/blob/master/airflow/sensors/s3_key_sensor.py#L71-L74],
> which makes it impossible to use Jinja strings as an arg to {{bucket_key}},
> since these don't get rendered in the scope of the DAG {{*.py}} file itself.
> Below is an example; I'm using Airflow 1.9.0 with Python 3.5.3:
> Given the below DAG code, where "my_s3_key" is
> {{s3://bucket/prefix/object.txt:}}
> {code:java}
> dag = DAG('sample_dag', start_date=datetime(2019, 8, 1, 12, 15))
> s3_variable_sensor = S3KeySensor(
> task_id='s3_variable_sensor',
> bucket_key=Variable.get('my_s3_key'),
> dag=dag
> )
> s3_jinja_sensor = S3KeySensor(
> task_id='s3_jinja_sensor',
> bucket_key="{{ var.value.my_s3_key }}",
> dag=dag
> )
> {code}
> Executing the first task will run just fine while the next task will throw
> the following exception:
> {code:java}
> airflow.exceptions.AirflowException: Please provide a bucket_name.
> {code}
> This ticket is to propose a code change that will move input validation out
> of the constructor to allow for Jinja-templated strings to be passed into
> both {{bucket_name}} and {{bucket_key}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5880) Enforce unique task ids
[ https://issues.apache.org/jira/browse/AIRFLOW-5880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973816#comment-16973816 ] ASF subversion and git services commented on AIRFLOW-5880: -- Commit fd425e89608cdef4f378d64a003da7c1a71fd830 in airflow's branch refs/heads/master from Kaxil Naik [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=fd425e8 ] [AIRFLOW-5880] Enforce unique task ids (#6549) > Enforce unique task ids > --- > > Key: AIRFLOW-5880 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5880 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Kaxil Naik >Priority: Major > Fix For: 2.0.0 > > > Currently, task with equal ids in a DAG simply overwrite each other and the > last assigned task is saved. This can lead to unexpected behaviour and an > exception should be raised. > We've had a PendingDeprecationWarning on this for 4 years, time to get rid of > it: > [https://github.com/apache/airflow/commit/385add2bf3eb7cd6103cfec0e5516234eeb72443] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5880) Enforce unique task ids
[ https://issues.apache.org/jira/browse/AIRFLOW-5880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-5880. - Fix Version/s: 2.0.0 Resolution: Fixed > Enforce unique task ids > --- > > Key: AIRFLOW-5880 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5880 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Kaxil Naik >Priority: Major > Fix For: 2.0.0 > > > Currently, task with equal ids in a DAG simply overwrite each other and the > last assigned task is saved. This can lead to unexpected behaviour and an > exception should be raised. > We've had a PendingDeprecationWarning on this for 4 years, time to get rid of > it: > [https://github.com/apache/airflow/commit/385add2bf3eb7cd6103cfec0e5516234eeb72443] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5880) Enforce unique task ids
[ https://issues.apache.org/jira/browse/AIRFLOW-5880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973815#comment-16973815 ] ASF GitHub Bot commented on AIRFLOW-5880: - kaxil commented on pull request #6549: [AIRFLOW-5880] Enforce unique task ids URL: https://github.com/apache/airflow/pull/6549 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Enforce unique task ids > --- > > Key: AIRFLOW-5880 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5880 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Kaxil Naik >Priority: Major > > Currently, task with equal ids in a DAG simply overwrite each other and the > last assigned task is saved. This can lead to unexpected behaviour and an > exception should be raised. > We've had a PendingDeprecationWarning on this for 4 years, time to get rid of > it: > [https://github.com/apache/airflow/commit/385add2bf3eb7cd6103cfec0e5516234eeb72443] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] kaxil merged pull request #6549: [AIRFLOW-5880] Enforce unique task ids
kaxil merged pull request #6549: [AIRFLOW-5880] Enforce unique task ids URL: https://github.com/apache/airflow/pull/6549 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run
codecov-io edited a comment on issue #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run URL: https://github.com/apache/airflow/pull/4751#issuecomment-466029246 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4751?src=pr=h1) Report > Merging [#4751](https://codecov.io/gh/apache/airflow/pull/4751?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/d633d3ac44c395e6c43cd388f98fba1ce1c435a3?src=pr=desc) will **decrease** coverage by `0.43%`. > The diff coverage is `85.18%`. [](https://codecov.io/gh/apache/airflow/pull/4751?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4751 +/- ## == - Coverage 83.81% 83.38% -0.44% == Files 645 645 Lines 3735637379 +23 == - Hits3130931167 -142 - Misses 6047 6212 +165 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4751?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `73.88% <100%> (-1.09%)` | :arrow_down: | | [airflow/models/dagrun.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFncnVuLnB5) | `96.6% <100%> (ø)` | :arrow_up: | | [airflow/ti\_deps/dep\_context.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcF9jb250ZXh0LnB5) | `100% <100%> (ø)` | :arrow_up: | | [airflow/ti\_deps/deps/trigger\_rule\_dep.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvdHJpZ2dlcl9ydWxlX2RlcC5weQ==) | `88.42% <80.95%> (-2.37%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/executors/sequential\_executor.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvc2VxdWVudGlhbF9leGVjdXRvci5weQ==) | `47.61% <0%> (-52.39%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `77.14% <0%> (-21.43%)` | :arrow_down: | | ... and [8 more](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4751?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4751?src=pr=footer). Last update [d633d3a...bb9d48b](https://codecov.io/gh/apache/airflow/pull/4751?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on issue #6579: [AIRFLOW-5926] Fix race-condition in TestCliWebServer
kaxil commented on issue #6579: [AIRFLOW-5926] Fix race-condition in TestCliWebServer URL: https://github.com/apache/airflow/pull/6579#issuecomment-553656839 Thanks @mik-laj , appreciate it This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil merged pull request #6581: [AIRFLOW-XXX] Fix the docstring for Dataproc get_job method
kaxil merged pull request #6581: [AIRFLOW-XXX] Fix the docstring for Dataproc get_job method URL: https://github.com/apache/airflow/pull/6581 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #6579: [AIRFLOW-5926] Fix race-condition in TestCliWebServer
mik-laj commented on issue #6579: [AIRFLOW-5926] Fix race-condition in TestCliWebServer URL: https://github.com/apache/airflow/pull/6579#issuecomment-553656575 @kaxil I added this description as a comment to the code, but I will also add it in the PR and ticket description This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on issue #6581: [AIRFLOW-XXX] Fix the docstring for Dataproc get_job method
kaxil commented on issue #6581: [AIRFLOW-XXX] Fix the docstring for Dataproc get_job method URL: https://github.com/apache/airflow/pull/6581#issuecomment-553656526 Thanks @mik-laj This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil edited a comment on issue #6579: [AIRFLOW-5926] Fix race-condition in TestCliWebServer
kaxil edited a comment on issue #6579: [AIRFLOW-5926] Fix race-condition in TestCliWebServer URL: https://github.com/apache/airflow/pull/6579#issuecomment-553654692 @mik-laj Can you please specify what is the race condition either in the Jira issue or the PR description! It is important that we follow our own Contribution guidelines :D I am fine where it is straight-forward and self-explanatory in PR title like "Adding new operator/hook" but for BugFixes a short 1-liner or a description would help users know about issue without having to look at the code. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5925) Upgrade funcsigs dependency
[ https://issues.apache.org/jira/browse/AIRFLOW-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973810#comment-16973810 ] ASF subversion and git services commented on AIRFLOW-5925: -- Commit 0a08cd3aa4b075723a26ee52499447747f06ab48 in airflow's branch refs/heads/master from aoen [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=0a08cd3 ] [AIRFLOW-5925] Relax funcsigs and psutil version requirements (#6580) > Upgrade funcsigs dependency > --- > > Key: AIRFLOW-5925 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5925 > Project: Apache Airflow > Issue Type: Bug > Components: dependencies >Affects Versions: 1.10.6 >Reporter: Dan Davydov >Assignee: Dan Davydov >Priority: Minor > > Loosen funcsigs version range restrictions to allow it to be upgraded. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] aoen commented on issue #6580: [AIRFLOW-5925] Relax funcsigs version requirement
aoen commented on issue #6580: [AIRFLOW-5925] Relax funcsigs version requirement URL: https://github.com/apache/airflow/pull/6580#issuecomment-553655244 Thank you very much Kaxil :)! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5925) Upgrade funcsigs dependency
[ https://issues.apache.org/jira/browse/AIRFLOW-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973809#comment-16973809 ] ASF GitHub Bot commented on AIRFLOW-5925: - aoen commented on pull request #6580: [AIRFLOW-5925] Relax funcsigs version requirement URL: https://github.com/apache/airflow/pull/6580 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Upgrade funcsigs dependency > --- > > Key: AIRFLOW-5925 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5925 > Project: Apache Airflow > Issue Type: Bug > Components: dependencies >Affects Versions: 1.10.6 >Reporter: Dan Davydov >Assignee: Dan Davydov >Priority: Minor > > Loosen funcsigs version range restrictions to allow it to be upgraded. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] aoen merged pull request #6580: [AIRFLOW-5925] Relax funcsigs version requirement
aoen merged pull request #6580: [AIRFLOW-5925] Relax funcsigs version requirement URL: https://github.com/apache/airflow/pull/6580 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on issue #6579: [AIRFLOW-5926] Fix race-condition in TestCliWebServer
kaxil commented on issue #6579: [AIRFLOW-5926] Fix race-condition in TestCliWebServer URL: https://github.com/apache/airflow/pull/6579#issuecomment-553654692 @mik-laj Can you please specify what is the race condition either in the Jira issue or the PR description! It is important that we follow our own Contribution guidelines :D This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil opened a new pull request #6581: [AIRFLOW-XXX] Fix the docstring for Dataproc get_job method
kaxil opened a new pull request #6581: [AIRFLOW-XXX] Fix the docstring for
Dataproc get_job method
URL: https://github.com/apache/airflow/pull/6581
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
Fix the docstring for Dataproc get_job method
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #5913: [AIRFLOW-5073] Change SQLSensor to keep poking receving NULL, '0'
codecov-io edited a comment on issue #5913: [AIRFLOW-5073] Change SQLSensor to keep poking receving NULL, '0' URL: https://github.com/apache/airflow/pull/5913#issuecomment-546476497 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5913?src=pr=h1) Report > Merging [#5913](https://codecov.io/gh/apache/airflow/pull/5913?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/d633d3ac44c395e6c43cd388f98fba1ce1c435a3?src=pr=desc) will **decrease** coverage by `0.32%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/5913?src=pr=tree) ```diff @@Coverage Diff @@ ## master#5913 +/- ## == - Coverage 83.81% 83.48% -0.33% == Files 645 645 Lines 3735637353 -3 == - Hits3130931186 -123 - Misses 6047 6167 +120 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5913?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/sensors/sql\_sensor.py](https://codecov.io/gh/apache/airflow/pull/5913/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL3NxbF9zZW5zb3IucHk=) | `100% <ø> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/5913/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/5913/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/5913/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/5913/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/5913/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `77.14% <0%> (-21.43%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/5913/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (-3.63%)` | :arrow_down: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/5913/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.28% <0%> (-0.51%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/5913/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `58.31% <0%> (+0.32%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5913?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5913?src=pr=footer). Last update [d633d3a...64fdade](https://codecov.io/gh/apache/airflow/pull/5913?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #5913: [AIRFLOW-5073] Change SQLSensor to keep poking receving NULL, '0'
codecov-io edited a comment on issue #5913: [AIRFLOW-5073] Change SQLSensor to keep poking receving NULL, '0' URL: https://github.com/apache/airflow/pull/5913#issuecomment-546476497 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5913?src=pr=h1) Report > Merging [#5913](https://codecov.io/gh/apache/airflow/pull/5913?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/d633d3ac44c395e6c43cd388f98fba1ce1c435a3?src=pr=desc) will **decrease** coverage by `0.32%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/5913?src=pr=tree) ```diff @@Coverage Diff @@ ## master#5913 +/- ## == - Coverage 83.81% 83.48% -0.33% == Files 645 645 Lines 3735637353 -3 == - Hits3130931186 -123 - Misses 6047 6167 +120 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5913?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/sensors/sql\_sensor.py](https://codecov.io/gh/apache/airflow/pull/5913/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL3NxbF9zZW5zb3IucHk=) | `100% <ø> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/5913/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/5913/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/5913/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/5913/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/5913/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `77.14% <0%> (-21.43%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/5913/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (-3.63%)` | :arrow_down: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/5913/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.28% <0%> (-0.51%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/5913/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `58.31% <0%> (+0.32%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5913?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5913?src=pr=footer). Last update [d633d3a...64fdade](https://codecov.io/gh/apache/airflow/pull/5913?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] deshraj commented on issue #6380: [AIRFLOW-3632] Allow replace_microseconds in trigger_dag REST request
deshraj commented on issue #6380: [AIRFLOW-3632] Allow replace_microseconds in trigger_dag REST request URL: https://github.com/apache/airflow/pull/6380#issuecomment-553624429 No worries. Take care. :) Happy to help if needed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] jj-ookla edited a comment on issue #6422: [AIRFLOW-5690] Change log level local_task_job.py
jj-ookla edited a comment on issue #6422: [AIRFLOW-5690] Change log level local_task_job.py URL: https://github.com/apache/airflow/pull/6422#issuecomment-553618379 > adityav It is actually a bit more than that. What it implies is that the heart beating is being throttled because of the high heart rate. @Fokko Is "high heart rate" referring to `job_heartbeat_sec` or `scheduler_heartbeat_sec`? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6568: [AIRFLOW-5919][part of AIRFLOW-5893][depends on AIRFLOW-5926] Group tests for the Users/Roles/Perms commands
codecov-io edited a comment on issue #6568: [AIRFLOW-5919][part of AIRFLOW-5893][depends on AIRFLOW-5926] Group tests for the Users/Roles/Perms commands URL: https://github.com/apache/airflow/pull/6568#issuecomment-553615632 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6568?src=pr=h1) Report > Merging [#6568](https://codecov.io/gh/apache/airflow/pull/6568?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/d633d3ac44c395e6c43cd388f98fba1ce1c435a3?src=pr=desc) will **decrease** coverage by `0.3%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6568?src=pr=tree) ```diff @@Coverage Diff@@ ## master #6568 +/- ## = - Coverage 83.81% 83.5% -0.31% = Files 645 645 Lines 37356 37356 = - Hits31309 31195 -114 - Misses 60476161 +114 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6568?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6568/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `100% <0%> (ø)` | :arrow_up: | | [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6568/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `100% <0%> (ø)` | :arrow_up: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6568/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `100% <0%> (ø)` | :arrow_up: | | [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/6568/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `100% <0%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6568/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6568/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6568/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/6568/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5) | `76.11% <0%> (ø)` | :arrow_up: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6568/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6568/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `77.14% <0%> (-21.43%)` | :arrow_down: | | ... and [16 more](https://codecov.io/gh/apache/airflow/pull/6568/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6568?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6568?src=pr=footer). Last update [d633d3a...cfa8764](https://codecov.io/gh/apache/airflow/pull/6568?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6568: [AIRFLOW-5919][part of AIRFLOW-5893][depends on AIRFLOW-5926] Group tests for the Users/Roles/Perms commands
codecov-io commented on issue #6568: [AIRFLOW-5919][part of AIRFLOW-5893][depends on AIRFLOW-5926] Group tests for the Users/Roles/Perms commands URL: https://github.com/apache/airflow/pull/6568#issuecomment-553615632 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6568?src=pr=h1) Report > Merging [#6568](https://codecov.io/gh/apache/airflow/pull/6568?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/d633d3ac44c395e6c43cd388f98fba1ce1c435a3?src=pr=desc) will **decrease** coverage by `0.88%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6568?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6568 +/- ## == - Coverage 83.81% 82.92% -0.89% == Files 645 645 Lines 3735637356 == - Hits3130930979 -330 - Misses 6047 6377 +330 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6568?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6568/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6568/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6568/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/6568/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6568/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6568/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6568/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/6568/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5) | `41.04% <0%> (-35.08%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6568/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6568/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `77.14% <0%> (-21.43%)` | :arrow_down: | | ... and [16 more](https://codecov.io/gh/apache/airflow/pull/6568/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6568?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6568?src=pr=footer). Last update [d633d3a...cfa8764](https://codecov.io/gh/apache/airflow/pull/6568?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6580: [AIRFLOW-5925] Relax funcsigs version requirement
codecov-io commented on issue #6580: [AIRFLOW-5925] Relax funcsigs version requirement URL: https://github.com/apache/airflow/pull/6580#issuecomment-553608441 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6580?src=pr=h1) Report > Merging [#6580](https://codecov.io/gh/apache/airflow/pull/6580?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/d633d3ac44c395e6c43cd388f98fba1ce1c435a3?src=pr=desc) will **decrease** coverage by `0.3%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6580?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6580 +/- ## == - Coverage 83.81% 83.51% -0.31% == Files 645 645 Lines 3735637356 == - Hits3130931196 -113 - Misses 6047 6160 +113 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6580?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6580/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6580/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6580/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6580/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6580/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `77.14% <0%> (-21.43%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6580/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (-3.63%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6580/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `58.31% <0%> (+0.32%)` | :arrow_up: | | [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6580/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `90% <0%> (+5%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6580?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6580?src=pr=footer). Last update [d633d3a...809484c](https://codecov.io/gh/apache/airflow/pull/6580?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6579: [AIRFLOW-5926] Fix race-condition in TestCliWebServer
codecov-io edited a comment on issue #6579: [AIRFLOW-5926] Fix race-condition in TestCliWebServer URL: https://github.com/apache/airflow/pull/6579#issuecomment-553599058 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6579?src=pr=h1) Report > Merging [#6579](https://codecov.io/gh/apache/airflow/pull/6579?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/d633d3ac44c395e6c43cd388f98fba1ce1c435a3?src=pr=desc) will **decrease** coverage by `0.31%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6579?src=pr=tree) ```diff @@Coverage Diff@@ ## master #6579 +/- ## = - Coverage 83.81% 83.5% -0.32% = Files 645 645 Lines 37356 37356 = - Hits31309 31193 -116 - Misses 60476163 +116 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6579?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6579/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `100% <0%> (ø)` | :arrow_up: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6579/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `100% <0%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6579/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6579/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6579/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6579/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6579/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `77.14% <0%> (-21.43%)` | :arrow_down: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6579/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `93.22% <0%> (ø)` | :arrow_up: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6579/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (-3.63%)` | :arrow_down: | | [airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/6579/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5) | `77.6% <0%> (ø)` | :arrow_up: | | ... and [4 more](https://codecov.io/gh/apache/airflow/pull/6579/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6579?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6579?src=pr=footer). Last update [d633d3a...27d383a](https://codecov.io/gh/apache/airflow/pull/6579?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6579: [AIRFLOW-5926] Fix race-condition in TestCliWebServer
codecov-io commented on issue #6579: [AIRFLOW-5926] Fix race-condition in TestCliWebServer URL: https://github.com/apache/airflow/pull/6579#issuecomment-553599058 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6579?src=pr=h1) Report > Merging [#6579](https://codecov.io/gh/apache/airflow/pull/6579?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/d633d3ac44c395e6c43cd388f98fba1ce1c435a3?src=pr=desc) will **decrease** coverage by `0.55%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6579?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6579 +/- ## == - Coverage 83.81% 83.26% -0.56% == Files 645 645 Lines 3735637356 == - Hits3130931103 -206 - Misses 6047 6253 +206 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6579?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6579/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6579/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6579/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6579/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6579/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6579/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6579/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `77.14% <0%> (-21.43%)` | :arrow_down: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6579/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `86.44% <0%> (-6.78%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6579/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (-3.63%)` | :arrow_down: | | [airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/6579/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5) | `75.82% <0%> (-1.79%)` | :arrow_down: | | ... and [4 more](https://codecov.io/gh/apache/airflow/pull/6579/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6579?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6579?src=pr=footer). Last update [d633d3a...27d383a](https://codecov.io/gh/apache/airflow/pull/6579?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6565: [AIRFLOW-5909] Enable mapping to BYTEs type to sql_to_gcs operator
codecov-io edited a comment on issue #6565: [AIRFLOW-5909] Enable mapping to BYTEs type to sql_to_gcs operator URL: https://github.com/apache/airflow/pull/6565#issuecomment-553578834 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6565?src=pr=h1) Report > Merging [#6565](https://codecov.io/gh/apache/airflow/pull/6565?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/d633d3ac44c395e6c43cd388f98fba1ce1c435a3?src=pr=desc) will **decrease** coverage by `0.3%`. > The diff coverage is `75%`. [](https://codecov.io/gh/apache/airflow/pull/6565?src=pr=tree) ```diff @@Coverage Diff@@ ## master #6565 +/- ## = - Coverage 83.81% 83.5% -0.31% = Files 645 645 Lines 37356 37362 +6 = - Hits31309 31201 -108 - Misses 60476161 +114 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6565?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/operators/sql\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/6565/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvc3FsX3RvX2djcy5weQ==) | `90.65% <75%> (-1.43%)` | :arrow_down: | | [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6565/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `100% <0%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6565/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6565/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6565/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6565/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6565/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `77.14% <0%> (-21.43%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6565/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (-3.63%)` | :arrow_down: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6565/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `93.22% <0%> (ø)` | :arrow_up: | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/6565/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `91.52% <0%> (ø)` | :arrow_up: | | ... and [3 more](https://codecov.io/gh/apache/airflow/pull/6565/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6565?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6565?src=pr=footer). Last update [d633d3a...d42588f](https://codecov.io/gh/apache/airflow/pull/6565?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6565: [AIRFLOW-5909] Enable mapping to BYTEs type to sql_to_gcs operator
codecov-io commented on issue #6565: [AIRFLOW-5909] Enable mapping to BYTEs type to sql_to_gcs operator URL: https://github.com/apache/airflow/pull/6565#issuecomment-553578834 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6565?src=pr=h1) Report > Merging [#6565](https://codecov.io/gh/apache/airflow/pull/6565?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/d633d3ac44c395e6c43cd388f98fba1ce1c435a3?src=pr=desc) will **decrease** coverage by `0.37%`. > The diff coverage is `75%`. [](https://codecov.io/gh/apache/airflow/pull/6565?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6565 +/- ## == - Coverage 83.81% 83.43% -0.38% == Files 645 645 Lines 3735637362 +6 == - Hits3130931173 -136 - Misses 6047 6189 +142 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6565?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/operators/sql\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/6565/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvc3FsX3RvX2djcy5weQ==) | `90.65% <75%> (-1.43%)` | :arrow_down: | | [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6565/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6565/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6565/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6565/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6565/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6565/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `77.14% <0%> (-21.43%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6565/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (-3.63%)` | :arrow_down: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6565/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `91.52% <0%> (-1.7%)` | :arrow_down: | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/6565/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `89.83% <0%> (-1.7%)` | :arrow_down: | | ... and [3 more](https://codecov.io/gh/apache/airflow/pull/6565/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6565?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6565?src=pr=footer). Last update [d633d3a...d42588f](https://codecov.io/gh/apache/airflow/pull/6565?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6579: [AIRFLOW-5926] Fix race-condition in TestCliWebServer
mik-laj commented on a change in pull request #6579: [AIRFLOW-5926] Fix
race-condition in TestCliWebServer
URL: https://github.com/apache/airflow/pull/6579#discussion_r345963424
##
File path: tests/cli/test_cli.py
##
@@ -761,6 +761,31 @@ class TestCliWebServer(unittest.TestCase):
def setUpClass(cls):
cls.parser = cli.CLIFactory.get_parser()
+def setUp(self) -> None:
+self._check_processes()
+self._clean_pidfiles()
+
+def _check_processes(self):
+try:
+# Confirm that webserver hasn't been launched.
+# pgrep returns exit status 1 if no process matched.
+self.assertEqual(1, subprocess.Popen(["pgrep", "-f", "-c",
"airflow webserver"]).wait())
+self.assertEqual(1, subprocess.Popen(["pgrep", "-c",
"gunicorn"]).wait())
+except: # noqa: E722
+subprocess.Popen(["ps", "-ax"]).wait()
+raise
+
+def tearDown(self) -> None:
+self._check_processes()
+
+def _clean_pidfiles(self):
+pidfile_webserver = cli.setup_locations("webserver")[0]
+pidfile_monitor = cli.setup_locations("webserver-monitor")[0]
Review comment:
Sometimes these files may exist if the process was killed by ``kill -9``
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6579: [AIRFLOW-5926] Fix race-condition in TestCliWebServer
mik-laj commented on a change in pull request #6579: [AIRFLOW-5926] Fix
race-condition in TestCliWebServer
URL: https://github.com/apache/airflow/pull/6579#discussion_r345963424
##
File path: tests/cli/test_cli.py
##
@@ -761,6 +761,31 @@ class TestCliWebServer(unittest.TestCase):
def setUpClass(cls):
cls.parser = cli.CLIFactory.get_parser()
+def setUp(self) -> None:
+self._check_processes()
+self._clean_pidfiles()
+
+def _check_processes(self):
+try:
+# Confirm that webserver hasn't been launched.
+# pgrep returns exit status 1 if no process matched.
+self.assertEqual(1, subprocess.Popen(["pgrep", "-f", "-c",
"airflow webserver"]).wait())
+self.assertEqual(1, subprocess.Popen(["pgrep", "-c",
"gunicorn"]).wait())
+except: # noqa: E722
+subprocess.Popen(["ps", "-ax"]).wait()
+raise
+
+def tearDown(self) -> None:
+self._check_processes()
+
+def _clean_pidfiles(self):
+pidfile_webserver = cli.setup_locations("webserver")[0]
+pidfile_monitor = cli.setup_locations("webserver-monitor")[0]
Review comment:
Sometimes these files may exist if the test was killed by ``kill -9``
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-5925) Upgrade funcsigs dependency
[
https://issues.apache.org/jira/browse/AIRFLOW-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973631#comment-16973631
]
ASF GitHub Bot commented on AIRFLOW-5925:
-
aoen commented on pull request #6580: [AIRFLOW-5925] Relax funcsigs version
requirement
URL: https://github.com/apache/airflow/pull/6580
Make sure you have checked _all_ steps below.
### Jira
- [X] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5925
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [X] Here are some details about my PR, including screenshots of any UI
changes:
Loosen funcsigs version range restrictions to allow it to be upgraded.
### Tests
- [X] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Just allowing a dependency to be upgraded, no additional tests necessary,
especially since it's the same major version so should be backwards compatible.
### Commits
- [X] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ X In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
N/A
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Upgrade funcsigs dependency
> ---
>
> Key: AIRFLOW-5925
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5925
> Project: Apache Airflow
> Issue Type: Bug
> Components: dependencies
>Affects Versions: 1.10.6
>Reporter: Dan Davydov
>Assignee: Dan Davydov
>Priority: Minor
>
> Loosen funcsigs version range restrictions to allow it to be upgraded.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] aoen opened a new pull request #6580: [AIRFLOW-5925] Relax funcsigs version requirement
aoen opened a new pull request #6580: [AIRFLOW-5925] Relax funcsigs version
requirement
URL: https://github.com/apache/airflow/pull/6580
Make sure you have checked _all_ steps below.
### Jira
- [X] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5925
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [X] Here are some details about my PR, including screenshots of any UI
changes:
Loosen funcsigs version range restrictions to allow it to be upgraded.
### Tests
- [X] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Just allowing a dependency to be upgraded, no additional tests necessary,
especially since it's the same major version so should be backwards compatible.
### Commits
- [X] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ X In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
N/A
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6576: [AIRFLOW-5922] Add option to specify the mysql client library used in MySqlHook
codecov-io edited a comment on issue #6576: [AIRFLOW-5922] Add option to specify the mysql client library used in MySqlHook URL: https://github.com/apache/airflow/pull/6576#issuecomment-553564200 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6576?src=pr=h1) Report > Merging [#6576](https://codecov.io/gh/apache/airflow/pull/6576?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/d633d3ac44c395e6c43cd388f98fba1ce1c435a3?src=pr=desc) will **decrease** coverage by `0.32%`. > The diff coverage is `57.14%`. [](https://codecov.io/gh/apache/airflow/pull/6576?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6576 +/- ## == - Coverage 83.81% 83.49% -0.33% == Files 645 645 Lines 3735637372 +16 == - Hits3130931202 -107 - Misses 6047 6170 +123 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6576?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/hooks/mysql\_hook.py](https://codecov.io/gh/apache/airflow/pull/6576/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9teXNxbF9ob29rLnB5) | `83.52% <57.14%> (-9.23%)` | :arrow_down: | | [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6576/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `100% <0%> (ø)` | :arrow_up: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6576/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `100% <0%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6576/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6576/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6576/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6576/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6576/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `77.14% <0%> (-21.43%)` | :arrow_down: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6576/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `93.22% <0%> (ø)` | :arrow_up: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6576/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (-3.63%)` | :arrow_down: | | ... and [5 more](https://codecov.io/gh/apache/airflow/pull/6576/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6576?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6576?src=pr=footer). Last update [d633d3a...f463d13](https://codecov.io/gh/apache/airflow/pull/6576?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6579: [AIRFLOW-5926] Fix race-condition in TestCliWebServer
mik-laj commented on a change in pull request #6579: [AIRFLOW-5926] Fix
race-condition in TestCliWebServer
URL: https://github.com/apache/airflow/pull/6579#discussion_r345960636
##
File path: tests/cli/test_cli.py
##
@@ -801,30 +817,24 @@ def test_cli_webserver_foreground_with_pid(self):
@unittest.skipIf("TRAVIS" in os.environ and bool(os.environ["TRAVIS"]),
"Skipping test due to lack of required file permission")
def test_cli_webserver_background(self):
-# Confirm that webserver hasn't been launched.
-self.assertEqual(1, subprocess.Popen(["pgrep", "-c",
"airflow"]).wait())
-self.assertEqual(1, subprocess.Popen(["pgrep", "-c",
"gunicorn"]).wait())
+pidfile_webserver = cli.setup_locations("webserver")[0]
+pidfile_monitor = cli.setup_locations("webserver-monitor")[0]
-# Run webserver in background.
+# Run webserver as deamon in background. Note that the wait method is
not called.
subprocess.Popen(["airflow", "webserver", "-D"])
-pidfile = cli.setup_locations("webserver")[0]
-self._wait_pidfile(pidfile)
+
+pid_monitor = self._wait_pidfile(pidfile_monitor)
+self._wait_pidfile(pidfile_webserver)
Review comment:
If we want to do pgrep for gunicorn then we need to check if procces has
started
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6576: [AIRFLOW-5922] Add option to specify the mysql client library used in MySqlHook
codecov-io commented on issue #6576: [AIRFLOW-5922] Add option to specify the mysql client library used in MySqlHook URL: https://github.com/apache/airflow/pull/6576#issuecomment-553564200 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6576?src=pr=h1) Report > Merging [#6576](https://codecov.io/gh/apache/airflow/pull/6576?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/d633d3ac44c395e6c43cd388f98fba1ce1c435a3?src=pr=desc) will **decrease** coverage by `0.54%`. > The diff coverage is `57.14%`. [](https://codecov.io/gh/apache/airflow/pull/6576?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6576 +/- ## == - Coverage 83.81% 83.26% -0.55% == Files 645 645 Lines 3735637372 +16 == - Hits3130931117 -192 - Misses 6047 6255 +208 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6576?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/hooks/mysql\_hook.py](https://codecov.io/gh/apache/airflow/pull/6576/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9teXNxbF9ob29rLnB5) | `83.52% <57.14%> (-9.23%)` | :arrow_down: | | [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6576/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6576/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6576/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6576/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6576/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6576/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6576/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `77.14% <0%> (-21.43%)` | :arrow_down: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6576/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `86.44% <0%> (-6.78%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6576/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (-3.63%)` | :arrow_down: | | ... and [5 more](https://codecov.io/gh/apache/airflow/pull/6576/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6576?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6576?src=pr=footer). Last update [d633d3a...f463d13](https://codecov.io/gh/apache/airflow/pull/6576?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6579: [AIRFLOW-5926] Fix race-condition in TestCliWebServer
mik-laj commented on a change in pull request #6579: [AIRFLOW-5926] Fix
race-condition in TestCliWebServer
URL: https://github.com/apache/airflow/pull/6579#discussion_r345960220
##
File path: tests/cli/test_cli.py
##
@@ -801,30 +817,24 @@ def test_cli_webserver_foreground_with_pid(self):
@unittest.skipIf("TRAVIS" in os.environ and bool(os.environ["TRAVIS"]),
"Skipping test due to lack of required file permission")
def test_cli_webserver_background(self):
-# Confirm that webserver hasn't been launched.
-self.assertEqual(1, subprocess.Popen(["pgrep", "-c",
"airflow"]).wait())
-self.assertEqual(1, subprocess.Popen(["pgrep", "-c",
"gunicorn"]).wait())
+pidfile_webserver = cli.setup_locations("webserver")[0]
+pidfile_monitor = cli.setup_locations("webserver-monitor")[0]
-# Run webserver in background.
+# Run webserver as deamon in background. Note that the wait method is
not called.
subprocess.Popen(["airflow", "webserver", "-D"])
-pidfile = cli.setup_locations("webserver")[0]
-self._wait_pidfile(pidfile)
+
+pid_monitor = self._wait_pidfile(pidfile_monitor)
+self._wait_pidfile(pidfile_webserver)
# Assert that gunicorn and its monitor are launched.
-self.assertEqual(0, subprocess.Popen(["pgrep", "-c",
"airflow"]).wait())
+self.assertEqual(0, subprocess.Popen(["pgrep", "-f", "-c", "airflow
webserver"]).wait())
self.assertEqual(0, subprocess.Popen(["pgrep", "-c",
"gunicorn"]).wait())
# Terminate monitor process.
-pidfile = cli.setup_locations("webserver-monitor")[0]
-pid = self._wait_pidfile(pidfile)
-proc = psutil.Process(pid)
+proc = psutil.Process(pid_monitor)
Review comment:
It is necessary to check whether the monitor has started earlier, otherwise
we are not sure if pgrep will be successful.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (AIRFLOW-5925) Upgrade funcsigs dependency
[ https://issues.apache.org/jira/browse/AIRFLOW-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dan Davydov updated AIRFLOW-5925: - Summary: Upgrade funcsigs dependency (was: Upgrade some dependencies) > Upgrade funcsigs dependency > --- > > Key: AIRFLOW-5925 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5925 > Project: Apache Airflow > Issue Type: Bug > Components: dependencies >Affects Versions: 1.10.6 >Reporter: Dan Davydov >Assignee: Dan Davydov >Priority: Minor > > Loosen some version range restrictions to allow upgrades for dependencies of > funcsigs and psutils. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5925) Upgrade funcsigs dependency
[ https://issues.apache.org/jira/browse/AIRFLOW-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dan Davydov updated AIRFLOW-5925: - Description: Loosen funcsigs version range restrictions to allow it to be upgraded. (was: Loosen some version range restrictions to allow upgrades for dependencies of funcsigs and psutils.) > Upgrade funcsigs dependency > --- > > Key: AIRFLOW-5925 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5925 > Project: Apache Airflow > Issue Type: Bug > Components: dependencies >Affects Versions: 1.10.6 >Reporter: Dan Davydov >Assignee: Dan Davydov >Priority: Minor > > Loosen funcsigs version range restrictions to allow it to be upgraded. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] mik-laj commented on a change in pull request #6579: [AIRFLOW-5926] Fix race-condition in TestCliWebServer
mik-laj commented on a change in pull request #6579: [AIRFLOW-5926] Fix
race-condition in TestCliWebServer
URL: https://github.com/apache/airflow/pull/6579#discussion_r345959254
##
File path: tests/cli/test_cli.py
##
@@ -801,30 +817,24 @@ def test_cli_webserver_foreground_with_pid(self):
@unittest.skipIf("TRAVIS" in os.environ and bool(os.environ["TRAVIS"]),
"Skipping test due to lack of required file permission")
def test_cli_webserver_background(self):
-# Confirm that webserver hasn't been launched.
-self.assertEqual(1, subprocess.Popen(["pgrep", "-c",
"airflow"]).wait())
-self.assertEqual(1, subprocess.Popen(["pgrep", "-c",
"gunicorn"]).wait())
+pidfile_webserver = cli.setup_locations("webserver")[0]
+pidfile_monitor = cli.setup_locations("webserver-monitor")[0]
-# Run webserver in background.
+# Run webserver as deamon in background. Note that the wait method is
not called.
subprocess.Popen(["airflow", "webserver", "-D"])
-pidfile = cli.setup_locations("webserver")[0]
-self._wait_pidfile(pidfile)
+
+pid_monitor = self._wait_pidfile(pidfile_monitor)
+self._wait_pidfile(pidfile_webserver)
# Assert that gunicorn and its monitor are launched.
-self.assertEqual(0, subprocess.Popen(["pgrep", "-c",
"airflow"]).wait())
+self.assertEqual(0, subprocess.Popen(["pgrep", "-f", "-c", "airflow
webserver"]).wait())
Review comment:
Daemon is not found without the -f option. Sometimes pgrep would find the
user process (824 linie) and report success, but if it didn't, the test would
fail.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-5926) Fix race-condition in TestCliWebServer
[
https://issues.apache.org/jira/browse/AIRFLOW-5926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973617#comment-16973617
]
ASF GitHub Bot commented on AIRFLOW-5926:
-
mik-laj commented on pull request #6579: [AIRFLOW-5926] Fix race-condition in
TestCliWebServer
URL: https://github.com/apache/airflow/pull/6579
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5926
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Fix race-condition in TestCliWebServer
> ---
>
> Key: AIRFLOW-5926
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5926
> Project: Apache Airflow
> Issue Type: Bug
> Components: cli
>Affects Versions: 1.10.6
>Reporter: Kamil Bregula
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] mik-laj opened a new pull request #6579: [AIRFLOW-5926] Fix race-condition in TestCliWebServer
mik-laj opened a new pull request #6579: [AIRFLOW-5926] Fix race-condition in
TestCliWebServer
URL: https://github.com/apache/airflow/pull/6579
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5926
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6575: [AIRFLOW-5921] Add bulk_load_custom to MySqlHook
codecov-io edited a comment on issue #6575: [AIRFLOW-5921] Add bulk_load_custom to MySqlHook URL: https://github.com/apache/airflow/pull/6575#issuecomment-553557551 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6575?src=pr=h1) Report > Merging [#6575](https://codecov.io/gh/apache/airflow/pull/6575?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/d633d3ac44c395e6c43cd388f98fba1ce1c435a3?src=pr=desc) will **decrease** coverage by `0.31%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6575?src=pr=tree) ```diff @@Coverage Diff@@ ## master #6575 +/- ## = - Coverage 83.81% 83.5% -0.32% = Files 645 645 Lines 37356 37362 +6 = - Hits31309 31198 -111 - Misses 60476164 +117 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6575?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/hooks/mysql\_hook.py](https://codecov.io/gh/apache/airflow/pull/6575/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9teXNxbF9ob29rLnB5) | `93.33% <100%> (+0.57%)` | :arrow_up: | | [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6575/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `100% <0%> (ø)` | :arrow_up: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6575/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `100% <0%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6575/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6575/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6575/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6575/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6575/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `77.14% <0%> (-21.43%)` | :arrow_down: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6575/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `93.22% <0%> (ø)` | :arrow_up: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6575/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (-3.63%)` | :arrow_down: | | ... and [5 more](https://codecov.io/gh/apache/airflow/pull/6575/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6575?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6575?src=pr=footer). Last update [d633d3a...7a91fbf](https://codecov.io/gh/apache/airflow/pull/6575?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6575: [AIRFLOW-5921] Add bulk_load_custom to MySqlHook
codecov-io commented on issue #6575: [AIRFLOW-5921] Add bulk_load_custom to MySqlHook URL: https://github.com/apache/airflow/pull/6575#issuecomment-553557551 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6575?src=pr=h1) Report > Merging [#6575](https://codecov.io/gh/apache/airflow/pull/6575?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/d633d3ac44c395e6c43cd388f98fba1ce1c435a3?src=pr=desc) will **decrease** coverage by `0.54%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6575?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6575 +/- ## == - Coverage 83.81% 83.26% -0.55% == Files 645 645 Lines 3735637362 +6 == - Hits3130931109 -200 - Misses 6047 6253 +206 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6575?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/hooks/mysql\_hook.py](https://codecov.io/gh/apache/airflow/pull/6575/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9teXNxbF9ob29rLnB5) | `93.33% <100%> (+0.57%)` | :arrow_up: | | [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6575/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6575/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6575/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6575/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6575/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6575/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6575/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `77.14% <0%> (-21.43%)` | :arrow_down: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6575/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `86.44% <0%> (-6.78%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6575/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (-3.63%)` | :arrow_down: | | ... and [5 more](https://codecov.io/gh/apache/airflow/pull/6575/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6575?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6575?src=pr=footer). Last update [d633d3a...7a91fbf](https://codecov.io/gh/apache/airflow/pull/6575?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-5926) Fix race-condition in TestCliWebServer
Kamil Bregula created AIRFLOW-5926: -- Summary: Fix race-condition in TestCliWebServer Key: AIRFLOW-5926 URL: https://issues.apache.org/jira/browse/AIRFLOW-5926 Project: Apache Airflow Issue Type: Bug Components: cli Affects Versions: 1.10.6 Reporter: Kamil Bregula -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #6574: [AIRFLOW-5924] Automatically unify bucket name and key in S3Hook
codecov-io edited a comment on issue #6574: [AIRFLOW-5924] Automatically unify bucket name and key in S3Hook URL: https://github.com/apache/airflow/pull/6574#issuecomment-553548307 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6574?src=pr=h1) Report > Merging [#6574](https://codecov.io/gh/apache/airflow/pull/6574?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/d633d3ac44c395e6c43cd388f98fba1ce1c435a3?src=pr=desc) will **decrease** coverage by `0.3%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6574?src=pr=tree) ```diff @@Coverage Diff@@ ## master #6574 +/- ## = - Coverage 83.81% 83.5% -0.31% = Files 645 645 Lines 37356 37367 +11 = - Hits31309 31205 -104 - Misses 60476162 +115 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6574?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/providers/amazon/aws/hooks/s3.py](https://codecov.io/gh/apache/airflow/pull/6574/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9ob29rcy9zMy5weQ==) | `96.48% <100%> (+0.73%)` | :arrow_up: | | [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6574/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `100% <0%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6574/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6574/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6574/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6574/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6574/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `77.14% <0%> (-21.43%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6574/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (-3.63%)` | :arrow_down: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6574/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `93.22% <0%> (ø)` | :arrow_up: | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/6574/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `91.52% <0%> (ø)` | :arrow_up: | | ... and [4 more](https://codecov.io/gh/apache/airflow/pull/6574/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6574?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6574?src=pr=footer). Last update [d633d3a...0c29fb1](https://codecov.io/gh/apache/airflow/pull/6574?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6574: [AIRFLOW-5924] Automatically unify bucket name and key in S3Hook
codecov-io commented on issue #6574: [AIRFLOW-5924] Automatically unify bucket name and key in S3Hook URL: https://github.com/apache/airflow/pull/6574#issuecomment-553548307 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6574?src=pr=h1) Report > Merging [#6574](https://codecov.io/gh/apache/airflow/pull/6574?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/d633d3ac44c395e6c43cd388f98fba1ce1c435a3?src=pr=desc) will **decrease** coverage by `0.37%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6574?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6574 +/- ## == - Coverage 83.81% 83.43% -0.38% == Files 645 645 Lines 3735637367 +11 == - Hits3130931179 -130 - Misses 6047 6188 +141 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6574?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/providers/amazon/aws/hooks/s3.py](https://codecov.io/gh/apache/airflow/pull/6574/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9ob29rcy9zMy5weQ==) | `96.48% <100%> (+0.73%)` | :arrow_up: | | [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6574/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6574/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6574/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6574/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6574/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6574/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `77.14% <0%> (-21.43%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6574/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (-3.63%)` | :arrow_down: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6574/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `91.52% <0%> (-1.7%)` | :arrow_down: | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/6574/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `89.83% <0%> (-1.7%)` | :arrow_down: | | ... and [4 more](https://codecov.io/gh/apache/airflow/pull/6574/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6574?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6574?src=pr=footer). Last update [d633d3a...0c29fb1](https://codecov.io/gh/apache/airflow/pull/6574?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-5925) Upgrade some dependencies
Dan Davydov created AIRFLOW-5925: Summary: Upgrade some dependencies Key: AIRFLOW-5925 URL: https://issues.apache.org/jira/browse/AIRFLOW-5925 Project: Apache Airflow Issue Type: Bug Components: dependencies Affects Versions: 1.10.6 Reporter: Dan Davydov Assignee: Dan Davydov Loosen some version range restrictions to allow upgrades for dependencies of funcsigs and psutils. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5249) BigQueryCheckOperator fails for datasets outside of 'US' region
[
https://issues.apache.org/jira/browse/AIRFLOW-5249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973575#comment-16973575
]
Paul De commented on AIRFLOW-5249:
--
I can use Dataset location = EU and now seems to work so not an issue for me
anymore
> BigQueryCheckOperator fails for datasets outside of 'US' region
> ---
>
> Key: AIRFLOW-5249
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5249
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: 1.10.2
>Reporter: Michael
>Assignee: Thomas Pilewicz
>Priority: Blocker
>
> When I try to use the BigQueryCheckOperator or BigQueryValueCheckOperator on
> a dataset that is not in the 'US' location my task fails with the following
> error
> {code:java}
> [2019-08-15 07:26:19,378] {__init__.py:1580} ERROR - BigQuery job status
> check failed. Final error was: 404
> Traceback (most recent call last):
> File
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/bigquery_hook.py",
> line 1241, in run_with_configuration
> jobId=self.running_job_id).execute()
> File "/usr/local/lib/python3.6/site-packages/googleapiclient/_helpers.py",
> line 130, in positional_wrapper
> return wrapped(*args, **kwargs)
> File "/usr/local/lib/python3.6/site-packages/googleapiclient/http.py", line
> 855, in execute
> raise HttpError(resp, content, uri=self.uri)
> googleapiclient.errors.HttpError: https://www.googleapis.com/bigquery/v2/projects/anz-data-cde-airflow/jobs/job_ISDpiVtd7U1p-6N9wT378LfwoFHc?alt=json
> returned "Not found: Job
> anz-data-cde-airflow:job_ISDpiVtd7U1p-6N9wT378LfwoFHc">
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File "/usr/local/lib/python3.6/site-packages/airflow/models/__init__.py",
> line 1441, in _run_raw_task
> result = task_copy.execute(context=context)
> File
> "/usr/local/lib/python3.6/site-packages/airflow/operators/check_operator.py",
> line 81, in execute
> records = self.get_db_hook().get_first(self.sql)
> File "/usr/local/lib/python3.6/site-packages/airflow/hooks/dbapi_hook.py",
> line 138, in get_first
> cur.execute(sql)
> File
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/bigquery_hook.py",
> line 1821, in execute
> self.job_id = self.run_query(sql)
> File
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/bigquery_hook.py",
> line 849, in run_query
> return self.run_with_configuration(configuration)
> File
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/bigquery_hook.py",
> line 1263, in run_with_configuration
> format(err.resp.status))
> Exception: BigQuery job status check failed. Final error was: 404
> [2019-08-15 07:26:19,388] {__init__.py:1611} INFO - Marking task as FAILED.
> {code}
> This is the same error I get when I try to run the BigQuery operator without
> specifying a location. When I run the same operator on a dataset that is in
> the US region It succeeds.
> The BigQueryCheckOperator does not accept a location as one of its arguments
> and does not pass a location to the BigQueryHook, I believe this is the
> source of the problem.
>
> I realise a task (AIRFLOW-3601) was already created to fix a similar issue to
> this one, but the referenced task calls out the two operators I'm having an
> issue with as out of scope and after commenting on that task I have not
> received a response.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] codecov-io commented on issue #6571: [AIRFLOW-5923] Use absolute paths in GCP system tests
codecov-io commented on issue #6571: [AIRFLOW-5923] Use absolute paths in GCP system tests URL: https://github.com/apache/airflow/pull/6571#issuecomment-553517629 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6571?src=pr=h1) Report > Merging [#6571](https://codecov.io/gh/apache/airflow/pull/6571?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/d633d3ac44c395e6c43cd388f98fba1ce1c435a3?src=pr=desc) will **decrease** coverage by `0.3%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6571?src=pr=tree) ```diff @@Coverage Diff@@ ## master #6571 +/- ## = - Coverage 83.81% 83.5% -0.31% = Files 645 645 Lines 37356 37356 = - Hits31309 31195 -114 - Misses 60476161 +114 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6571?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6571/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6571/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6571/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6571/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6571/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `77.14% <0%> (-21.43%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6571/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (-3.63%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6571/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `58.15% <0%> (+0.16%)` | :arrow_up: | | [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6571/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `90% <0%> (+5%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6571?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6571?src=pr=footer). Last update [d633d3a...699d640](https://codecov.io/gh/apache/airflow/pull/6571?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-5924) Automatically unify bucket name and key in S3Hook
[ https://issues.apache.org/jira/browse/AIRFLOW-5924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-5924: -- Fix Version/s: 2.0.0 > Automatically unify bucket name and key in S3Hook > - > > Key: AIRFLOW-5924 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5924 > Project: Apache Airflow > Issue Type: New Feature > Components: aws, hooks >Affects Versions: 2.0.0, 1.10.6 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (AIRFLOW-5921) Add bulk_load_custom to MySqlHook
[ https://issues.apache.org/jira/browse/AIRFLOW-5921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-5921 started by Felix Uellendall. - > Add bulk_load_custom to MySqlHook > - > > Key: AIRFLOW-5921 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5921 > Project: Apache Airflow > Issue Type: New Feature > Components: hooks >Affects Versions: 2.0.0, 1.10.6 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (AIRFLOW-5924) Automatically unify bucket name and key in S3Hook
[ https://issues.apache.org/jira/browse/AIRFLOW-5924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-5924 started by Felix Uellendall. - > Automatically unify bucket name and key in S3Hook > - > > Key: AIRFLOW-5924 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5924 > Project: Apache Airflow > Issue Type: New Feature > Components: aws, hooks >Affects Versions: 2.0.0, 1.10.6 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (AIRFLOW-5922) Add option to specify the mysql client library used in MySqlHook
[ https://issues.apache.org/jira/browse/AIRFLOW-5922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-5922 started by Felix Uellendall. - > Add option to specify the mysql client library used in MySqlHook > > > Key: AIRFLOW-5922 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5922 > Project: Apache Airflow > Issue Type: New Feature > Components: dependencies, hooks >Affects Versions: 2.0.0, 1.10.6 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (AIRFLOW-5908) Add download_file to S3 Hook
[ https://issues.apache.org/jira/browse/AIRFLOW-5908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-5908 started by Felix Uellendall. - > Add download_file to S3 Hook > > > Key: AIRFLOW-5908 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5908 > Project: Apache Airflow > Issue Type: New Feature > Components: aws, hooks >Affects Versions: 2.0.0, 1.10.6 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (AIRFLOW-5907) Add S3 to MySql Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-5907 started by Felix Uellendall. - > Add S3 to MySql Operator > > > Key: AIRFLOW-5907 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5907 > Project: Apache Airflow > Issue Type: New Feature > Components: operators >Affects Versions: 2.0.0, 1.10.6 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5908) Add download_file to S3 Hook
[
https://issues.apache.org/jira/browse/AIRFLOW-5908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973540#comment-16973540
]
ASF GitHub Bot commented on AIRFLOW-5908:
-
feluelle commented on pull request #6578: [DEPENDS ON AIRFLOW-5908][DEPENDS ON
AIRFLOW-5921][AIRFLOW-5907] Add S3 to MySql Operator
URL: https://github.com/apache/airflow/pull/6578
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5907
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add download_file to S3 Hook
>
>
> Key: AIRFLOW-5908
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5908
> Project: Apache Airflow
> Issue Type: New Feature
> Components: aws, hooks
>Affects Versions: 2.0.0, 1.10.6
>Reporter: Felix Uellendall
>Assignee: Felix Uellendall
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] feluelle opened a new pull request #6578: [DEPENDS ON AIRFLOW-5908][DEPENDS ON AIRFLOW-5921][AIRFLOW-5907] Add S3 to MySql Operator
feluelle opened a new pull request #6578: [DEPENDS ON AIRFLOW-5908][DEPENDS ON
AIRFLOW-5921][AIRFLOW-5907] Add S3 to MySql Operator
URL: https://github.com/apache/airflow/pull/6578
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5907
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-5924) Automatically unify bucket name and key in S3Hook
[
https://issues.apache.org/jira/browse/AIRFLOW-5924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973536#comment-16973536
]
ASF GitHub Bot commented on AIRFLOW-5924:
-
feluelle commented on pull request #6577: [DEPENDS ON
AIRFLOW-5924][AIRFLOW-5908] Add download_file to S3 Hook
URL: https://github.com/apache/airflow/pull/6577
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5908
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Automatically unify bucket name and key in S3Hook
> -
>
> Key: AIRFLOW-5924
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5924
> Project: Apache Airflow
> Issue Type: New Feature
> Components: aws, hooks
>Affects Versions: 2.0.0, 1.10.6
>Reporter: Felix Uellendall
>Assignee: Felix Uellendall
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] feluelle opened a new pull request #6577: [DEPENDS ON AIRFLOW-5924][AIRFLOW-5908] Add download_file to S3 Hook
feluelle opened a new pull request #6577: [DEPENDS ON
AIRFLOW-5924][AIRFLOW-5908] Add download_file to S3 Hook
URL: https://github.com/apache/airflow/pull/6577
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5908
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run
codecov-io edited a comment on issue #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run URL: https://github.com/apache/airflow/pull/4751#issuecomment-466029246 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4751?src=pr=h1) Report > Merging [#4751](https://codecov.io/gh/apache/airflow/pull/4751?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/d633d3ac44c395e6c43cd388f98fba1ce1c435a3?src=pr=desc) will **decrease** coverage by `0.31%`. > The diff coverage is `85.18%`. [](https://codecov.io/gh/apache/airflow/pull/4751?src=pr=tree) ```diff @@Coverage Diff@@ ## master #4751 +/- ## = - Coverage 83.81% 83.5% -0.32% = Files 645 645 Lines 37356 37379 +23 = - Hits31309 31212 -97 - Misses 60476167 +120 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4751?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `75.07% <100%> (+0.11%)` | :arrow_up: | | [airflow/models/dagrun.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFncnVuLnB5) | `96.6% <100%> (ø)` | :arrow_up: | | [airflow/ti\_deps/dep\_context.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcF9jb250ZXh0LnB5) | `100% <100%> (ø)` | :arrow_up: | | [airflow/ti\_deps/deps/trigger\_rule\_dep.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvdHJpZ2dlcl9ydWxlX2RlcC5weQ==) | `88.42% <80.95%> (-2.37%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `77.14% <0%> (-21.43%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (-3.63%)` | :arrow_down: | | ... and [1 more](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4751?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4751?src=pr=footer). Last update [d633d3a...7a73dcb](https://codecov.io/gh/apache/airflow/pull/4751?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run
codecov-io edited a comment on issue #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run URL: https://github.com/apache/airflow/pull/4751#issuecomment-466029246 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4751?src=pr=h1) Report > Merging [#4751](https://codecov.io/gh/apache/airflow/pull/4751?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/d633d3ac44c395e6c43cd388f98fba1ce1c435a3?src=pr=desc) will **decrease** coverage by `0.31%`. > The diff coverage is `85.18%`. [](https://codecov.io/gh/apache/airflow/pull/4751?src=pr=tree) ```diff @@Coverage Diff@@ ## master #4751 +/- ## = - Coverage 83.81% 83.5% -0.32% = Files 645 645 Lines 37356 37379 +23 = - Hits31309 31212 -97 - Misses 60476167 +120 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4751?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `75.07% <100%> (+0.11%)` | :arrow_up: | | [airflow/models/dagrun.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFncnVuLnB5) | `96.6% <100%> (ø)` | :arrow_up: | | [airflow/ti\_deps/dep\_context.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcF9jb250ZXh0LnB5) | `100% <100%> (ø)` | :arrow_up: | | [airflow/ti\_deps/deps/trigger\_rule\_dep.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvdHJpZ2dlcl9ydWxlX2RlcC5weQ==) | `88.42% <80.95%> (-2.37%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `77.14% <0%> (-21.43%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (-3.63%)` | :arrow_down: | | ... and [1 more](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4751?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4751?src=pr=footer). Last update [d633d3a...7a73dcb](https://codecov.io/gh/apache/airflow/pull/4751?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5922) Add option to specify the mysql client library used in MySqlHook
[
https://issues.apache.org/jira/browse/AIRFLOW-5922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973523#comment-16973523
]
ASF GitHub Bot commented on AIRFLOW-5922:
-
feluelle commented on pull request #6576: [AIRFLOW-5922] Add option to specify
the mysql client library used in MySqlHook
URL: https://github.com/apache/airflow/pull/6576
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5922
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
- add mysql-connector-python library
- fix pylint issues
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add option to specify the mysql client library used in MySqlHook
>
>
> Key: AIRFLOW-5922
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5922
> Project: Apache Airflow
> Issue Type: New Feature
> Components: dependencies, hooks
>Affects Versions: 2.0.0, 1.10.6
>Reporter: Felix Uellendall
>Assignee: Felix Uellendall
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] feluelle opened a new pull request #6576: [AIRFLOW-5922] Add option to specify the mysql client library used in MySqlHook
feluelle opened a new pull request #6576: [AIRFLOW-5922] Add option to specify
the mysql client library used in MySqlHook
URL: https://github.com/apache/airflow/pull/6576
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5922
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
- add mysql-connector-python library
- fix pylint issues
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-5921) Add bulk_load_custom to MySqlHook
[
https://issues.apache.org/jira/browse/AIRFLOW-5921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973522#comment-16973522
]
ASF GitHub Bot commented on AIRFLOW-5921:
-
feluelle commented on pull request #6575: [AIRFLOW-5921] Add bulk_load_custom
to MySqlHook
URL: https://github.com/apache/airflow/pull/6575
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5921
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add bulk_load_custom to MySqlHook
> -
>
> Key: AIRFLOW-5921
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5921
> Project: Apache Airflow
> Issue Type: New Feature
> Components: hooks
>Affects Versions: 2.0.0, 1.10.6
>Reporter: Felix Uellendall
>Assignee: Felix Uellendall
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] feluelle opened a new pull request #6575: [AIRFLOW-5921] Add bulk_load_custom to MySqlHook
feluelle opened a new pull request #6575: [AIRFLOW-5921] Add bulk_load_custom
to MySqlHook
URL: https://github.com/apache/airflow/pull/6575
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5921
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-5924) Automatically unify bucket name and key in S3Hook
[
https://issues.apache.org/jira/browse/AIRFLOW-5924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973513#comment-16973513
]
ASF GitHub Bot commented on AIRFLOW-5924:
-
feluelle commented on pull request #6574: [AIRFLOW-5924] Automatically unify
bucket name and key in S3Hook
URL: https://github.com/apache/airflow/pull/6574
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5924
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
- change provide_bucket_name to provide bucket name also for function with
keys
- refactoring
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Automatically unify bucket name and key in S3Hook
> -
>
> Key: AIRFLOW-5924
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5924
> Project: Apache Airflow
> Issue Type: New Feature
> Components: aws, hooks
>Affects Versions: 2.0.0, 1.10.6
>Reporter: Felix Uellendall
>Assignee: Felix Uellendall
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] feluelle opened a new pull request #6574: [AIRFLOW-5924] Automatically unify bucket name and key in S3Hook
feluelle opened a new pull request #6574: [AIRFLOW-5924] Automatically unify
bucket name and key in S3Hook
URL: https://github.com/apache/airflow/pull/6574
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5924
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
- change provide_bucket_name to provide bucket name also for function with
keys
- refactoring
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-5904) "Trigger DAG" should not be redirected to the "tree" page but to the "dag_default_view" page
[
https://issues.apache.org/jira/browse/AIRFLOW-5904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973509#comment-16973509
]
ASF GitHub Bot commented on AIRFLOW-5904:
-
GnunuX commented on pull request #6573: [AIRFLOW-5904] "Trigger DAG" should
redirect to the "dag_default_view" page
URL: https://github.com/apache/airflow/pull/6573
…" page
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
"Trigger DAG" should redirect to the "dag_default_view" page.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> "Trigger DAG" should not be redirected to the "tree" page but to the
> "dag_default_view" page
>
>
> Key: AIRFLOW-5904
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5904
> Project: Apache Airflow
> Issue Type: Bug
> Components: configuration
>Affects Versions: 1.10.6
>Reporter: Emmanuel Garette
>Priority: Minor
>
> When we are on the page of "DAG" and we click on "Trigger DAG", we are
> redirected to the page "tree".
> However, I changed the configuration element "dag_default_view" to "graph".
> I think we should be redirected to the configured page name.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] GnunuX opened a new pull request #6573: [AIRFLOW-5904] "Trigger DAG" should redirect to the "dag_default_view" page
GnunuX opened a new pull request #6573: [AIRFLOW-5904] "Trigger DAG" should
redirect to the "dag_default_view" page
URL: https://github.com/apache/airflow/pull/6573
…" page
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
"Trigger DAG" should redirect to the "dag_default_view" page.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (AIRFLOW-5907) Add S3 to MySql Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-5907: -- Summary: Add S3 to MySql Operator (was: Add S3 to Mysql Operator) > Add S3 to MySql Operator > > > Key: AIRFLOW-5907 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5907 > Project: Apache Airflow > Issue Type: New Feature > Components: operators >Affects Versions: 2.0.0, 1.10.6 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5907) Add S3 to Mysql Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-5907: -- Summary: Add S3 to Mysql Operator (was: S3 to Mysql Operator) > Add S3 to Mysql Operator > > > Key: AIRFLOW-5907 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5907 > Project: Apache Airflow > Issue Type: New Feature > Components: operators >Affects Versions: 2.0.0, 1.10.6 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5904) "Trigger DAG" should not be redirected to the "tree" page but to the "dag_default_view" page
[ https://issues.apache.org/jira/browse/AIRFLOW-5904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973504#comment-16973504 ] ASF GitHub Bot commented on AIRFLOW-5904: - GnunuX commented on pull request #6556: [AIRFLOW-5904] "Trigger DAG" should redirect to the "dag_default_view… URL: https://github.com/apache/airflow/pull/6556 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > "Trigger DAG" should not be redirected to the "tree" page but to the > "dag_default_view" page > > > Key: AIRFLOW-5904 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5904 > Project: Apache Airflow > Issue Type: Bug > Components: configuration >Affects Versions: 1.10.6 >Reporter: Emmanuel Garette >Priority: Minor > > When we are on the page of "DAG" and we click on "Trigger DAG", we are > redirected to the page "tree". > However, I changed the configuration element "dag_default_view" to "graph". > I think we should be redirected to the configured page name. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] GnunuX closed pull request #6556: [AIRFLOW-5904] "Trigger DAG" should redirect to the "dag_default_view…
GnunuX closed pull request #6556: [AIRFLOW-5904] "Trigger DAG" should redirect to the "dag_default_view… URL: https://github.com/apache/airflow/pull/6556 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3489) PostgresToGoogleCloudStorageOperator doesn't handle PostgreSQL json properly
[
https://issues.apache.org/jira/browse/AIRFLOW-3489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973493#comment-16973493
]
ASF GitHub Bot commented on AIRFLOW-3489:
-
nuclearpinguin commented on pull request #6572: [AIRFLOW-3489] Improve json
data handling in PostgresToGcs operator
URL: https://github.com/apache/airflow/pull/6572
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3489
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
According to the issue, `PostgresToGoogleCloudStorageOperator` had a problem
with handling postgres json data type in way that would allow to import the
exported file into BigQuery. So I've added a simple check and `json.dumps`.
I've checked if after that the data could be easily transferred to BQ (using
data transfer service) and everything worked. I've added also a system test.
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> PostgresToGoogleCloudStorageOperator doesn't handle PostgreSQL json properly
> -
>
> Key: AIRFLOW-3489
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3489
> Project: Apache Airflow
> Issue Type: Bug
> Components: gcp
>Reporter: Duan Shiqiang
>Priority: Major
> Labels: bigquery
>
> PostgresToGoogleCloudStorageOperator saves json data (postgres json or jsonb)
> as native python types (i.e. dictionary) to gcs new line separated json data.
> But it generates bigquery schema for that field as data type string which
> won't work if user want to import the gcs data into bigquery.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] nuclearpinguin opened a new pull request #6572: [AIRFLOW-3489] Improve json data handling in PostgresToGcs operator
nuclearpinguin opened a new pull request #6572: [AIRFLOW-3489] Improve json
data handling in PostgresToGcs operator
URL: https://github.com/apache/airflow/pull/6572
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3489
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
According to the issue, `PostgresToGoogleCloudStorageOperator` had a problem
with handling postgres json data type in way that would allow to import the
exported file into BigQuery. So I've added a simple check and `json.dumps`.
I've checked if after that the data could be easily transferred to BQ (using
data transfer service) and everything worked. I've added also a system test.
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-5924) Automatically unify bucket name and key in S3Hook
Felix Uellendall created AIRFLOW-5924: - Summary: Automatically unify bucket name and key in S3Hook Key: AIRFLOW-5924 URL: https://issues.apache.org/jira/browse/AIRFLOW-5924 Project: Apache Airflow Issue Type: New Feature Components: aws, hooks Affects Versions: 1.10.6, 2.0.0 Reporter: Felix Uellendall Assignee: Felix Uellendall -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5923) Use absolute paths in GCP system tests
[
https://issues.apache.org/jira/browse/AIRFLOW-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973481#comment-16973481
]
ASF GitHub Bot commented on AIRFLOW-5923:
-
nuclearpinguin commented on pull request #6571: [AIRFLOW-5923] Use absolute
paths in GCP system tests
URL: https://github.com/apache/airflow/pull/6571
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5923
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
In this PR I propose to use absolute path to example DAGs used in GCP system
tests. Used values should work well in breeze. I've also fixed system tests
that were moved to `providers.google.cloud`.
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Use absolute paths in GCP system tests
> --
>
> Key: AIRFLOW-5923
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5923
> Project: Apache Airflow
> Issue Type: Improvement
> Components: gcp, tests
>Affects Versions: 1.10.7
>Reporter: Tomasz Urbaszek
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
