[GitHub] [airflow] RosterIn commented on issue #7398: [AIRFLOW-6505] Let emoji encoded properly for json.dumps()
RosterIn commented on issue #7398: [AIRFLOW-6505] Let emoji encoded properly for json.dumps() URL: https://github.com/apache/airflow/pull/7398#issuecomment-584514650 @damon09273 `airflow/contrib/operators/sql_to_gcs.py` doesn't exist in master (which is why you get conflict) You need to edit: https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/operators/sql_to_gcs.py#L181 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] damon09273 closed pull request #7398: [AIRFLOW-6505] Let emoji encoded properly for json.dumps()
damon09273 closed pull request #7398: [AIRFLOW-6505] Let emoji encoded properly for json.dumps() URL: https://github.com/apache/airflow/pull/7398 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6505) Let emoji encoded properly for json.dumps() -- BaseSQLToGoogleCloudStorageOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034207#comment-17034207 ] ASF GitHub Bot commented on AIRFLOW-6505: - damon09273 commented on pull request #7398: [AIRFLOW-6505] Let emoji encoded properly for json.dumps() URL: https://github.com/apache/airflow/pull/7398 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Let emoji encoded properly for json.dumps() -- > BaseSQLToGoogleCloudStorageOperator > -- > > Key: AIRFLOW-6505 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6505 > Project: Apache Airflow > Issue Type: Bug > Components: contrib >Affects Versions: 1.10.8 > Environment: 1.10.7 >Reporter: Damon Liao >Assignee: Damon Liao >Priority: Blocker > Fix For: 1.10.7 > > > Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem > fixed by adding parameter `ensure_ascii=False`. > I've created a [PR|[https://github.com/apache/airflow/pull/7398]] to fix > this problem. > In [line|#L175]] BaseSQLToGoogleCloudStorageOperator. > > For example > the emoji encoded differentially when in use or not use > `ensure_ascii=False`. > In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb" > Not Use (only 2 slash):"\\\ud83c\\\udf7b" > > Ref: > [https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6505) Let emoji encoded properly for json.dumps() -- BaseSQLToGoogleCloudStorageOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damon Liao updated AIRFLOW-6505: Description: Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem fixed by adding parameter `ensure_ascii=False`. I've created a [PR|[https://github.com/apache/airflow/pull/7398]] to fix this problem. In [line|#L175]] BaseSQLToGoogleCloudStorageOperator. For example the emoji encoded differentially when in use or not use `ensure_ascii=False`. In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb" Not Use (only 2 slash):"\\\ud83c\\\udf7b" Ref: [https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly] was: Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem fixed by adding parameter `ensure_ascii=False`. I've created a [PR|[https://github.com/damon09273/airflow/pull/1]] to fix this problem. In [line|#L175]] BaseSQLToGoogleCloudStorageOperato. For example the emoji encoded differentially when in use or not use `ensure_ascii=False`. In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb" Not Use (only 2 slash):"\\\ud83c\\\udf7b" Ref: [https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly] > Let emoji encoded properly for json.dumps() -- > BaseSQLToGoogleCloudStorageOperator > -- > > Key: AIRFLOW-6505 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6505 > Project: Apache Airflow > Issue Type: Bug > Components: contrib >Affects Versions: 1.10.8 > Environment: 1.10.7 >Reporter: Damon Liao >Assignee: Damon Liao >Priority: Blocker > Fix For: 1.10.7 > > > Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem > fixed by adding parameter `ensure_ascii=False`. > I've created a [PR|[https://github.com/apache/airflow/pull/7398]] to fix > this problem. > In [line|#L175]] BaseSQLToGoogleCloudStorageOperator. > > For example > the emoji encoded differentially when in use or not use > `ensure_ascii=False`. > In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb" > Not Use (only 2 slash):"\\\ud83c\\\udf7b" > > Ref: > [https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6505) Let emoji encoded properly for json.dumps() -- BaseSQLToGoogleCloudStorageOperator
[
https://issues.apache.org/jira/browse/AIRFLOW-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034196#comment-17034196
]
ASF GitHub Bot commented on AIRFLOW-6505:
-
damon09273 commented on pull request #7398: [AIRFLOW-6505] Let emoji encoded
properly for json.dumps()
URL: https://github.com/apache/airflow/pull/7398
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW-6505) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-6505
### Description
Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem
fixed by adding parameter `ensure_ascii=False`.
In
[line](https://github.com/apache/airflow/blob/1.10.7/airflow/contrib/operators/sql_to_gcs.py#L175)
BaseSQLToGoogleCloudStorageOperator.
For example
the emoji encoded differentially when in use or not use
`ensure_ascii=False`.
In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"
Not Use:"ud83cudf7b"
Ref:
https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [x] Passes `flake8`
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
[GitHub] [airflow-site] XD-DENG commented on issue #238: Typo in community page
XD-DENG commented on issue #238: Typo in community page URL: https://github.com/apache/airflow-site/issues/238#issuecomment-584510155 Thanks @louisguitton This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] damon09273 opened a new pull request #7398: [AIRFLOW-6505] Let emoji encoded properly for json.dumps()
damon09273 opened a new pull request #7398: [AIRFLOW-6505] Let emoji encoded
properly for json.dumps()
URL: https://github.com/apache/airflow/pull/7398
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW-6505) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-6505
### Description
Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem
fixed by adding parameter `ensure_ascii=False`.
In
[line](https://github.com/apache/airflow/blob/1.10.7/airflow/contrib/operators/sql_to_gcs.py#L175)
BaseSQLToGoogleCloudStorageOperator.
For example
the emoji encoded differentially when in use or not use
`ensure_ascii=False`.
In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"
Not Use:"ud83cudf7b"
Ref:
https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [x] Passes `flake8`
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[airflow-site] branch master updated: Add mail list unsubscribe link (#250)
This is an automated email from the ASF dual-hosted git repository. xddeng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/airflow-site.git The following commit(s) were added to refs/heads/master by this push: new c598425 Add mail list unsubscribe link (#250) c598425 is described below commit c598425ce4f515814ff943fe0bf442c950fecf64 Author: Fokko Driesprong AuthorDate: Tue Feb 11 08:33:41 2020 +0100 Add mail list unsubscribe link (#250) --- landing-pages/site/content/en/community/_index.html | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/landing-pages/site/content/en/community/_index.html b/landing-pages/site/content/en/community/_index.html index 8628577..c77199f 100644 --- a/landing-pages/site/content/en/community/_index.html +++ b/landing-pages/site/content/en/community/_index.html @@ -16,10 +16,10 @@ menu: -Users list: mailto:users-subscr...@airflow.apache.org;>users-subscr...@airflow.apache.org +Users list: mailto:users-subscr...@airflow.apache.org;>users-subscr...@airflow.apache.org and mailto:users-unsubscr...@airflow.apache.org;>users-unsubscr...@airflow.apache.org to unsubscribe -Dev list: mailto:dev-subscr...@airflow.apache.org;>dev-subscr...@airflow.apache.org +Dev list: mailto:dev-subscr...@airflow.apache.org;>dev-subscr...@airflow.apache.org and mailto:dev-unsubscr...@airflow.apache.org;>dev-unsubscr...@airflow.apache.org to unsubscribe
[GitHub] [airflow-site] XD-DENG merged pull request #250: Add unsubscribe link
XD-DENG merged pull request #250: Add unsubscribe link URL: https://github.com/apache/airflow-site/pull/250 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] damon09273 closed pull request #7397: Let emoji encoded properly for json.dumps() -- BaseSQLToGoogleCloudStorageOperator
damon09273 closed pull request #7397: Let emoji encoded properly for json.dumps() -- BaseSQLToGoogleCloudStorageOperator URL: https://github.com/apache/airflow/pull/7397 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] damon09273 opened a new pull request #7397: Let emoji encoded properly for json.dumps() -- BaseSQLToGoogleCloudStorageOperator
damon09273 opened a new pull request #7397: Let emoji encoded properly for
json.dumps() -- BaseSQLToGoogleCloudStorageOperator
URL: https://github.com/apache/airflow/pull/7397
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW-6505) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-6505
### Description
Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem
fixed by adding parameter `ensure_ascii=False`.
In
[line](https://github.com/apache/airflow/blob/1.10.7/airflow/contrib/operators/sql_to_gcs.py#L175)
BaseSQLToGoogleCloudStorageOperato.
For example
the emoji encoded differentially when in use or not use
`ensure_ascii=False`.
In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"
Not Use:"ud83cudf7b"
Ref:
https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [x] Passes `flake8`
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on issue #6576: [AIRFLOW-5922] Add option to specify the mysql client library used in MySqlHook
feluelle commented on issue #6576: [AIRFLOW-5922] Add option to specify the mysql client library used in MySqlHook URL: https://github.com/apache/airflow/pull/6576#issuecomment-584507024 @potiuk @ashb @nuclearpinguin do you have any idea what I can do to fix the segmentation error? I tried to reorder the imports but that didn't help :/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow-site] Fokko opened a new pull request #250: Add unsubscribe link
Fokko opened a new pull request #250: Add unsubscribe link URL: https://github.com/apache/airflow-site/pull/250 It isn't clear how to unsubscribe from the mail list. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[airflow-site] branch master updated: Fix typo - Commiter => Committer (#247)
This is an automated email from the ASF dual-hosted git repository.
fokko pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow-site.git
The following commit(s) were added to refs/heads/master by this push:
new 72de3af Fix typo - Commiter => Committer (#247)
72de3af is described below
commit 72de3af50a099b4537175fff0e714eaa5f47ca3f
Author: Louis Guitton
AuthorDate: Tue Feb 11 08:20:10 2020 +0100
Fix typo - Commiter => Committer (#247)
Follow-up #243, Fix #238
---
docs-archive/1.10.4/changelog.html | 2 +-
docs-archive/1.10.5/changelog.html | 2 +-
docs-archive/1.10.6/_static/_gen/css/main-custom.min.css | 2 +-
docs-archive/1.10.6/changelog.html | 4 ++--
docs-archive/1.10.7/_static/_gen/css/main-custom.min.css | 2 +-
docs-archive/1.10.7/changelog.html | 4 ++--
docs-archive/1.10.8/_static/_gen/css/main-custom.min.css | 2 +-
docs-archive/1.10.8/changelog.html | 4 ++--
landing-pages/site/assets/scss/_community-page.scss| 4 ++--
landing-pages/site/assets/scss/_list-boxes.scss| 2 +-
...pe-2019-thoughts-and-insights-by-airflow-committers.md} | 4 ++--
.../site/data/{commiters.json => committers.json} | 0
landing-pages/site/layouts/community/list.html | 14 +++---
.../partials/boxes/{commiter.html => committer.html} | 14 +++---
landing-pages/src/index.js | 6 +++---
.../src/js/{commitersList.js => committersList.js} | 0
16 files changed, 33 insertions(+), 33 deletions(-)
diff --git a/docs-archive/1.10.4/changelog.html
b/docs-archive/1.10.4/changelog.html
index bb268a4..458bb7e 100644
--- a/docs-archive/1.10.4/changelog.html
+++ b/docs-archive/1.10.4/changelog.html
@@ -620,7 +620,7 @@
[AIRFLOW-XXX] Reduce log spam in tests (#5174)
[AIRFLOW-XXX] Speed up tests for PythonSensor (#5158)
[AIRFLOW-XXX] Add Bas Harenslak to committer list (#5157)
-[AIRFLOW-XXX] Add Jarek Potiuk to commiter list (#5132)
+[AIRFLOW-XXX] Add Jarek Potiuk to committer list (#5132)
[AIRFLOW-XXX] Update docstring for SchedulerJob (#5105)
[AIRFLOW-XXX] Fix docstrings for CassandraToGoogleCloudStorageOperator
(#5103)
[AIRFLOW-XXX] update SlackWebhookHook and SlackWebhookOperator
docstring (#5074)
diff --git a/docs-archive/1.10.5/changelog.html
b/docs-archive/1.10.5/changelog.html
index 566365c..b67121e 100644
--- a/docs-archive/1.10.5/changelog.html
+++ b/docs-archive/1.10.5/changelog.html
@@ -738,7 +738,7 @@
[AIRFLOW-XXX] Reduce log spam in tests (#5174)
[AIRFLOW-XXX] Speed up tests for PythonSensor (#5158)
[AIRFLOW-XXX] Add Bas Harenslak to committer list (#5157)
-[AIRFLOW-XXX] Add Jarek Potiuk to commiter list (#5132)
+[AIRFLOW-XXX] Add Jarek Potiuk to committer list (#5132)
[AIRFLOW-XXX] Update docstring for SchedulerJob (#5105)
[AIRFLOW-XXX] Fix docstrings for CassandraToGoogleCloudStorageOperator
(#5103)
[AIRFLOW-XXX] update SlackWebhookHook and SlackWebhookOperator
docstring (#5074)
diff --git a/docs-archive/1.10.6/_static/_gen/css/main-custom.min.css
b/docs-archive/1.10.6/_static/_gen/css/main-custom.min.css
index 2d686bf..c8ebf36 100644
--- a/docs-archive/1.10.6/_static/_gen/css/main-custom.min.css
+++ b/docs-archive/1.10.6/_static/_gen/css/main-custom.min.css
@@ -1 +1 @@
-@charset "UTF-8";@import
"https://fonts.googleapis.com/css?family=Rubik:500=swap";@import
"https://fonts.googleapis.com/css?family=Roboto:400,400i,500,700=swap";@import
"https://fonts.googleapis.com/css?family=Roboto+Mono:400,700=swap";.header__large--cerulean-blue{font-family:rubik,sans-serif;font-weight:500;font-size:72px;line-height:1.17;color:#017cee}.header__large--shamrock{font-family:rubik,sans-serif;font-weight:500;font-size:72px;line-height:1.17;color:#00
[...]
\ No newline at end of file
+@charset "UTF-8";@import
"https://fonts.googleapis.com/css?family=Rubik:500=swap";@import
"https://fonts.googleapis.com/css?family=Roboto:400,400i,500,700=swap";@import
"https://fonts.googleapis.com/css?family=Roboto+Mono:400,700=swap";.header__large--cerulean-blue{font-family:rubik,sans-serif;font-weight:500;font-size:72px;line-height:1.17;color:#017cee}.header__large--shamrock{font-family:rubik,sans-serif;font-weight:500;font-size:72px;line-height:1.17;color:#00
[...]
diff --git a/docs-archive/1.10.6/changelog.html
b/docs-archive/1.10.6/changelog.html
index c5f048d..7ac9e42 100644
--- a/docs-archive/1.10.6/changelog.html
+++ b/docs-archive/1.10.6/changelog.html
@@ -1285,7 +1285,7 @@ https://www.sphinx-doc.org/en/master/templating.html
[AIRFLOW-XXX] Reduce log spam in tests (#5174)
[AIRFLOW-XXX] Speed up tests for PythonSensor (#5158)
[AIRFLOW-XXX] Add Bas Harenslak to committer list (#5157)
-[AIRFLOW-XXX] Add Jarek Potiuk to commiter list (#5132)
+[AIRFLOW-XXX] Add Jarek Potiuk to committer list (#5132)
[AIRFLOW-XXX] Update
[GitHub] [airflow-site] Fokko commented on a change in pull request #246: Expand the guide for contributors
Fokko commented on a change in pull request #246: Expand the guide for contributors URL: https://github.com/apache/airflow-site/pull/246#discussion_r377472874 ## File path: CONTRIBUTE.md ## @@ -17,9 +17,10 @@ under the License. --> -## General directory structure +Contrubitor Guide Review comment: ```suggestion Contributor Guide ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow-site] Fokko merged pull request #247: Fix typo - Commiter => Committer
Fokko merged pull request #247: Fix typo - Commiter => Committer URL: https://github.com/apache/airflow-site/pull/247 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow-site] Fokko opened a new pull request #249: Remove AIP-11 in favor of master
Fokko opened a new pull request #249: Remove AIP-11 in favor of master URL: https://github.com/apache/airflow-site/pull/249 I believe this is a relic from the past. I tried to click the _Suggest a change on this page_ on here: https://airflow.apache.org/community/ But I got a 404. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-6771) Error in formatting in _collect_errors
Gregory Golberg created AIRFLOW-6771:
Summary: Error in formatting in _collect_errors
Key: AIRFLOW-6771
URL: https://issues.apache.org/jira/browse/AIRFLOW-6771
Project: Apache Airflow
Issue Type: Bug
Components: DagRun
Affects Versions: 1.9.0
Reporter: Gregory Golberg
In airflow/jobs.py, in _collect_errors() method (on [line
2323|https://github.com/apache/airflow/blob/v1-9-stable/airflow/jobs.py#L2323]
the code is
{code}
@provide_session
def _collect_errors(self, ti_status, session=None):
err = ''
if ti_status.failed:
err += (
"---\n"
"Some task instances failed:\n%s".format(ti_status.failed))
{code}
But that just results in %s -- last line should be instead
{code}
"Some task instances failed:\n{}".format(ti_status.failed))
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] jithin97 commented on a change in pull request #7396: [AIRFLOW-6770] Run particular test using breeze CLI bug fix
jithin97 commented on a change in pull request #7396: [AIRFLOW-6770] Run particular test using breeze CLI bug fix URL: https://github.com/apache/airflow/pull/7396#discussion_r377459173 ## File path: .travis.yml ## @@ -101,7 +101,7 @@ jobs: ENABLED_INTEGRATIONS="cassandra kerberos mongo openldap rabbitmq redis" RUN_INTEGRATION_TESTS=all stage: test - script: ./scripts/ci/ci_run_airflow_testing.sh tests/tests_core.py + script: ./scripts/ci/ci_run_airflow_testing.sh tests/test_core.py Review comment: fixed!!! https://github.com/apache/airflow/pull/7396/commits/f6be34fd8725ae44b156f0e51076d92bf69a36a0 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] jithin97 commented on a change in pull request #7396: [AIRFLOW-6770] Run particular test using breeze CLI bug fix
jithin97 commented on a change in pull request #7396: [AIRFLOW-6770] Run
particular test using breeze CLI bug fix
URL: https://github.com/apache/airflow/pull/7396#discussion_r377459124
##
File path: scripts/ci/in_container/entrypoint_ci.sh
##
@@ -211,6 +211,10 @@ fi
TEST_DIR="tests/"
+if [[ ${#@} -gt 0 && -n "$1" ]]; then
+TEST_DIR="$1"
+fi
+
if [[ -n ${RUNTIME} ]]; then
CI_ARGS+=("--runtime" "${RUNTIME}" "-rpfExX")
TEST_DIR="tests/runtime"
Review comment:
fixed!!!
https://github.com/apache/airflow/pull/7396/commits/f6be34fd8725ae44b156f0e51076d92bf69a36a0
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] jithin97 commented on a change in pull request #7396: [AIRFLOW-6770] Run particular test using breeze CLI bug fix
jithin97 commented on a change in pull request #7396: [AIRFLOW-6770] Run particular test using breeze CLI bug fix URL: https://github.com/apache/airflow/pull/7396#discussion_r377459152 ## File path: scripts/ci/in_container/entrypoint_ci.sh ## @@ -211,6 +211,10 @@ fi TEST_DIR="tests/" Review comment: fixed!!! https://github.com/apache/airflow/pull/7396/commits/f6be34fd8725ae44b156f0e51076d92bf69a36a0 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7396: [AIRFLOW-6770] Run particular test using breeze CLI bug fix
potiuk commented on issue #7396: [AIRFLOW-6770] Run particular test using breeze CLI bug fix URL: https://github.com/apache/airflow/pull/7396#issuecomment-584491660 Thanks @jithin97 ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7396: [AIRFLOW-6770] Run particular test using breeze CLI bug fix
codecov-io commented on issue #7396: [AIRFLOW-6770] Run particular test using breeze CLI bug fix URL: https://github.com/apache/airflow/pull/7396#issuecomment-584488896 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7396?src=pr=h1) Report > Merging [#7396](https://codecov.io/gh/apache/airflow/pull/7396?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/1e00243014382d4cb7152ca7c5011b97cbd733b0?src=pr=desc) will **decrease** coverage by `0.14%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7396?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7396 +/- ## == - Coverage86.6% 86.46% -0.15% == Files 873 873 Lines 4072540725 == - Hits3527135213 -58 - Misses 5454 5512 +58 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7396?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/7396/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5) | `50.67% <0%> (-37.84%)` | :arrow_down: | | [airflow/executors/base\_executor.py](https://codecov.io/gh/apache/airflow/pull/7396/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvYmFzZV9leGVjdXRvci5weQ==) | `93.58% <0%> (-2.57%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7396?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7396?src=pr=footer). Last update [1e00243...8421b26](https://codecov.io/gh/apache/airflow/pull/7396?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7396: [AIRFLOW-6770] Run particular test using breeze CLI bug fix
potiuk commented on a change in pull request #7396: [AIRFLOW-6770] Run
particular test using breeze CLI bug fix
URL: https://github.com/apache/airflow/pull/7396#discussion_r377454766
##
File path: scripts/ci/in_container/entrypoint_ci.sh
##
@@ -211,6 +211,10 @@ fi
TEST_DIR="tests/"
+if [[ ${#@} -gt 0 && -n "$1" ]]; then
+TEST_DIR="$1"
+fi
+
if [[ -n ${RUNTIME} ]]; then
CI_ARGS+=("--runtime" "${RUNTIME}" "-rpfExX")
TEST_DIR="tests/runtime"
Review comment:
```suggestion
TESTS_TO_RUN="tests/runtime"
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7396: [AIRFLOW-6770] Run particular test using breeze CLI bug fix
potiuk commented on a change in pull request #7396: [AIRFLOW-6770] Run particular test using breeze CLI bug fix URL: https://github.com/apache/airflow/pull/7396#discussion_r377454608 ## File path: scripts/ci/in_container/entrypoint_ci.sh ## @@ -211,6 +211,10 @@ fi TEST_DIR="tests/" Review comment: ```suggestion TESTS_TO_RUN="tests/" ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7396: [AIRFLOW-6770] Run particular test using breeze CLI bug fix
potiuk commented on a change in pull request #7396: [AIRFLOW-6770] Run
particular test using breeze CLI bug fix
URL: https://github.com/apache/airflow/pull/7396#discussion_r377454957
##
File path: scripts/ci/in_container/entrypoint_ci.sh
##
@@ -211,6 +211,10 @@ fi
TEST_DIR="tests/"
+if [[ ${#@} -gt 0 && -n "$1" ]]; then
+TEST_DIR="$1"
+fi
+
if [[ -n ${RUNTIME} ]]; then
CI_ARGS+=("--runtime" "${RUNTIME}" "-rpfExX")
TEST_DIR="tests/runtime"
Review comment:
And also we should change TEST_DIR below to TESTS_TO_RUN :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7396: [AIRFLOW-6770] Run particular test using breeze CLI bug fix
potiuk commented on a change in pull request #7396: [AIRFLOW-6770] Run particular test using breeze CLI bug fix URL: https://github.com/apache/airflow/pull/7396#discussion_r377454669 ## File path: scripts/ci/in_container/entrypoint_ci.sh ## @@ -211,6 +211,10 @@ fi TEST_DIR="tests/" Review comment: I think this will be better name :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7396: [AIRFLOW-6770] Run particular test using breeze CLI bug fix
potiuk commented on a change in pull request #7396: [AIRFLOW-6770] Run particular test using breeze CLI bug fix URL: https://github.com/apache/airflow/pull/7396#discussion_r377454371 ## File path: .travis.yml ## @@ -101,7 +101,7 @@ jobs: ENABLED_INTEGRATIONS="cassandra kerberos mongo openldap rabbitmq redis" RUN_INTEGRATION_TESTS=all stage: test - script: ./scripts/ci/ci_run_airflow_testing.sh tests/tests_core.py + script: ./scripts/ci/ci_run_airflow_testing.sh tests/test_core.py Review comment: ```suggestion ``` That's an interesting one. Actually it's a mistake here. We should not constraint ourselves in this test only to test_core. It is likely a remnant from some testing done when I was implementing it. Good that you noticed it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6770) Unable to run specific test using breeze CLI
[
https://issues.apache.org/jira/browse/AIRFLOW-6770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034126#comment-17034126
]
ASF GitHub Bot commented on AIRFLOW-6770:
-
jithin97 commented on pull request #7396: [AIRFLOW-6770] Run particular test
using breeze CLI bug fix
URL: https://github.com/apache/airflow/pull/7396
### Description
Running a particular test from breeze CLI runs all the tests within _tests/_
directory instead of running only the tests specified by **test-target** flag
**Before:**
```
⇒ ./breeze --test-target tests/test_core.py
.
.
.
Starting the tests with those pytest arguments: tests/
test session starts
platform linux -- Python 3.6.10, pytest-5.3.5, py-1.8.1, pluggy-0.13.1 --
/usr/local/bin/python
cachedir: .pytest_cache
rootdir: /opt/airflow, inifile: pytest.ini
plugins: requests-mock-1.7.0, cov-2.8.1, flaky-3.6.1, instafail-0.4.1.post0,
celery-4.4.0
collected 5055 items / 1 skipped / 5054 selected
```
**After:**
```
⇒ ./breeze --test-target tests/test_core.py
.
.
.
Starting the tests with those pytest arguments: tests/test_core.py
test session starts
platform linux -- Python 3.6.10, pytest-5.3.5, py-1.8.1, pluggy-0.13.1 --
/usr/local/bin/python
cachedir: .pytest_cache
rootdir: /opt/airflow, inifile: pytest.ini
plugins: requests-mock-1.7.0, cov-2.8.1, flaky-3.6.1, instafail-0.4.1.post0,
celery-4.4.0
collected 24 items
```
---
Issue link: WILL BE INSERTED BY
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
Make sure to mark the boxes below before creating PR: [x]
- [x] Description above provides context of the change
- [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- =
JIRA ID*
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Commits follow "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
* For document-only changes commit message can start with
`[AIRFLOW-]`.
---
In case of fundamental code change, Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
for more information.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Unable to run specific test using breeze CLI
>
>
> Key: AIRFLOW-6770
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6770
> Project: Apache Airflow
> Issue Type: Bug
> Components: breeze
>Affects Versions: 1.10.9
>Reporter: Jithin Sukumar
>Assignee: Jithin Sukumar
>Priority: Minor
>
> {code:bash}
> ./breeze -t tests/test_core.py
> {code}
> is supposed to run only tests/test_core.py but instead runs all tests within
> tests/ directory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] jithin97 opened a new pull request #7396: [AIRFLOW-6770] Run particular test using breeze CLI bug fix
jithin97 opened a new pull request #7396: [AIRFLOW-6770] Run particular test using breeze CLI bug fix URL: https://github.com/apache/airflow/pull/7396 ### Description Running a particular test from breeze CLI runs all the tests within _tests/_ directory instead of running only the tests specified by **test-target** flag **Before:** ``` ⇒ ./breeze --test-target tests/test_core.py . . . Starting the tests with those pytest arguments: tests/ test session starts platform linux -- Python 3.6.10, pytest-5.3.5, py-1.8.1, pluggy-0.13.1 -- /usr/local/bin/python cachedir: .pytest_cache rootdir: /opt/airflow, inifile: pytest.ini plugins: requests-mock-1.7.0, cov-2.8.1, flaky-3.6.1, instafail-0.4.1.post0, celery-4.4.0 collected 5055 items / 1 skipped / 5054 selected ``` **After:** ``` ⇒ ./breeze --test-target tests/test_core.py . . . Starting the tests with those pytest arguments: tests/test_core.py test session starts platform linux -- Python 3.6.10, pytest-5.3.5, py-1.8.1, pluggy-0.13.1 -- /usr/local/bin/python cachedir: .pytest_cache rootdir: /opt/airflow, inifile: pytest.ini plugins: requests-mock-1.7.0, cov-2.8.1, flaky-3.6.1, instafail-0.4.1.post0, celery-4.4.0 collected 24 items ``` --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-6770) Unable to run specific test using breeze CLI
Jithin Sukumar created AIRFLOW-6770:
---
Summary: Unable to run specific test using breeze CLI
Key: AIRFLOW-6770
URL: https://issues.apache.org/jira/browse/AIRFLOW-6770
Project: Apache Airflow
Issue Type: Bug
Components: breeze
Affects Versions: 1.10.9
Reporter: Jithin Sukumar
Assignee: Jithin Sukumar
{code:bash}
./breeze -t tests/test_core.py
{code}
is supposed to run only tests/test_core.py but instead runs all tests within
tests/ directory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] yuqian90 removed a comment on issue #7276: [AIRFLOW-5391] Do not run skipped tasks when they are cleared
yuqian90 removed a comment on issue #7276: [AIRFLOW-5391] Do not run skipped tasks when they are cleared URL: https://github.com/apache/airflow/pull/7276#issuecomment-579271946 This is work-in-progress and untested. I want to get some opinions from committers first to see if it's worth pursuing. I'll then add tests and make the PR complete. One immediate shortcoming that I can see: This PR only makes skipped tasks respect the decision of upstream `BranchPythonOperator` when cleared. Other variants of branching exist, namely `BaseBranchOperator` and `ShortCircuitOperator`. This PR does not make tasks respect the decision of those operators when cleared. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5629) Support specify priority and priorityClassName spec in KubernetesPodOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034018#comment-17034018 ] ASF GitHub Bot commented on AIRFLOW-5629: - simis2626 commented on pull request #7395: [AIRFLOW-5629] Implement K8s priorityClassName in KubernetesPo… URL: https://github.com/apache/airflow/pull/7395 …dOperator Changes allow user to specify priority_class_name within KubernetesPodOperator which sets the Pod.spec value to influence scheduling and preemption behaviour of workloads generated by Airflow. pre-commit hook issue with BREEZE.rst adding whitespace - Can't push without committing whitespace change. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [ ] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Support specify priority and priorityClassName spec in KubernetesPodOperator > > > Key: AIRFLOW-5629 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5629 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Affects Versions: 2.0.0 >Reporter: Yaliang Wang >Assignee: Simon Levett >Priority: Major > Labels: features > > Since Kubernetes 1.14, Pod can have priority and priority class to indicate > the importance of the > pod([https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/]). > We should support to config the priority and priority class in the > KubernetesPodOperator -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] simis2626 opened a new pull request #7395: [AIRFLOW-5629] Implement K8s priorityClassName in KubernetesPo…
simis2626 opened a new pull request #7395: [AIRFLOW-5629] Implement K8s priorityClassName in KubernetesPo… URL: https://github.com/apache/airflow/pull/7395 …dOperator Changes allow user to specify priority_class_name within KubernetesPodOperator which sets the Pod.spec value to influence scheduling and preemption behaviour of workloads generated by Airflow. pre-commit hook issue with BREEZE.rst adding whitespace - Can't push without committing whitespace change. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [ ] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] boring-cyborg[bot] commented on issue #7395: [AIRFLOW-5629] Implement K8s priorityClassName in KubernetesPo…
boring-cyborg[bot] commented on issue #7395: [AIRFLOW-5629] Implement K8s priorityClassName in KubernetesPo… URL: https://github.com/apache/airflow/pull/7395#issuecomment-584409286 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst) Here are some useful points: - Pay attention to the quality of your code (flake8, pylint and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that. - In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/master/docs/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it. - Consider using [Breeze environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations. - Be patient and persistent. It might take some time to get a review or get the final approval from Committers. Apache Airflow is a community-driven project and together we are making it better . In case of doubts contact the developers at: Mailing List: d...@airflow.apache.org Slack: https://apache-airflow-slack.herokuapp.com/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (AIRFLOW-5629) Support specify priority and priorityClassName spec in KubernetesPodOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Levett reassigned AIRFLOW-5629: - Assignee: Simon Levett (was: Yaliang Wang) Working on a PR for this. > Support specify priority and priorityClassName spec in KubernetesPodOperator > > > Key: AIRFLOW-5629 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5629 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Affects Versions: 2.0.0 >Reporter: Yaliang Wang >Assignee: Simon Levett >Priority: Major > Labels: features > > Since Kubernetes 1.14, Pod can have priority and priority class to indicate > the importance of the > pod([https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/]). > We should support to config the priority and priority class in the > KubernetesPodOperator -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] stale[bot] closed pull request #2708: [AIRFLOW-1746] Add a Nomad operator to trigger job from Airflow
stale[bot] closed pull request #2708: [AIRFLOW-1746] Add a Nomad operator to trigger job from Airflow URL: https://github.com/apache/airflow/pull/2708 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-1746) Add a Nomad operator to trigger job from Airflow
[ https://issues.apache.org/jira/browse/AIRFLOW-1746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033986#comment-17033986 ] ASF GitHub Bot commented on AIRFLOW-1746: - stale[bot] commented on pull request #2708: [AIRFLOW-1746] Add a Nomad operator to trigger job from Airflow URL: https://github.com/apache/airflow/pull/2708 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add a Nomad operator to trigger job from Airflow > > > Key: AIRFLOW-1746 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1746 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib >Reporter: Eyal Trabelsi >Assignee: Eyal Trabelsi >Priority: Major > > We recently face the need to trigger nomad jobs from Airflow and no operator > are available for that. > The way the operator works is to register a nomad job and dispatch the job , > than check the status of the job using similar method like boto-core > (https://github.com/boto/botocore/blob/5a07b477114b11e6dc5f676f5db810972565b113/botocore/docs/waiter.py) > The operator uses https://github.com/jrxFive/python-nomad which is a wrap > over nomad rest api of nomad written in python. > Link to the PR : https://github.com/apache/incubator-airflow/pull/2708 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] spektom commented on issue #5785: [AIRFLOW-5176] Add Azure Data Explorer (Kusto) operator
spektom commented on issue #5785: [AIRFLOW-5176] Add Azure Data Explorer (Kusto) operator URL: https://github.com/apache/airflow/pull/5785#issuecomment-584335178 > Great work @spektom, sorry it took 6 months to land! No problem, thanks for reviewing and helping shape it! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-5176) Add integration with Azure Data Explorer
[ https://issues.apache.org/jira/browse/AIRFLOW-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor resolved AIRFLOW-5176. Fix Version/s: 2.0.0 Resolution: Fixed > Add integration with Azure Data Explorer > > > Key: AIRFLOW-5176 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5176 > Project: Apache Airflow > Issue Type: New Feature > Components: hooks, operators >Affects Versions: 1.10.4, 2.0.0 >Reporter: Michael Spector >Assignee: Michael Spector >Priority: Major > Fix For: 2.0.0 > > > Add a hook and an operator that allow sending queries to Azure Data Explorer > (Kusto) cluster. > ADX (Azure Data Explorer) is relatively new but very promising analytics data > store / data processing offering in Azure: > [https://azure.microsoft.com/en-us/services/data-explorer/] > > PR: [https://github.com/apache/airflow/pull/5785] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5176) Add integration with Azure Data Explorer
[ https://issues.apache.org/jira/browse/AIRFLOW-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033921#comment-17033921 ] ASF GitHub Bot commented on AIRFLOW-5176: - ashb commented on pull request #5785: [AIRFLOW-5176] Add Azure Data Explorer (Kusto) operator URL: https://github.com/apache/airflow/pull/5785 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add integration with Azure Data Explorer > > > Key: AIRFLOW-5176 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5176 > Project: Apache Airflow > Issue Type: New Feature > Components: hooks, operators >Affects Versions: 1.10.4, 2.0.0 >Reporter: Michael Spector >Assignee: Michael Spector >Priority: Major > > Add a hook and an operator that allow sending queries to Azure Data Explorer > (Kusto) cluster. > ADX (Azure Data Explorer) is relatively new but very promising analytics data > store / data processing offering in Azure: > [https://azure.microsoft.com/en-us/services/data-explorer/] > > PR: [https://github.com/apache/airflow/pull/5785] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5176) Add integration with Azure Data Explorer
[ https://issues.apache.org/jira/browse/AIRFLOW-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033922#comment-17033922 ] ASF subversion and git services commented on AIRFLOW-5176: -- Commit 1e00243014382d4cb7152ca7c5011b97cbd733b0 in airflow's branch refs/heads/master from Michael Spector [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=1e00243 ] [AIRFLOW-5176] Add Azure Data Explorer (Kusto) operator (#5785) > Add integration with Azure Data Explorer > > > Key: AIRFLOW-5176 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5176 > Project: Apache Airflow > Issue Type: New Feature > Components: hooks, operators >Affects Versions: 1.10.4, 2.0.0 >Reporter: Michael Spector >Assignee: Michael Spector >Priority: Major > Fix For: 2.0.0 > > > Add a hook and an operator that allow sending queries to Azure Data Explorer > (Kusto) cluster. > ADX (Azure Data Explorer) is relatively new but very promising analytics data > store / data processing offering in Azure: > [https://azure.microsoft.com/en-us/services/data-explorer/] > > PR: [https://github.com/apache/airflow/pull/5785] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] boring-cyborg[bot] commented on issue #5785: [AIRFLOW-5176] Add Azure Data Explorer (Kusto) operator
boring-cyborg[bot] commented on issue #5785: [AIRFLOW-5176] Add Azure Data Explorer (Kusto) operator URL: https://github.com/apache/airflow/pull/5785#issuecomment-584334338 Awesome work, congrats on your first merged pull request! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on issue #5785: [AIRFLOW-5176] Add Azure Data Explorer (Kusto) operator
ashb commented on issue #5785: [AIRFLOW-5176] Add Azure Data Explorer (Kusto) operator URL: https://github.com/apache/airflow/pull/5785#issuecomment-584334509 Great work @spektom, sorry it took 6 months to land! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb merged pull request #5785: [AIRFLOW-5176] Add Azure Data Explorer (Kusto) operator
ashb merged pull request #5785: [AIRFLOW-5176] Add Azure Data Explorer (Kusto) operator URL: https://github.com/apache/airflow/pull/5785 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-6769) Multiple DAGs cause infinite loop in logging
[
https://issues.apache.org/jira/browse/AIRFLOW-6769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Boden updated AIRFLOW-6769:
-
Description:
After adding a second DAG to Airflow the scheduler spirals into an infinite
loop and is unable to schedule any tasks. The recursion appears to have to do
with logging. I've attached the output from scheduler and my requirements.txt
If I reset Airflow (`airflow resetdb`) and remove all but 1 DAG from the folder
there is no issue. I don't believe it's a specific DAG as I have tried with
various different test ones. I don't even have to schedule/run them, just
having Airflow recognize them causes the loop.
I'm using MySQL and Celery as my store and worker.
I have also attached part of the log file for one of my DAGs outputted by the
scheduler ($AIRFLOW_HOME/logs/scheduler/\{dag_name}/latest/\{dag_name}.log). It
seems `collect_dags` is being called every second by the scheduler.
was:
After adding a second DAG to Airflow the scheduler spirals into an infinite
loop and is unable to schedule any tasks. The recursion appears to have to do
with logging. I've attached the output from scheduler and my requirements.txt
If I reset Airflow (`airflow resetdb`) and remove all but 1 DAG from the folder
there is no issue. I don't believe it's a specific DAG as I have tried with
various different test ones. I don't even have to schedule/run them, just
having Airflow recognize them causes the loop.
I'm using MySQL and Celery as my store and worker.
> Multiple DAGs cause infinite loop in logging
>
>
> Key: AIRFLOW-6769
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6769
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 1.10.9
> Environment: macOS 10.15.2
> Python 3.7.5
>Reporter: Chris Boden
>Priority: Major
> Attachments: requirements.txt, scheduler-dag.log, scheduler.log
>
>
> After adding a second DAG to Airflow the scheduler spirals into an infinite
> loop and is unable to schedule any tasks. The recursion appears to have to do
> with logging. I've attached the output from scheduler and my requirements.txt
> If I reset Airflow (`airflow resetdb`) and remove all but 1 DAG from the
> folder there is no issue. I don't believe it's a specific DAG as I have tried
> with various different test ones. I don't even have to schedule/run them,
> just having Airflow recognize them causes the loop.
> I'm using MySQL and Celery as my store and worker.
> I have also attached part of the log file for one of my DAGs outputted by the
> scheduler ($AIRFLOW_HOME/logs/scheduler/\{dag_name}/latest/\{dag_name}.log).
> It seems `collect_dags` is being called every second by the scheduler.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6769) Multiple DAGs cause infinite loop in logging
[ https://issues.apache.org/jira/browse/AIRFLOW-6769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Boden updated AIRFLOW-6769: - Attachment: scheduler-dag.log > Multiple DAGs cause infinite loop in logging > > > Key: AIRFLOW-6769 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6769 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Affects Versions: 1.10.9 > Environment: macOS 10.15.2 > Python 3.7.5 >Reporter: Chris Boden >Priority: Major > Attachments: requirements.txt, scheduler-dag.log, scheduler.log > > > After adding a second DAG to Airflow the scheduler spirals into an infinite > loop and is unable to schedule any tasks. The recursion appears to have to do > with logging. I've attached the output from scheduler and my requirements.txt > If I reset Airflow (`airflow resetdb`) and remove all but 1 DAG from the > folder there is no issue. I don't believe it's a specific DAG as I have tried > with various different test ones. I don't even have to schedule/run them, > just having Airflow recognize them causes the loop. > I'm using MySQL and Celery as my store and worker. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6769) Multiple DAGs cause infinite loop in logging
Chris Boden created AIRFLOW-6769: Summary: Multiple DAGs cause infinite loop in logging Key: AIRFLOW-6769 URL: https://issues.apache.org/jira/browse/AIRFLOW-6769 Project: Apache Airflow Issue Type: Bug Components: scheduler Affects Versions: 1.10.9 Environment: macOS 10.15.2 Python 3.7.5 Reporter: Chris Boden Attachments: requirements.txt, scheduler.log After adding a second DAG to Airflow the scheduler spirals into an infinite loop and is unable to schedule any tasks. The recursion appears to have to do with logging. I've attached the output from scheduler and my requirements.txt If I reset Airflow (`airflow resetdb`) and remove all but 1 DAG from the folder there is no issue. I don't believe it's a specific DAG as I have tried with various different test ones. I don't even have to schedule/run them, just having Airflow recognize them causes the loop. I'm using MySQL and Celery as my store and worker. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] stale[bot] commented on issue #6256: [AIRFLOW-5590] Add run_id to trigger DAG run API response
stale[bot] commented on issue #6256: [AIRFLOW-5590] Add run_id to trigger DAG run API response URL: https://github.com/apache/airflow/pull/6256#issuecomment-584258610 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] blcksrx commented on issue #7358: [AIRFLOW-1202] Create Elasticsearch Hook
blcksrx commented on issue #7358: [AIRFLOW-1202] Create Elasticsearch Hook URL: https://github.com/apache/airflow/pull/7358#issuecomment-584257515 Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7358: [AIRFLOW-1202] Create Elasticsearch Hook
codecov-io edited a comment on issue #7358: [AIRFLOW-1202] Create Elasticsearch Hook URL: https://github.com/apache/airflow/pull/7358#issuecomment-581934830 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7358?src=pr=h1) Report > Merging [#7358](https://codecov.io/gh/apache/airflow/pull/7358?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/319714f07c470bca999b2f6becbb340c81e79e29?src=pr=desc) will **decrease** coverage by `0.46%`. > The diff coverage is `47.5%`. [](https://codecov.io/gh/apache/airflow/pull/7358?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7358 +/- ## == - Coverage 86.58% 86.12% -0.47% == Files 871 872 +1 Lines 4066040700 +40 == - Hits3520735053 -154 - Misses 5453 5647 +194 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7358?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/7358/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `95.07% <ø> (ø)` | :arrow_up: | | [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/7358/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `98.26% <100%> (+0.01%)` | :arrow_up: | | [...low/providers/elasticsearch/hooks/elasticsearch.py](https://codecov.io/gh/apache/airflow/pull/7358/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZWxhc3RpY3NlYXJjaC9ob29rcy9lbGFzdGljc2VhcmNoLnB5) | `46.15% <46.15%> (ø)` | | | [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7358/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `100% <0%> (ø)` | :arrow_up: | | [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/7358/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `100% <0%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7358/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7358/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `100% <0%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7358/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/7358/diff?src=pr=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==) | `76.08% <0%> (ø)` | :arrow_up: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7358/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0%> (-45.08%)` | :arrow_down: | | ... and [14 more](https://codecov.io/gh/apache/airflow/pull/7358/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7358?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7358?src=pr=footer). Last update [319714f...b168505](https://codecov.io/gh/apache/airflow/pull/7358?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (AIRFLOW-6369) clear cli command needs a 'conf' option
[
https://issues.apache.org/jira/browse/AIRFLOW-6369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yajna Pandith reassigned AIRFLOW-6369:
--
Assignee: Yajna Pandith
> clear cli command needs a 'conf' option
> ---
>
> Key: AIRFLOW-6369
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6369
> Project: Apache Airflow
> Issue Type: Improvement
> Components: cli, core, DagRun
>Affects Versions: 1.10.6
>Reporter: t oo
>Assignee: Yajna Pandith
>Priority: Major
> Labels: gsoc, gsoc2020, mentor
>
> key-value pairs of conf can be passed into trigger_dag command
> ie
> --conf '{"ric":"amzn"}'
> clear command needs this feature too
> ie in case exec_date is important and there was a failure halfway in the 1st
> dagrun due to bad conf being sent on trigger_dag command and want to run the
> same execdate but with new conf on 2nd dagrun
> alternative solution would be a new delete_dag_run cli command so never need
> to 'clear' but can do a 2nd DagRun for same exec date
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6767) Default workgroup name for Athena is incorrect
[ https://issues.apache.org/jira/browse/AIRFLOW-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033743#comment-17033743 ] ASF subversion and git services commented on AIRFLOW-6767: -- Commit b7aa778b38df2f116a1c20031e72fea8b97315bf in airflow's branch refs/heads/master from Bhavika Tekwani [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=b7aa778 ] [AIRFLOW-6767] Correct name for default Athena workgroup (#7394) * Change default wg name * Change workgroup in tests > Default workgroup name for Athena is incorrect > -- > > Key: AIRFLOW-6767 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6767 > Project: Apache Airflow > Issue Type: Improvement > Components: aws >Affects Versions: 1.10.9 >Reporter: Bhavika >Assignee: Bhavika >Priority: Minor > Labels: easy-fix, pull-request-available > Fix For: 1.10.10 > > > The default Athena workgroup name is set to "default" when it should be > "primary". See this comment - > [https://github.com/apache/airflow/pull/6871#issuecomment-584066252] > I'll submit a fix for the incorrect name. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6767) Default workgroup name for Athena is incorrect
[ https://issues.apache.org/jira/browse/AIRFLOW-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-6767. Resolution: Fixed > Default workgroup name for Athena is incorrect > -- > > Key: AIRFLOW-6767 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6767 > Project: Apache Airflow > Issue Type: Improvement > Components: aws >Affects Versions: 1.10.9 >Reporter: Bhavika >Assignee: Bhavika >Priority: Minor > Labels: easy-fix, pull-request-available > Fix For: 1.10.10 > > > The default Athena workgroup name is set to "default" when it should be > "primary". See this comment - > [https://github.com/apache/airflow/pull/6871#issuecomment-584066252] > I'll submit a fix for the incorrect name. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6767) Default workgroup name for Athena is incorrect
[ https://issues.apache.org/jira/browse/AIRFLOW-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033741#comment-17033741 ] ASF GitHub Bot commented on AIRFLOW-6767: - mik-laj commented on pull request #7394: [AIRFLOW-6767] Correct name for default Athena workgroup URL: https://github.com/apache/airflow/pull/7394 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Default workgroup name for Athena is incorrect > -- > > Key: AIRFLOW-6767 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6767 > Project: Apache Airflow > Issue Type: Improvement > Components: aws >Affects Versions: 1.10.9 >Reporter: Bhavika >Assignee: Bhavika >Priority: Minor > Labels: easy-fix, pull-request-available > Fix For: 1.10.10 > > > The default Athena workgroup name is set to "default" when it should be > "primary". See this comment - > [https://github.com/apache/airflow/pull/6871#issuecomment-584066252] > I'll submit a fix for the incorrect name. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] mik-laj merged pull request #7394: [AIRFLOW-6767] Correct name for default Athena workgroup
mik-laj merged pull request #7394: [AIRFLOW-6767] Correct name for default Athena workgroup URL: https://github.com/apache/airflow/pull/7394 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] leonardoam commented on issue #6337: [AIRFLOW-5659] - Add support for ephemeral storage on KubernetesPodOp…
leonardoam commented on issue #6337: [AIRFLOW-5659] - Add support for ephemeral storage on KubernetesPodOp… URL: https://github.com/apache/airflow/pull/6337#issuecomment-584208987 @potiuk branch rebased. Thx for your help! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5615) BaseJob subclasses shouldn't implement own heartbeat logic
[
https://issues.apache.org/jira/browse/AIRFLOW-5615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033725#comment-17033725
]
ASF GitHub Bot commented on AIRFLOW-5615:
-
ashb commented on pull request #6311: [AIRFLOW-5615] Reduce duplicated logic
around job heartbeating
URL: https://github.com/apache/airflow/pull/6311
### Jira
- [x] https://issues.apache.org/jira/browse/AIRFLOW-5615
### Description
- [x] Both SchedulerJob and LocalTaskJob have their own timers and decide
when
to call heartbeat based upon that. THis makes those jobs harder to
follow, so I've moved the logic to BaseJob
### Tests
- [x] Tests added
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> BaseJob subclasses shouldn't implement own heartbeat logic
> --
>
> Key: AIRFLOW-5615
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5615
> Project: Apache Airflow
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.10.5
>Reporter: Ash Berlin-Taylor
>Assignee: Ash Berlin-Taylor
>Priority: Trivial
> Fix For: 2.0.0
>
>
> Both SchedulerJob and LocalTaskJob have their own timers and decide when to
> call heartbeat based upon that.
> That logic should be removed and live in BaseJob to simplify the code.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5615) BaseJob subclasses shouldn't implement own heartbeat logic
[ https://issues.apache.org/jira/browse/AIRFLOW-5615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033724#comment-17033724 ] ASF GitHub Bot commented on AIRFLOW-5615: - stale[bot] commented on pull request #6311: [AIRFLOW-5615] Reduce duplicated logic around job heartbeating URL: https://github.com/apache/airflow/pull/6311 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > BaseJob subclasses shouldn't implement own heartbeat logic > -- > > Key: AIRFLOW-5615 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5615 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 1.10.5 >Reporter: Ash Berlin-Taylor >Assignee: Ash Berlin-Taylor >Priority: Trivial > Fix For: 2.0.0 > > > Both SchedulerJob and LocalTaskJob have their own timers and decide when to > call heartbeat based upon that. > That logic should be removed and live in BaseJob to simplify the code. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] ashb opened a new pull request #6311: [AIRFLOW-5615] Reduce duplicated logic around job heartbeating
ashb opened a new pull request #6311: [AIRFLOW-5615] Reduce duplicated logic
around job heartbeating
URL: https://github.com/apache/airflow/pull/6311
### Jira
- [x] https://issues.apache.org/jira/browse/AIRFLOW-5615
### Description
- [x] Both SchedulerJob and LocalTaskJob have their own timers and decide
when
to call heartbeat based upon that. THis makes those jobs harder to
follow, so I've moved the logic to BaseJob
### Tests
- [x] Tests added
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] stale[bot] closed pull request #6311: [AIRFLOW-5615] Reduce duplicated logic around job heartbeating
stale[bot] closed pull request #6311: [AIRFLOW-5615] Reduce duplicated logic around job heartbeating URL: https://github.com/apache/airflow/pull/6311 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7391: [AIRFLOW-6663] Prepare backporting packages
codecov-io commented on issue #7391: [AIRFLOW-6663] Prepare backporting packages URL: https://github.com/apache/airflow/pull/7391#issuecomment-584201606 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7391?src=pr=h1) Report > Merging [#7391](https://codecov.io/gh/apache/airflow/pull/7391?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/53ba6d607c9a3548a996c0c716d1ffb8da0b291b?src=pr=desc) will **increase** coverage by `0.28%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7391?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7391 +/- ## == + Coverage86.3% 86.58% +0.28% == Files 871 871 Lines 4066040660 == + Hits3509135207 +116 + Misses 5569 5453 -116 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7391?src=pr=tree) | Coverage Δ | | |---|---|---| | [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `93.61% <0%> (+23.4%)` | :arrow_up: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `74.5% <0%> (+23.52%)` | :arrow_up: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `92.25% <0%> (+45.07%)` | :arrow_up: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `100% <0%> (+47.05%)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7391/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `100% <0%> (+55.55%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7391?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7391?src=pr=footer). Last update [53ba6d6...c450e45](https://codecov.io/gh/apache/airflow/pull/7391?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6652: [AIRFLOW-5548] [AIRFLOW-5550] REST API enhancement - dag info, task …
codecov-io edited a comment on issue #6652: [AIRFLOW-5548] [AIRFLOW-5550] REST API enhancement - dag info, task … URL: https://github.com/apache/airflow/pull/6652#issuecomment-558277657 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6652?src=pr=h1) Report > Merging [#6652](https://codecov.io/gh/apache/airflow/pull/6652?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/b738c9eaca11f4ed7dd680d797335e2eef9817b8?src=pr=desc) will **increase** coverage by `53.52%`. > The diff coverage is `33.66%`. [](https://codecov.io/gh/apache/airflow/pull/6652?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#6652 +/- ## === + Coverage 32.93% 86.45% +53.52% === Files 871 874+3 Lines 4066040760 +100 === + Hits1339235240+21848 + Misses 27268 5520-21748 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6652?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/www/api/experimental/endpoints.py](https://codecov.io/gh/apache/airflow/pull/6652/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvYXBpL2V4cGVyaW1lbnRhbC9lbmRwb2ludHMucHk=) | `75.54% <23.72%> (+41.75%)` | :arrow_up: | | [airflow/api/common/experimental/get\_task.py](https://codecov.io/gh/apache/airflow/pull/6652/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9nZXRfdGFzay5weQ==) | `50% <28.57%> (-16.67%)` | :arrow_down: | | [airflow/api/common/experimental/get\_tasks.py](https://codecov.io/gh/apache/airflow/pull/6652/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9nZXRfdGFza3MucHk=) | `50% <50%> (ø)` | | | [airflow/api/common/experimental/get\_dags.py](https://codecov.io/gh/apache/airflow/pull/6652/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9nZXRfZGFncy5weQ==) | `58.33% <58.33%> (ø)` | | | [airflow/api/common/experimental/get\_dag.py](https://codecov.io/gh/apache/airflow/pull/6652/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9nZXRfZGFnLnB5) | `66.66% <66.66%> (ø)` | | | [airflow/bin/airflow](https://codecov.io/gh/apache/airflow/pull/6652/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vYWlyZmxvdw==) | `84.61% <0%> (ø)` | :arrow_up: | | [airflow/plugins\_manager.py](https://codecov.io/gh/apache/airflow/pull/6652/diff?src=pr=tree#diff-YWlyZmxvdy9wbHVnaW5zX21hbmFnZXIucHk=) | `86.95% <0%> (+0.72%)` | :arrow_up: | | ... and [745 more](https://codecov.io/gh/apache/airflow/pull/6652/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6652?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6652?src=pr=footer). Last update [b738c9e...159dfd3](https://codecov.io/gh/apache/airflow/pull/6652?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6652: [AIRFLOW-5548] [AIRFLOW-5550] REST API enhancement - dag info, task …
codecov-io edited a comment on issue #6652: [AIRFLOW-5548] [AIRFLOW-5550] REST API enhancement - dag info, task … URL: https://github.com/apache/airflow/pull/6652#issuecomment-558277657 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6652?src=pr=h1) Report > Merging [#6652](https://codecov.io/gh/apache/airflow/pull/6652?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/b738c9eaca11f4ed7dd680d797335e2eef9817b8?src=pr=desc) will **increase** coverage by `53.33%`. > The diff coverage is `33.66%`. [](https://codecov.io/gh/apache/airflow/pull/6652?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#6652 +/- ## === + Coverage 32.93% 86.26% +53.33% === Files 871 874+3 Lines 4066040760 +100 === + Hits1339235163+21771 + Misses 27268 5597-21671 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6652?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/www/api/experimental/endpoints.py](https://codecov.io/gh/apache/airflow/pull/6652/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvYXBpL2V4cGVyaW1lbnRhbC9lbmRwb2ludHMucHk=) | `75.54% <23.72%> (+41.75%)` | :arrow_up: | | [airflow/api/common/experimental/get\_task.py](https://codecov.io/gh/apache/airflow/pull/6652/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9nZXRfdGFzay5weQ==) | `50% <28.57%> (-16.67%)` | :arrow_down: | | [airflow/api/common/experimental/get\_tasks.py](https://codecov.io/gh/apache/airflow/pull/6652/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9nZXRfdGFza3MucHk=) | `50% <50%> (ø)` | | | [airflow/api/common/experimental/get\_dags.py](https://codecov.io/gh/apache/airflow/pull/6652/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9nZXRfZGFncy5weQ==) | `58.33% <58.33%> (ø)` | | | [airflow/api/common/experimental/get\_dag.py](https://codecov.io/gh/apache/airflow/pull/6652/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9nZXRfZGFnLnB5) | `66.66% <66.66%> (ø)` | | | [airflow/bin/airflow](https://codecov.io/gh/apache/airflow/pull/6652/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vYWlyZmxvdw==) | `84.61% <0%> (ø)` | :arrow_up: | | [airflow/plugins\_manager.py](https://codecov.io/gh/apache/airflow/pull/6652/diff?src=pr=tree#diff-YWlyZmxvdy9wbHVnaW5zX21hbmFnZXIucHk=) | `86.95% <0%> (+0.72%)` | :arrow_up: | | ... and [742 more](https://codecov.io/gh/apache/airflow/pull/6652/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6652?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6652?src=pr=footer). Last update [b738c9e...159dfd3](https://codecov.io/gh/apache/airflow/pull/6652?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] blcksrx commented on a change in pull request #7358: [AIRFLOW-1202] Create Elasticsearch Hook
blcksrx commented on a change in pull request #7358: [AIRFLOW-1202] Create
Elasticsearch Hook
URL: https://github.com/apache/airflow/pull/7358#discussion_r377157022
##
File path: airflow/providers/elasticsearch/hooks/elasticsearch.py
##
@@ -0,0 +1,89 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from es.elastic.api import connect
+
+from airflow.hooks.dbapi_hook import DbApiHook
+
+
+class ElasticsearchHook(DbApiHook):
+"""
+Interact with Elasticsearch through the elasticsearch-dbapi
+"""
+
+conn_name_attr = 'elasticsearch_conn_id'
+default_conn_name = 'elasticsearch_default'
+
+def __init__(self, *args, **kwargs):
+super().__init__(*args, **kwargs)
+self.schema = kwargs.pop("schema", None)
+self.connection = kwargs.pop("connection", None)
Review comment:
Actually, `self.connection` can not be as string and I did this:
```
def __init__(self,
schema: str = "http",
connection=None,
*args,
**kwargs):
super().__init__(*args, **kwargs)
self.schema = schema
self.connection = connection
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] bhavika commented on issue #7394: [AIRFLOW-6767] Correct name for default Athena workgroup
bhavika commented on issue #7394: [AIRFLOW-6767] Correct name for default Athena workgroup URL: https://github.com/apache/airflow/pull/7394#issuecomment-584197494 :wave: I'm not entirely sure how CodeCov does this but I haven't reduced the scope of any tests or deleted them so the reduction in coverage is probably due to something else. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6337: [AIRFLOW-5659] - Add support for ephemeral storage on KubernetesPodOp…
codecov-io edited a comment on issue #6337: [AIRFLOW-5659] - Add support for ephemeral storage on KubernetesPodOp… URL: https://github.com/apache/airflow/pull/6337#issuecomment-584135185 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6337?src=pr=h1) Report > Merging [#6337](https://codecov.io/gh/apache/airflow/pull/6337?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/53ba6d607c9a3548a996c0c716d1ffb8da0b291b?src=pr=desc) will **increase** coverage by `0.28%`. > The diff coverage is `50%`. [](https://codecov.io/gh/apache/airflow/pull/6337?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6337 +/- ## == + Coverage86.3% 86.59% +0.28% == Files 871 871 Lines 4066040662 +2 == + Hits3509135210 +119 + Misses 5569 5452 -117 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6337?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/pod\_generator.py](https://codecov.io/gh/apache/airflow/pull/6337/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9nZW5lcmF0b3IucHk=) | `95.95% <ø> (ø)` | :arrow_up: | | [airflow/kubernetes/pod.py](https://codecov.io/gh/apache/airflow/pull/6337/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZC5weQ==) | `92.5% <50%> (+0.39%)` | :arrow_up: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6337/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `88.12% <0%> (+0.19%)` | :arrow_up: | | [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/6337/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `93.61% <0%> (+23.4%)` | :arrow_up: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6337/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `74.5% <0%> (+23.52%)` | :arrow_up: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6337/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `92.25% <0%> (+45.07%)` | :arrow_up: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6337/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `100% <0%> (+47.05%)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6337/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `100% <0%> (+55.55%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6337?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6337?src=pr=footer). Last update [53ba6d6...0b17c3a](https://codecov.io/gh/apache/airflow/pull/6337?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] blcksrx commented on a change in pull request #7358: [AIRFLOW-1202] Create Elasticsearch Hook
blcksrx commented on a change in pull request #7358: [AIRFLOW-1202] Create
Elasticsearch Hook
URL: https://github.com/apache/airflow/pull/7358#discussion_r376952688
##
File path: airflow/providers/elasticsearch/hooks/elasticsearch.py
##
@@ -0,0 +1,89 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from es.elastic.api import connect
+
+from airflow.hooks.dbapi_hook import DbApiHook
+
+
+class ElasticsearchHook(DbApiHook):
+"""
+Interact with Elasticsearch through the elasticsearch-dbapi
+"""
+
+conn_name_attr = 'elasticsearch_conn_id'
+default_conn_name = 'elasticsearch_default'
+
+def __init__(self, *args, **kwargs):
+super().__init__(*args, **kwargs)
+self.schema = kwargs.pop("schema", None)
+self.connection = kwargs.pop("connection", None)
Review comment:
Done!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7394: [AIRFLOW-6767] Correct name for default Athena workgroup
codecov-io commented on issue #7394: [AIRFLOW-6767] Correct name for default Athena workgroup URL: https://github.com/apache/airflow/pull/7394#issuecomment-584195791 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7394?src=pr=h1) Report > Merging [#7394](https://codecov.io/gh/apache/airflow/pull/7394?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/53ba6d607c9a3548a996c0c716d1ffb8da0b291b?src=pr=desc) will **decrease** coverage by `0.17%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7394?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7394 +/- ## == - Coverage86.3% 86.12% -0.18% == Files 871 871 Lines 4066040660 == - Hits3509135020 -71 - Misses 5569 5640 +71 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7394?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/providers/amazon/aws/hooks/athena.py](https://codecov.io/gh/apache/airflow/pull/7394/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9ob29rcy9hdGhlbmEucHk=) | `67.18% <ø> (ø)` | :arrow_up: | | [airflow/providers/amazon/aws/operators/athena.py](https://codecov.io/gh/apache/airflow/pull/7394/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9vcGVyYXRvcnMvYXRoZW5hLnB5) | `74.5% <ø> (ø)` | :arrow_up: | | [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7394/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0%> (-64.16%)` | :arrow_down: | | [airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/7394/diff?src=pr=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==) | `30.43% <0%> (-45.66%)` | :arrow_down: | | [airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/7394/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5) | `55% <0%> (-45%)` | :arrow_down: | | [airflow/providers/apache/hive/hooks/hive.py](https://codecov.io/gh/apache/airflow/pull/7394/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvaG9va3MvaGl2ZS5weQ==) | `76.02% <0%> (-1.54%)` | :arrow_down: | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7394/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `90.9% <0%> (-0.83%)` | :arrow_down: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7394/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.19% <0%> (-0.15%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7394/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `88.12% <0%> (+0.19%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7394?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7394?src=pr=footer). Last update [53ba6d6...0fb123a](https://codecov.io/gh/apache/airflow/pull/7394?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (AIRFLOW-3534) KubernetesPodOperator breaks with active log-collection for long running tasks
[
https://issues.apache.org/jira/browse/AIRFLOW-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033699#comment-17033699
]
William Lachance edited comment on AIRFLOW-3534 at 2/10/20 3:46 PM:
My intuition at this point is that this is an issue with the Kubernetes python
library, but more investigation would be required to establish that for certain.
was (Author: wlach):
My intuition at this point is that this is an issue with the Kubernetes
library, but more investigation would be required to establish that for certain.
> KubernetesPodOperator breaks with active log-collection for long running tasks
> --
>
> Key: AIRFLOW-3534
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3534
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: 1.10.1
>Reporter: Christian Thiel
>Priority: Major
> Labels: kubernetes
>
> If a KubernetesPodOperator is started with get_logs=True, the pod breaks if
> no logs are produced after ~30 seconds due to http client timeout.
> The error occurs in two ways:
> 1. If the script doesn't write anything to stdout there are three WARNINGS of
> the connectionpool trying to get the logs:
> {code:python}
> // Some comments here
> 2018-12-17 15:23:15,092] {{logging_mixin.py:95}} WARNING - 2018-12-17
> 15:23:15,092 WARNING Retrying (Retry(total=2, connect=None, read=None,
> redirect=None, status=None)) after connection broken by
> 'ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed
> connection without response',))':
> /k8s/clusters/local/api/v1/namespaces/my-namespace/pods/my-pod/log?container=base=True=10
> {code}
> Followed by a {code:python}http.client.RemoteDisconnected: Remote end closed
> connection without response{code}
> originating from _monitor_pod in /contrib/kubernetes/pod_launcher.py
> Full Traceback:
> {code:python}
> Traceback (most recent call last):
> File "/opt/conda/lib/python3.6/site-packages/airflow/models.py", line 1659,
> in _run_raw_task
> result = task_copy.execute(context=context)
> File
> "/opt/conda/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py",
> line 123, in execute
> get_logs=self.get_logs)
> File
> "/opt/conda/lib/python3.6/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
> line 90, in run_pod
> return self._monitor_pod(pod, get_logs)
> File
> "/opt/conda/lib/python3.6/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
> line 102, in _monitor_pod
> _preload_content=False)
> File
> "/opt/conda/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py",
> line 18583, in read_namespaced_pod_log
> (data) = self.read_namespaced_pod_log_with_http_info(name, namespace,
> **kwargs)
> File
> "/opt/conda/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py",
> line 18689, in read_namespaced_pod_log_with_http_info
> collection_formats=collection_formats)
> File
> "/opt/conda/lib/python3.6/site-packages/kubernetes/client/api_client.py",
> line 321, in call_api
> _return_http_data_only, collection_formats, _preload_content,
> _request_timeout)
> File
> "/opt/conda/lib/python3.6/site-packages/kubernetes/client/api_client.py",
> line 155, in __call_api
> _request_timeout=_request_timeout)
> File
> "/opt/conda/lib/python3.6/site-packages/kubernetes/client/api_client.py",
> line 342, in request
> headers=headers)
> File "/opt/conda/lib/python3.6/site-packages/kubernetes/client/rest.py",
> line 231, in GET
> query_params=query_params)
> File "/opt/conda/lib/python3.6/site-packages/kubernetes/client/rest.py",
> line 205, in request
> headers=headers)
> File "/opt/conda/lib/python3.6/site-packages/urllib3/request.py", line 68,
> in request
> **urlopen_kw)
> File "/opt/conda/lib/python3.6/site-packages/urllib3/request.py", line 89,
> in request_encode_url
> return self.urlopen(method, url, **extra_kw)
> File "/opt/conda/lib/python3.6/site-packages/urllib3/poolmanager.py", line
> 322, in urlopen
> response = conn.urlopen(method, u.request_uri, **kw)
> File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py",
> line 667, in urlopen
> **response_kw)
> File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py",
> line 667, in urlopen
> **response_kw)
> File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py",
> line 667, in urlopen
> **response_kw)
> File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py",
> line 638, in urlopen
> _stacktrace=sys.exc_info()[2])
> File "/opt/conda/lib/python3.6/site-packages/urllib3/util/retry.py", line
> 398, in increment
>
[jira] [Commented] (AIRFLOW-3534) KubernetesPodOperator breaks with active log-collection for long running tasks
[
https://issues.apache.org/jira/browse/AIRFLOW-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033699#comment-17033699
]
William Lachance commented on AIRFLOW-3534:
---
My intuition at this point is that this is an issue with the Kubernetes
library, but more investigation would be required to establish that for certain.
> KubernetesPodOperator breaks with active log-collection for long running tasks
> --
>
> Key: AIRFLOW-3534
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3534
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: 1.10.1
>Reporter: Christian Thiel
>Priority: Major
> Labels: kubernetes
>
> If a KubernetesPodOperator is started with get_logs=True, the pod breaks if
> no logs are produced after ~30 seconds due to http client timeout.
> The error occurs in two ways:
> 1. If the script doesn't write anything to stdout there are three WARNINGS of
> the connectionpool trying to get the logs:
> {code:python}
> // Some comments here
> 2018-12-17 15:23:15,092] {{logging_mixin.py:95}} WARNING - 2018-12-17
> 15:23:15,092 WARNING Retrying (Retry(total=2, connect=None, read=None,
> redirect=None, status=None)) after connection broken by
> 'ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed
> connection without response',))':
> /k8s/clusters/local/api/v1/namespaces/my-namespace/pods/my-pod/log?container=base=True=10
> {code}
> Followed by a {code:python}http.client.RemoteDisconnected: Remote end closed
> connection without response{code}
> originating from _monitor_pod in /contrib/kubernetes/pod_launcher.py
> Full Traceback:
> {code:python}
> Traceback (most recent call last):
> File "/opt/conda/lib/python3.6/site-packages/airflow/models.py", line 1659,
> in _run_raw_task
> result = task_copy.execute(context=context)
> File
> "/opt/conda/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py",
> line 123, in execute
> get_logs=self.get_logs)
> File
> "/opt/conda/lib/python3.6/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
> line 90, in run_pod
> return self._monitor_pod(pod, get_logs)
> File
> "/opt/conda/lib/python3.6/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
> line 102, in _monitor_pod
> _preload_content=False)
> File
> "/opt/conda/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py",
> line 18583, in read_namespaced_pod_log
> (data) = self.read_namespaced_pod_log_with_http_info(name, namespace,
> **kwargs)
> File
> "/opt/conda/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py",
> line 18689, in read_namespaced_pod_log_with_http_info
> collection_formats=collection_formats)
> File
> "/opt/conda/lib/python3.6/site-packages/kubernetes/client/api_client.py",
> line 321, in call_api
> _return_http_data_only, collection_formats, _preload_content,
> _request_timeout)
> File
> "/opt/conda/lib/python3.6/site-packages/kubernetes/client/api_client.py",
> line 155, in __call_api
> _request_timeout=_request_timeout)
> File
> "/opt/conda/lib/python3.6/site-packages/kubernetes/client/api_client.py",
> line 342, in request
> headers=headers)
> File "/opt/conda/lib/python3.6/site-packages/kubernetes/client/rest.py",
> line 231, in GET
> query_params=query_params)
> File "/opt/conda/lib/python3.6/site-packages/kubernetes/client/rest.py",
> line 205, in request
> headers=headers)
> File "/opt/conda/lib/python3.6/site-packages/urllib3/request.py", line 68,
> in request
> **urlopen_kw)
> File "/opt/conda/lib/python3.6/site-packages/urllib3/request.py", line 89,
> in request_encode_url
> return self.urlopen(method, url, **extra_kw)
> File "/opt/conda/lib/python3.6/site-packages/urllib3/poolmanager.py", line
> 322, in urlopen
> response = conn.urlopen(method, u.request_uri, **kw)
> File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py",
> line 667, in urlopen
> **response_kw)
> File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py",
> line 667, in urlopen
> **response_kw)
> File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py",
> line 667, in urlopen
> **response_kw)
> File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py",
> line 638, in urlopen
> _stacktrace=sys.exc_info()[2])
> File "/opt/conda/lib/python3.6/site-packages/urllib3/util/retry.py", line
> 398, in increment
> raise MaxRetryError(_pool, url, error or ResponseError(cause))
> urllib3.exceptions.MaxRetryError:
> HTTPSConnectionPool(host='rancher.benteler.net', port=443): Max retries
> exceeded with url:
>
[jira] [Commented] (AIRFLOW-6505) Let emoji encoded properly for json.dumps() -- BaseSQLToGoogleCloudStorageOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033698#comment-17033698 ] Kamil Bregula commented on AIRFLOW-6505: [~Damon09273] Can you create PR in the Apache repository? I missed this PR because I only browse PRs from the Apache repository. > Let emoji encoded properly for json.dumps() -- > BaseSQLToGoogleCloudStorageOperator > -- > > Key: AIRFLOW-6505 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6505 > Project: Apache Airflow > Issue Type: Bug > Components: contrib >Affects Versions: 1.10.8 > Environment: 1.10.7 >Reporter: Damon Liao >Assignee: Damon Liao >Priority: Blocker > Fix For: 1.10.7 > > > Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem > fixed by adding parameter `ensure_ascii=False`. > I've created a [PR|[https://github.com/damon09273/airflow/pull/1]] to fix > this problem. > In [line|#L175]] BaseSQLToGoogleCloudStorageOperato. > > For example > the emoji encoded differentially when in use or not use > `ensure_ascii=False`. > In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb" > Not Use (only 2 slash):"\\\ud83c\\\udf7b" > > Ref: > [https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6505) Let emoji encoded properly for json.dumps() -- BaseSQLToGoogleCloudStorageOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033694#comment-17033694 ] Roster commented on AIRFLOW-6505: - Ran into this one too and came up with this Jira [~Damon09273] i think you wanted to PR your code to Airflow... you PRed it to your own fork thus no one see it. > Let emoji encoded properly for json.dumps() -- > BaseSQLToGoogleCloudStorageOperator > -- > > Key: AIRFLOW-6505 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6505 > Project: Apache Airflow > Issue Type: Bug > Components: contrib >Affects Versions: 1.10.8 > Environment: 1.10.7 >Reporter: Damon Liao >Assignee: Damon Liao >Priority: Blocker > Fix For: 1.10.7 > > > Emoji can't be encoded properly when ` json.dumps()` and 'UTF-8', problem > fixed by adding parameter `ensure_ascii=False`. > I've created a [PR|[https://github.com/damon09273/airflow/pull/1]] to fix > this problem. > In [line|#L175]] BaseSQLToGoogleCloudStorageOperato. > > For example > the emoji encoded differentially when in use or not use > `ensure_ascii=False`. > In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb" > Not Use (only 2 slash):"\\\ud83c\\\udf7b" > > Ref: > [https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-3534) KubernetesPodOperator breaks with active log-collection for long running tasks
[
https://issues.apache.org/jira/browse/AIRFLOW-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033693#comment-17033693
]
Kamil Bregula commented on AIRFLOW-3534:
Do you think this is somehow caused by Airflow or is it a bug in the Kubernetes
library? I would like to have clarity.
> KubernetesPodOperator breaks with active log-collection for long running tasks
> --
>
> Key: AIRFLOW-3534
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3534
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: 1.10.1
>Reporter: Christian Thiel
>Priority: Major
> Labels: kubernetes
>
> If a KubernetesPodOperator is started with get_logs=True, the pod breaks if
> no logs are produced after ~30 seconds due to http client timeout.
> The error occurs in two ways:
> 1. If the script doesn't write anything to stdout there are three WARNINGS of
> the connectionpool trying to get the logs:
> {code:python}
> // Some comments here
> 2018-12-17 15:23:15,092] {{logging_mixin.py:95}} WARNING - 2018-12-17
> 15:23:15,092 WARNING Retrying (Retry(total=2, connect=None, read=None,
> redirect=None, status=None)) after connection broken by
> 'ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed
> connection without response',))':
> /k8s/clusters/local/api/v1/namespaces/my-namespace/pods/my-pod/log?container=base=True=10
> {code}
> Followed by a {code:python}http.client.RemoteDisconnected: Remote end closed
> connection without response{code}
> originating from _monitor_pod in /contrib/kubernetes/pod_launcher.py
> Full Traceback:
> {code:python}
> Traceback (most recent call last):
> File "/opt/conda/lib/python3.6/site-packages/airflow/models.py", line 1659,
> in _run_raw_task
> result = task_copy.execute(context=context)
> File
> "/opt/conda/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py",
> line 123, in execute
> get_logs=self.get_logs)
> File
> "/opt/conda/lib/python3.6/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
> line 90, in run_pod
> return self._monitor_pod(pod, get_logs)
> File
> "/opt/conda/lib/python3.6/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
> line 102, in _monitor_pod
> _preload_content=False)
> File
> "/opt/conda/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py",
> line 18583, in read_namespaced_pod_log
> (data) = self.read_namespaced_pod_log_with_http_info(name, namespace,
> **kwargs)
> File
> "/opt/conda/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py",
> line 18689, in read_namespaced_pod_log_with_http_info
> collection_formats=collection_formats)
> File
> "/opt/conda/lib/python3.6/site-packages/kubernetes/client/api_client.py",
> line 321, in call_api
> _return_http_data_only, collection_formats, _preload_content,
> _request_timeout)
> File
> "/opt/conda/lib/python3.6/site-packages/kubernetes/client/api_client.py",
> line 155, in __call_api
> _request_timeout=_request_timeout)
> File
> "/opt/conda/lib/python3.6/site-packages/kubernetes/client/api_client.py",
> line 342, in request
> headers=headers)
> File "/opt/conda/lib/python3.6/site-packages/kubernetes/client/rest.py",
> line 231, in GET
> query_params=query_params)
> File "/opt/conda/lib/python3.6/site-packages/kubernetes/client/rest.py",
> line 205, in request
> headers=headers)
> File "/opt/conda/lib/python3.6/site-packages/urllib3/request.py", line 68,
> in request
> **urlopen_kw)
> File "/opt/conda/lib/python3.6/site-packages/urllib3/request.py", line 89,
> in request_encode_url
> return self.urlopen(method, url, **extra_kw)
> File "/opt/conda/lib/python3.6/site-packages/urllib3/poolmanager.py", line
> 322, in urlopen
> response = conn.urlopen(method, u.request_uri, **kw)
> File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py",
> line 667, in urlopen
> **response_kw)
> File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py",
> line 667, in urlopen
> **response_kw)
> File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py",
> line 667, in urlopen
> **response_kw)
> File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py",
> line 638, in urlopen
> _stacktrace=sys.exc_info()[2])
> File "/opt/conda/lib/python3.6/site-packages/urllib3/util/retry.py", line
> 398, in increment
> raise MaxRetryError(_pool, url, error or ResponseError(cause))
> urllib3.exceptions.MaxRetryError:
> HTTPSConnectionPool(host='rancher.benteler.net', port=443): Max retries
> exceeded with url:
>
[jira] [Comment Edited] (AIRFLOW-3534) KubernetesPodOperator breaks with active log-collection for long running tasks
[
https://issues.apache.org/jira/browse/AIRFLOW-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17023334#comment-17023334
]
William Lachance edited comment on AIRFLOW-3534 at 2/10/20 3:33 PM:
I've been looking into this one as it's pretty annoying for us
(https://github.com/mozilla/telemetry-airflow/issues/844).
It appears that it's possible to override the default timeouts in the method by
passing a _`_request_timeout` paramater to the `read_namespaced_pod_log`
method:_
_[https://raw.githubusercontent.com/kubernetes-client/python/master/kubernetes/client/api/core_v1_api.py_]
Reference on how this parameter gets processed:
[https://github.com/kubernetes-client/python/blob/5181d2389eacd92fa0f72237da87286ed0d45a3e/kubernetes/client/rest.py#L141]
I did up a quick prototype of this here:
[https://github.com/wlach/airflow/commit/e7ae01ac608d3f944b691875a9cf90dceb60ebcc]
Unfortunately it doesn't seem to solve the problem (I'm still getting the error
after 5 minutes of no output in my testing). Will do more research as time
allows...
was (Author: wlach):
I've been looking into this one as it's pretty annoying for us ().
It appears that it's possible to override the default timeouts in the method by
passing a _`_request_timeout` paramater to the `read_namespaced_pod_log`
method:_
_https://raw.githubusercontent.com/kubernetes-client/python/master/kubernetes/client/api/core_v1_api.py_
Reference on how this parameter gets processed:
https://github.com/kubernetes-client/python/blob/5181d2389eacd92fa0f72237da87286ed0d45a3e/kubernetes/client/rest.py#L141
I did up a quick prototype of this here:
[https://github.com/wlach/airflow/commit/e7ae01ac608d3f944b691875a9cf90dceb60ebcc]
Unfortunately it doesn't seem to solve the problem (I'm still getting the error
after 5 minutes of no output in my testing). Will do more research as time
allows...
> KubernetesPodOperator breaks with active log-collection for long running tasks
> --
>
> Key: AIRFLOW-3534
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3534
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: 1.10.1
>Reporter: Christian Thiel
>Priority: Major
> Labels: kubernetes
>
> If a KubernetesPodOperator is started with get_logs=True, the pod breaks if
> no logs are produced after ~30 seconds due to http client timeout.
> The error occurs in two ways:
> 1. If the script doesn't write anything to stdout there are three WARNINGS of
> the connectionpool trying to get the logs:
> {code:python}
> // Some comments here
> 2018-12-17 15:23:15,092] {{logging_mixin.py:95}} WARNING - 2018-12-17
> 15:23:15,092 WARNING Retrying (Retry(total=2, connect=None, read=None,
> redirect=None, status=None)) after connection broken by
> 'ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed
> connection without response',))':
> /k8s/clusters/local/api/v1/namespaces/my-namespace/pods/my-pod/log?container=base=True=10
> {code}
> Followed by a {code:python}http.client.RemoteDisconnected: Remote end closed
> connection without response{code}
> originating from _monitor_pod in /contrib/kubernetes/pod_launcher.py
> Full Traceback:
> {code:python}
> Traceback (most recent call last):
> File "/opt/conda/lib/python3.6/site-packages/airflow/models.py", line 1659,
> in _run_raw_task
> result = task_copy.execute(context=context)
> File
> "/opt/conda/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py",
> line 123, in execute
> get_logs=self.get_logs)
> File
> "/opt/conda/lib/python3.6/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
> line 90, in run_pod
> return self._monitor_pod(pod, get_logs)
> File
> "/opt/conda/lib/python3.6/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
> line 102, in _monitor_pod
> _preload_content=False)
> File
> "/opt/conda/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py",
> line 18583, in read_namespaced_pod_log
> (data) = self.read_namespaced_pod_log_with_http_info(name, namespace,
> **kwargs)
> File
> "/opt/conda/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py",
> line 18689, in read_namespaced_pod_log_with_http_info
> collection_formats=collection_formats)
> File
> "/opt/conda/lib/python3.6/site-packages/kubernetes/client/api_client.py",
> line 321, in call_api
> _return_http_data_only, collection_formats, _preload_content,
> _request_timeout)
> File
> "/opt/conda/lib/python3.6/site-packages/kubernetes/client/api_client.py",
> line 155, in __call_api
> _request_timeout=_request_timeout)
> File
>
[jira] [Commented] (AIRFLOW-3534) KubernetesPodOperator breaks with active log-collection for long running tasks
[
https://issues.apache.org/jira/browse/AIRFLOW-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033689#comment-17033689
]
William Lachance commented on AIRFLOW-3534:
---
I was using the latest stable version (10.0.1) but can reproduce this problem
on the pre-release version (11.0.0b2) as well.
> KubernetesPodOperator breaks with active log-collection for long running tasks
> --
>
> Key: AIRFLOW-3534
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3534
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: 1.10.1
>Reporter: Christian Thiel
>Priority: Major
> Labels: kubernetes
>
> If a KubernetesPodOperator is started with get_logs=True, the pod breaks if
> no logs are produced after ~30 seconds due to http client timeout.
> The error occurs in two ways:
> 1. If the script doesn't write anything to stdout there are three WARNINGS of
> the connectionpool trying to get the logs:
> {code:python}
> // Some comments here
> 2018-12-17 15:23:15,092] {{logging_mixin.py:95}} WARNING - 2018-12-17
> 15:23:15,092 WARNING Retrying (Retry(total=2, connect=None, read=None,
> redirect=None, status=None)) after connection broken by
> 'ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed
> connection without response',))':
> /k8s/clusters/local/api/v1/namespaces/my-namespace/pods/my-pod/log?container=base=True=10
> {code}
> Followed by a {code:python}http.client.RemoteDisconnected: Remote end closed
> connection without response{code}
> originating from _monitor_pod in /contrib/kubernetes/pod_launcher.py
> Full Traceback:
> {code:python}
> Traceback (most recent call last):
> File "/opt/conda/lib/python3.6/site-packages/airflow/models.py", line 1659,
> in _run_raw_task
> result = task_copy.execute(context=context)
> File
> "/opt/conda/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py",
> line 123, in execute
> get_logs=self.get_logs)
> File
> "/opt/conda/lib/python3.6/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
> line 90, in run_pod
> return self._monitor_pod(pod, get_logs)
> File
> "/opt/conda/lib/python3.6/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
> line 102, in _monitor_pod
> _preload_content=False)
> File
> "/opt/conda/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py",
> line 18583, in read_namespaced_pod_log
> (data) = self.read_namespaced_pod_log_with_http_info(name, namespace,
> **kwargs)
> File
> "/opt/conda/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py",
> line 18689, in read_namespaced_pod_log_with_http_info
> collection_formats=collection_formats)
> File
> "/opt/conda/lib/python3.6/site-packages/kubernetes/client/api_client.py",
> line 321, in call_api
> _return_http_data_only, collection_formats, _preload_content,
> _request_timeout)
> File
> "/opt/conda/lib/python3.6/site-packages/kubernetes/client/api_client.py",
> line 155, in __call_api
> _request_timeout=_request_timeout)
> File
> "/opt/conda/lib/python3.6/site-packages/kubernetes/client/api_client.py",
> line 342, in request
> headers=headers)
> File "/opt/conda/lib/python3.6/site-packages/kubernetes/client/rest.py",
> line 231, in GET
> query_params=query_params)
> File "/opt/conda/lib/python3.6/site-packages/kubernetes/client/rest.py",
> line 205, in request
> headers=headers)
> File "/opt/conda/lib/python3.6/site-packages/urllib3/request.py", line 68,
> in request
> **urlopen_kw)
> File "/opt/conda/lib/python3.6/site-packages/urllib3/request.py", line 89,
> in request_encode_url
> return self.urlopen(method, url, **extra_kw)
> File "/opt/conda/lib/python3.6/site-packages/urllib3/poolmanager.py", line
> 322, in urlopen
> response = conn.urlopen(method, u.request_uri, **kw)
> File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py",
> line 667, in urlopen
> **response_kw)
> File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py",
> line 667, in urlopen
> **response_kw)
> File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py",
> line 667, in urlopen
> **response_kw)
> File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py",
> line 638, in urlopen
> _stacktrace=sys.exc_info()[2])
> File "/opt/conda/lib/python3.6/site-packages/urllib3/util/retry.py", line
> 398, in increment
> raise MaxRetryError(_pool, url, error or ResponseError(cause))
> urllib3.exceptions.MaxRetryError:
> HTTPSConnectionPool(host='rancher.benteler.net', port=443): Max retries
> exceeded with url:
>
[GitHub] [airflow] ashb commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
ashb commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages URL: https://github.com/apache/airflow/pull/7391#discussion_r377129979 ## File path: MANIFEST-packages.in ## @@ -0,0 +1,24 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +include NOTICE +include LICENSE +include CHANGELOG.txt +include README.md +graft licenses/ Review comment: This was from CI logs This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7321: [AIRFLOW-XXXX] Remove duplication in BaseOperator docstring
codecov-io commented on issue #7321: [AIRFLOW-] Remove duplication in BaseOperator docstring URL: https://github.com/apache/airflow/pull/7321#issuecomment-584171117 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7321?src=pr=h1) Report > Merging [#7321](https://codecov.io/gh/apache/airflow/pull/7321?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/53ba6d607c9a3548a996c0c716d1ffb8da0b291b?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7321?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #7321 +/- ## == Coverage86.3% 86.3% == Files 871 871 Lines 40660 40660 == Hits35091 35091 Misses 55695569 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7321?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/7321/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `96.51% <ø> (ø)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7321?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7321?src=pr=footer). Last update [53ba6d6...2d1e37a](https://codecov.io/gh/apache/airflow/pull/7321?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7391: [AIRFLOW-6663] Prepare backporting packages
potiuk commented on issue #7391: [AIRFLOW-6663] Prepare backporting packages URL: https://github.com/apache/airflow/pull/7391#issuecomment-584166040 Fixed comments This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
potiuk commented on a change in pull request #7391: [AIRFLOW-6663] Prepare
backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r377115621
##
File path: setup.py
##
@@ -591,5 +594,144 @@ def do_setup():
)
+PROVIDERS_DEPENDENCIES = {
+"amazon": [aws],
+"apache.cassandra": [cassandra],
+"apache.druid": [druid],
+"apache.hdfs": [hdfs],
+"apache.hive": [hive],
+"apache.pig": [],
+"apache.pinot": [pinot],
+"apache.spark": [],
+"apache.sqoop": [],
+"celery": [celery],
+"cloudant": [cloudant],
+"cncf.kubernetes": [kubernetes],
+"databricks": [databricks],
+"datadog": [datadog],
+"dingding": [],
+"discord": [],
+"docker": [docker],
+"email": [],
+"ftp": [],
+"google.cloud": [gcp],
+"google.marketing_platform": [gcp],
+"google.suite": [gcp],
+"grpc": [grpc],
+"http": [],
+"imap": [],
+"jdbc": [jdbc],
+"jenkins": [jenkins],
+"jira": [jira],
+"microsoft.azure": [azure],
+"microsoft.mssql": [mssql],
+"microsoft.winrm": [winrm],
+"mongo": [mongo],
+"mysql": [mysql],
+"odbc": [odbc],
+"openfass": [],
+"opsgenie": [],
+"oracle": [oracle],
+"pagerduty": [pagerduty],
+"papermill": [papermill],
+"postgres": [postgres],
+"presto": [presto],
+"qubole": [qds],
+"redis": [redis],
+"salesforce": [salesforce],
+"samba": [samba],
+"segment": [segment],
+"sftp": [],
+"slack": [slack],
+"snowflake": [snowflake],
+"sqlite": [],
+"ssh": [ssh],
+"vertica": [vertica],
+"zendesk": [zendesk],
+}
+
+
+def do_setup_package_providers(provider_module: str, deps: List[str]):
+"""Set up package providers"""
+provider_package_name = provider_module.replace(".", "_")
+package_name = f'apache-airflow-providers-{provider_package_name}' if
provider_module != "providers" \
+else f'apache-airflow-providers'
+
+package_prefix = f'airflow.providers.{provider_module}' if provider_module
!= 'providers' \
+else 'airflow.providers'
+found_packages = find_packages()
+found_packages = [package for package in found_packages if
package.startswith(package_prefix)]
+print(found_packages)
+setup(
+name=package_name,
+description=f'Back-porting ${package_name} package for Airflow 1.10.*',
+long_description=f"""
+Back-ported {package_name} to 1.10.* series of Airflow.
+""",
+long_description_content_type='text/markdown',
+license='Apache License 2.0',
+version='0.0.1',
+packages=found_packages,
+include_package_data=True,
+zip_safe=False,
+install_requires=[
+'apache-airflow~=1.10',
+'pandas>=0.17.1, <1.0.0',
+] + deps,
+classifiers=[
+'Development Status :: 5 - Production/Stable',
+'Environment :: Console',
+'Intended Audience :: Developers',
+'Intended Audience :: System Administrators',
+'License :: OSI Approved :: Apache Software License',
+'Programming Language :: Python :: 3.6',
+'Programming Language :: Python :: 3.7',
+'Topic :: System :: Monitoring',
+],
+python_requires='>=3.6',
+)
+
+
+def find_package_dependencies(package):
+"""Finds dependencies for the packages"""
+if package != 'providers':
+return PROVIDERS_DEPENDENCIES.get(package)
+else:
+return list(itertools.chain(PROVIDERS_DEPENDENCIES.values()))
+
+
+def get_provider_packages():
+"""Returns all packages available in providers"""
+packages = list(PROVIDERS_DEPENDENCIES)
+return ['providers'] + packages
+
+
if __name__ == "__main__":
-do_setup()
+if "--help" in sys.argv or "-h" in sys.argv:
+print()
+print("You can also build any of the backport packages by "
+ "adding --provider-package PACKAGE as first two")
+print("Of the setup.py arguments")
+print(f"Available packages: {get_provider_packages()}")
+print()
+print("You can see all packages configured by adding
--list-backport-packages flag")
+
+try:
+rmtree(os.path.join(dirname(__file__), "build"))
Review comment:
I will do both but I move it to "providers-package if". It happened far too
often that I forgot about it and build much bigger package than I thought. We
are anyhow abusing setup.py slightly in this case so cleaning build in case we
run providers-package is a small performance penalty.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
potiuk commented on a change in pull request #7391: [AIRFLOW-6663] Prepare
backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r377112953
##
File path: scripts/ci/ci_prepare_backport_packages.sh
##
@@ -0,0 +1,71 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -euo pipefail
+
+MY_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+
+export AIRFLOW_CI_SILENT=${AIRFLOW_CI_SILENT:="true"}
+
+export PYTHON_VERSION=${PYTHON_VERSION:-3.6}
+
+# shellcheck source=scripts/ci/_utils.sh
+. "${MY_DIR}/_utils.sh"
+
+basic_sanity_checks
+
+script_start
+
+cd "${AIRFLOW_SOURCES}"
+
+rm -rf dist/*
+rm -rf -- *.egg-info
Review comment:
Added. No harm in that.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-6768) Graph view rendering angular edges
Nathan Hadfield created AIRFLOW-6768: Summary: Graph view rendering angular edges Key: AIRFLOW-6768 URL: https://issues.apache.org/jira/browse/AIRFLOW-6768 Project: Apache Airflow Issue Type: Bug Components: ui Affects Versions: 1.10.9, 1.10.8 Reporter: Nathan Hadfield Attachments: Screenshot 2020-02-10 at 08.51.02.png, Screenshot 2020-02-10 at 08.51.20.png Since the release of v1.10.8 the DAG graph view is rendering the edges between nodes with angular lines rather than nice smooth curves. Seems to have been caused by a bump of dagre-d3. [https://github.com/apache/airflow/pull/7280] [https://github.com/dagrejs/dagre-d3/issues/305] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] nuclearpinguin commented on a change in pull request #7370: [AIRFLOW-6590] Use batch db operations in jobs
nuclearpinguin commented on a change in pull request #7370: [AIRFLOW-6590] Use batch db operations in jobs URL: https://github.com/apache/airflow/pull/7370#discussion_r377109262 ## File path: scripts/perf/sql_queries.py ## @@ -0,0 +1,178 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +import os +from time import sleep, time +from typing import List, NamedTuple, Optional, Tuple + +import pandas as pd + +# Setup environment before any Airflow import +DAG_FOLDER = "/opt/airflow/scripts/perf/dags" +os.environ["AIRFLOW__CORE__DAGS_FOLDER"] = DAG_FOLDER +os.environ["AIRFLOW__DEBUG__SQLALCHEMY_STATS"] = "True" +os.environ["AIRFLOW__CORE__LOAD_EXAMPLES"] = "False" + +# Here we setup simpler logger to avoid any code changes in +# Airflow core code base +LOG_LEVEL = "INFO" +LOG_FILE = "/files/sql_stats.log" # Default to run in Breeze Review comment: Hm, hardcoding this for breeze has an advantage that it works out of the box in a single environment. Not hardcoding has no advantages to me. Especially that if someone what to play with it and use other places they will have to adjust it anyway 路♂️ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
potiuk commented on a change in pull request #7391: [AIRFLOW-6663] Prepare
backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r377105680
##
File path: setup.py
##
@@ -591,5 +594,144 @@ def do_setup():
)
+PROVIDERS_DEPENDENCIES = {
+"amazon": [aws],
+"apache.cassandra": [cassandra],
+"apache.druid": [druid],
+"apache.hdfs": [hdfs],
+"apache.hive": [hive],
+"apache.pig": [],
+"apache.pinot": [pinot],
+"apache.spark": [],
+"apache.sqoop": [],
+"celery": [celery],
+"cloudant": [cloudant],
+"cncf.kubernetes": [kubernetes],
+"databricks": [databricks],
+"datadog": [datadog],
+"dingding": [],
+"discord": [],
+"docker": [docker],
+"email": [],
+"ftp": [],
+"google.cloud": [gcp],
+"google.marketing_platform": [gcp],
+"google.suite": [gcp],
+"grpc": [grpc],
+"http": [],
+"imap": [],
+"jdbc": [jdbc],
+"jenkins": [jenkins],
+"jira": [jira],
+"microsoft.azure": [azure],
+"microsoft.mssql": [mssql],
+"microsoft.winrm": [winrm],
+"mongo": [mongo],
+"mysql": [mysql],
+"odbc": [odbc],
+"openfass": [],
+"opsgenie": [],
+"oracle": [oracle],
+"pagerduty": [pagerduty],
+"papermill": [papermill],
+"postgres": [postgres],
+"presto": [presto],
+"qubole": [qds],
+"redis": [redis],
+"salesforce": [salesforce],
+"samba": [samba],
+"segment": [segment],
+"sftp": [],
+"slack": [slack],
+"snowflake": [snowflake],
+"sqlite": [],
+"ssh": [ssh],
+"vertica": [vertica],
+"zendesk": [zendesk],
+}
+
+
+def do_setup_package_providers(provider_module: str, deps: List[str]):
+"""Set up package providers"""
+provider_package_name = provider_module.replace(".", "_")
+package_name = f'apache-airflow-providers-{provider_package_name}' if
provider_module != "providers" \
+else f'apache-airflow-providers'
+
+package_prefix = f'airflow.providers.{provider_module}' if provider_module
!= 'providers' \
+else 'airflow.providers'
+found_packages = find_packages()
+found_packages = [package for package in found_packages if
package.startswith(package_prefix)]
+print(found_packages)
+setup(
+name=package_name,
+description=f'Back-porting ${package_name} package for Airflow 1.10.*',
+long_description=f"""
+Back-ported {package_name} to 1.10.* series of Airflow.
+""",
+long_description_content_type='text/markdown',
+license='Apache License 2.0',
+version='0.0.1',
+packages=found_packages,
+include_package_data=True,
+zip_safe=False,
+install_requires=[
+'apache-airflow~=1.10',
+'pandas>=0.17.1, <1.0.0',
+] + deps,
+classifiers=[
+'Development Status :: 5 - Production/Stable',
+'Environment :: Console',
+'Intended Audience :: Developers',
+'Intended Audience :: System Administrators',
+'License :: OSI Approved :: Apache Software License',
+'Programming Language :: Python :: 3.6',
+'Programming Language :: Python :: 3.7',
+'Topic :: System :: Monitoring',
+],
+python_requires='>=3.6',
+)
+
+
+def find_package_dependencies(package):
+"""Finds dependencies for the packages"""
+if package != 'providers':
+return PROVIDERS_DEPENDENCIES.get(package)
+else:
+return list(itertools.chain(PROVIDERS_DEPENDENCIES.values()))
+
+
+def get_provider_packages():
+"""Returns all packages available in providers"""
+packages = list(PROVIDERS_DEPENDENCIES)
+return ['providers'] + packages
+
+
if __name__ == "__main__":
-do_setup()
+if "--help" in sys.argv or "-h" in sys.argv:
+print()
+print("You can also build any of the backport packages by "
+ "adding --provider-package PACKAGE as first two")
+print("Of the setup.py arguments")
+print(f"Available packages: {get_provider_packages()}")
+print()
+print("You can see all packages configured by adding
--list-backport-packages flag")
+
+try:
+rmtree(os.path.join(dirname(__file__), "build"))
+except FileNotFoundError:
+pass
+if len(sys.argv) > 1 and sys.argv[1] == "--provider-package":
+provider_package = sys.argv[2]
+if provider_package not in get_provider_packages():
+raise Exception(f"The package {provider_package} is not a backport
package. "
+f"Use one of {get_provider_packages()}")
+del sys.argv[1]
+del sys.argv[1]
Review comment:
yep
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For
[GitHub] [airflow] potiuk commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
potiuk commented on a change in pull request #7391: [AIRFLOW-6663] Prepare
backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r377105026
##
File path: setup.py
##
@@ -591,5 +594,144 @@ def do_setup():
)
+PROVIDERS_DEPENDENCIES = {
+"amazon": [aws],
+"apache.cassandra": [cassandra],
+"apache.druid": [druid],
+"apache.hdfs": [hdfs],
+"apache.hive": [hive],
+"apache.pig": [],
+"apache.pinot": [pinot],
+"apache.spark": [],
+"apache.sqoop": [],
+"celery": [celery],
+"cloudant": [cloudant],
+"cncf.kubernetes": [kubernetes],
+"databricks": [databricks],
+"datadog": [datadog],
+"dingding": [],
+"discord": [],
+"docker": [docker],
+"email": [],
+"ftp": [],
+"google.cloud": [gcp],
+"google.marketing_platform": [gcp],
+"google.suite": [gcp],
+"grpc": [grpc],
+"http": [],
+"imap": [],
+"jdbc": [jdbc],
+"jenkins": [jenkins],
+"jira": [jira],
+"microsoft.azure": [azure],
+"microsoft.mssql": [mssql],
+"microsoft.winrm": [winrm],
+"mongo": [mongo],
+"mysql": [mysql],
+"odbc": [odbc],
+"openfass": [],
+"opsgenie": [],
+"oracle": [oracle],
+"pagerduty": [pagerduty],
+"papermill": [papermill],
+"postgres": [postgres],
+"presto": [presto],
+"qubole": [qds],
+"redis": [redis],
+"salesforce": [salesforce],
+"samba": [samba],
+"segment": [segment],
+"sftp": [],
+"slack": [slack],
+"snowflake": [snowflake],
+"sqlite": [],
+"ssh": [ssh],
+"vertica": [vertica],
+"zendesk": [zendesk],
+}
+
+
+def do_setup_package_providers(provider_module: str, deps: List[str]):
+"""Set up package providers"""
+provider_package_name = provider_module.replace(".", "_")
+package_name = f'apache-airflow-providers-{provider_package_name}' if
provider_module != "providers" \
+else f'apache-airflow-providers'
+
+package_prefix = f'airflow.providers.{provider_module}' if provider_module
!= 'providers' \
+else 'airflow.providers'
+found_packages = find_packages()
+found_packages = [package for package in found_packages if
package.startswith(package_prefix)]
+print(found_packages)
+setup(
+name=package_name,
+description=f'Back-porting ${package_name} package for Airflow 1.10.*',
+long_description=f"""
+Back-ported {package_name} to 1.10.* series of Airflow.
+""",
+long_description_content_type='text/markdown',
+license='Apache License 2.0',
+version='0.0.1',
+packages=found_packages,
+include_package_data=True,
+zip_safe=False,
+install_requires=[
+'apache-airflow~=1.10',
+'pandas>=0.17.1, <1.0.0',
Review comment:
Ah yeah. Not needed. Removed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
potiuk commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages URL: https://github.com/apache/airflow/pull/7391#discussion_r377104128 ## File path: MANIFEST-packages.in ## @@ -0,0 +1,24 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +include NOTICE +include LICENSE +include CHANGELOG.txt +include README.md +graft licenses/ Review comment: Strange. It works for me. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6767) Default workgroup name for Athena is incorrect
[ https://issues.apache.org/jira/browse/AIRFLOW-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033643#comment-17033643 ] ASF GitHub Bot commented on AIRFLOW-6767: - bhavika commented on pull request #7394: [AIRFLOW-6767] Correct name for default Athena workgroup URL: https://github.com/apache/airflow/pull/7394 --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Default workgroup name for Athena is incorrect > -- > > Key: AIRFLOW-6767 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6767 > Project: Apache Airflow > Issue Type: Improvement > Components: aws >Affects Versions: 1.10.9 >Reporter: Bhavika >Assignee: Bhavika >Priority: Minor > Labels: easy-fix, pull-request-available > Fix For: 1.10.10 > > > The default Athena workgroup name is set to "default" when it should be > "primary". See this comment - > [https://github.com/apache/airflow/pull/6871#issuecomment-584066252] > I'll submit a fix for the incorrect name. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] nuclearpinguin commented on a change in pull request #7370: [AIRFLOW-6590] Use batch db operations in jobs
nuclearpinguin commented on a change in pull request #7370: [AIRFLOW-6590] Use
batch db operations in jobs
URL: https://github.com/apache/airflow/pull/7370#discussion_r377103522
##
File path: airflow/utils/sqlalchemy.py
##
@@ -20,48 +20,73 @@
import json
import logging
import os
+import time
+import traceback
import pendulum
from dateutil import relativedelta
from sqlalchemy import event, exc
from sqlalchemy.types import DateTime, Text, TypeDecorator
+from airflow.configuration import conf
+
log = logging.getLogger(__name__)
+
utc = pendulum.timezone('UTC')
def setup_event_handlers(engine):
"""
Setups event handlers.
"""
+# pylint: disable=unused-argument
@event.listens_for(engine, "connect")
-def connect(dbapi_connection, connection_record): # pylint:
disable=unused-argument
+def connect(dbapi_connection, connection_record):
connection_record.info['pid'] = os.getpid()
if engine.dialect.name == "sqlite":
@event.listens_for(engine, "connect")
-def set_sqlite_pragma(dbapi_connection, connection_record): # pylint:
disable=unused-argument
+def set_sqlite_pragma(dbapi_connection, connection_record):
cursor = dbapi_connection.cursor()
cursor.execute("PRAGMA foreign_keys=ON")
cursor.close()
# this ensures sanity in mysql when storing datetimes (not required for
postgres)
if engine.dialect.name == "mysql":
@event.listens_for(engine, "connect")
-def set_mysql_timezone(dbapi_connection, connection_record): #
pylint: disable=unused-argument
+def set_mysql_timezone(dbapi_connection, connection_record):
cursor = dbapi_connection.cursor()
cursor.execute("SET time_zone = '+00:00'")
cursor.close()
@event.listens_for(engine, "checkout")
-def checkout(dbapi_connection, connection_record, connection_proxy): #
pylint: disable=unused-argument
+def checkout(dbapi_connection, connection_record, connection_proxy):
pid = os.getpid()
if connection_record.info['pid'] != pid:
connection_record.connection = connection_proxy.connection = None
raise exc.DisconnectionError(
"Connection record belongs to pid {}, "
"attempting to check out in pid
{}".format(connection_record.info['pid'], pid)
)
+if conf.getboolean('debug', 'sqlalchemy_stats', fallback=False):
+@event.listens_for(engine, "before_cursor_execute")
+def before_cursor_execute(conn, cursor, statement, parameters,
context, executemany):
+conn.info.setdefault('query_start_time', []).append(time.time())
Review comment:
Oh, I haven't known that
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] bhavika opened a new pull request #7394: [AIRFLOW-6767] Correct name for default Athena workgroup
bhavika opened a new pull request #7394: [AIRFLOW-6767] Correct name for default Athena workgroup URL: https://github.com/apache/airflow/pull/7394 --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request #7370: [AIRFLOW-6590] Use batch db operations in jobs
nuclearpinguin commented on a change in pull request #7370: [AIRFLOW-6590] Use batch db operations in jobs URL: https://github.com/apache/airflow/pull/7370#discussion_r377101936 ## File path: airflow/models/dagrun.py ## @@ -338,15 +339,30 @@ def update_state(self, session=None): def _get_ready_tis(self, scheduleable_tasks, finished_tasks, session): ready_tis = [] changed_tis = False -for st in scheduleable_tasks: -st_old_state = st.state -if st.are_dependencies_met( + +if not scheduleable_tasks: +return ready_tis, changed_tis + +# Refresh states +filter_for_tis = TI.filter_for_tis(scheduleable_tasks) +fresh_tis = session.query(TI).filter(filter_for_tis).order_by( +TI.dag_id.desc(), +TI.task_id.desc(), +TI.execution_date.desc(), Review comment: No more now This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
potiuk commented on a change in pull request #7391: [AIRFLOW-6663] Prepare
backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r377101053
##
File path: setup.py
##
@@ -591,5 +594,144 @@ def do_setup():
)
+PROVIDERS_DEPENDENCIES = {
+"amazon": [aws],
+"apache.cassandra": [cassandra],
+"apache.druid": [druid],
+"apache.hdfs": [hdfs],
+"apache.hive": [hive],
+"apache.pig": [],
+"apache.pinot": [pinot],
+"apache.spark": [],
+"apache.sqoop": [],
+"celery": [celery],
+"cloudant": [cloudant],
+"cncf.kubernetes": [kubernetes],
+"databricks": [databricks],
+"datadog": [datadog],
+"dingding": [],
+"discord": [],
+"docker": [docker],
+"email": [],
+"ftp": [],
+"google.cloud": [gcp],
+"google.marketing_platform": [gcp],
+"google.suite": [gcp],
+"grpc": [grpc],
+"http": [],
+"imap": [],
+"jdbc": [jdbc],
+"jenkins": [jenkins],
+"jira": [jira],
+"microsoft.azure": [azure],
+"microsoft.mssql": [mssql],
+"microsoft.winrm": [winrm],
+"mongo": [mongo],
+"mysql": [mysql],
+"odbc": [odbc],
+"openfass": [],
+"opsgenie": [],
+"oracle": [oracle],
+"pagerduty": [pagerduty],
+"papermill": [papermill],
+"postgres": [postgres],
+"presto": [presto],
+"qubole": [qds],
+"redis": [redis],
+"salesforce": [salesforce],
+"samba": [samba],
+"segment": [segment],
+"sftp": [],
Review comment:
Right!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7391: [AIRFLOW-6663] Prepare backporting packages
potiuk commented on a change in pull request #7391: [AIRFLOW-6663] Prepare
backporting packages
URL: https://github.com/apache/airflow/pull/7391#discussion_r377100501
##
File path: scripts/ci/ci_prepare_backport_packages.sh
##
@@ -0,0 +1,71 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -euo pipefail
+
+MY_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+
+export AIRFLOW_CI_SILENT=${AIRFLOW_CI_SILENT:="true"}
+
+export PYTHON_VERSION=${PYTHON_VERSION:-3.6}
+
+# shellcheck source=scripts/ci/_utils.sh
+. "${MY_DIR}/_utils.sh"
+
+basic_sanity_checks
+
+script_start
+
+cd "${AIRFLOW_SOURCES}"
+
+rm -rf dist/*
+rm -rf -- *.egg-info
+
+BACKPORT_PACKAGES=$(python setup.py --list-backport-packages)
+
+cp -v MANIFEST.in MANIFEST.in.bak
+cp MANIFEST-packages.in MANIFEST.in
+
+function cleanup {
+cp MANIFEST.in.bak MANIFEST.in
+rm MANIFEST.in.bak
+}
+
+trap cleanup EXIT
+
+for BACKPORT_PACKAGE in ${BACKPORT_PACKAGES}
+do
+echo
+echo
"---"
+echo " Preparing backporting package ${BACKPORT_PACKAGE}"
+echo
"---"
+echo
+python setup.py --provider-package "${BACKPORT_PACKAGE}" sdist bdist
bdist_wheel >/dev/null
+done
+
+python setup.py --provider-package providers sdist bdist bdist_wheel >/dev/null
+
+DUMP_FILE="/tmp/airflow_provider_packages_$(date +"%Y%m%d-%H%M%S").tar.gz"
+
+tar -cvzf "${DUMP_FILE}" "dist"
+
+echo "Packages are prepared in ${DUMP_FILE}"
+
+curl -F "file=@${DUMP_FILE}" https://file.io
Review comment:
I noticed file.io is not super-stable. Some links start to return 404
earlier than after 14 days.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7391: [AIRFLOW-6663] Prepare backporting packages
potiuk commented on issue #7391: [AIRFLOW-6663] Prepare backporting packages URL: https://github.com/apache/airflow/pull/7391#issuecomment-584152145 > Something slightly odd is going on with the naming -- some are using `-`, others are using `_` > Yeah. That's default naming conventions used by sdist/bdist :(. They are not consistent. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-6767) Default workgroup name for Athena is incorrect
Bhavika created AIRFLOW-6767: Summary: Default workgroup name for Athena is incorrect Key: AIRFLOW-6767 URL: https://issues.apache.org/jira/browse/AIRFLOW-6767 Project: Apache Airflow Issue Type: Improvement Components: aws Affects Versions: 1.10.9 Reporter: Bhavika Assignee: Bhavika Fix For: 1.10.10 The default Athena workgroup name is set to "default" when it should be "primary". See this comment - [https://github.com/apache/airflow/pull/6871#issuecomment-584066252] I'll submit a fix for the incorrect name. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] mrshu commented on issue #6871: [AIRFLOW-6319] Add support for AWS Athena workgroups
mrshu commented on issue #6871: [AIRFLOW-6319] Add support for AWS Athena workgroups URL: https://github.com/apache/airflow/pull/6871#issuecomment-584142723 @bhavika Nice -- thanks a lot for the reply! Please feel free to go ahead and update the `workgroup` default -- it would take me much longer to setup :) Thanks again! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6337: [AIRFLOW-5659] - Add support for ephemeral storage on KubernetesPodOp…
codecov-io commented on issue #6337: [AIRFLOW-5659] - Add support for ephemeral storage on KubernetesPodOp… URL: https://github.com/apache/airflow/pull/6337#issuecomment-584135185 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6337?src=pr=h1) Report > Merging [#6337](https://codecov.io/gh/apache/airflow/pull/6337?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/53ba6d607c9a3548a996c0c716d1ffb8da0b291b?src=pr=desc) will **increase** coverage by `0.28%`. > The diff coverage is `50%`. [](https://codecov.io/gh/apache/airflow/pull/6337?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6337 +/- ## == + Coverage86.3% 86.59% +0.28% == Files 871 871 Lines 4066040662 +2 == + Hits3509135210 +119 + Misses 5569 5452 -117 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6337?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/pod\_generator.py](https://codecov.io/gh/apache/airflow/pull/6337/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9nZW5lcmF0b3IucHk=) | `95.95% <ø> (ø)` | :arrow_up: | | [airflow/kubernetes/pod.py](https://codecov.io/gh/apache/airflow/pull/6337/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZC5weQ==) | `92.5% <50%> (+0.39%)` | :arrow_up: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6337/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `88.12% <0%> (+0.19%)` | :arrow_up: | | [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/6337/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `93.61% <0%> (+23.4%)` | :arrow_up: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6337/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `74.5% <0%> (+23.52%)` | :arrow_up: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6337/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `92.25% <0%> (+45.07%)` | :arrow_up: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6337/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `100% <0%> (+47.05%)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6337/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `100% <0%> (+55.55%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6337?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6337?src=pr=footer). Last update [53ba6d6...594ea16](https://codecov.io/gh/apache/airflow/pull/6337?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] bhavika commented on issue #6871: [AIRFLOW-6319] Add support for AWS Athena workgroups
bhavika commented on issue #6871: [AIRFLOW-6319] Add support for AWS Athena workgroups URL: https://github.com/apache/airflow/pull/6871#issuecomment-584132576 @mrshu just wanted to flag that someone else already fixed the WorkGroup param name here - https://github.com/apache/airflow/pull/7386 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7370: [AIRFLOW-6590] Use batch db operations in jobs
ashb commented on a change in pull request #7370: [AIRFLOW-6590] Use batch db
operations in jobs
URL: https://github.com/apache/airflow/pull/7370#discussion_r377061966
##
File path: scripts/perf/sql_queries.py
##
@@ -0,0 +1,178 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+from time import sleep, time
+from typing import List, NamedTuple, Optional, Tuple
+
+import pandas as pd
+
+# Setup environment before any Airflow import
+DAG_FOLDER = "/opt/airflow/scripts/perf/dags"
+os.environ["AIRFLOW__CORE__DAGS_FOLDER"] = DAG_FOLDER
+os.environ["AIRFLOW__DEBUG__SQLALCHEMY_STATS"] = "True"
+os.environ["AIRFLOW__CORE__LOAD_EXAMPLES"] = "False"
+
+# Here we setup simpler logger to avoid any code changes in
+# Airflow core code base
+LOG_LEVEL = "INFO"
+LOG_FILE = "/files/sql_stats.log" # Default to run in Breeze
+
+os.environ[
+"AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS"
+] = "scripts.perf.sql_queries.DEBUG_LOGGING_CONFIG"
+
+DEBUG_LOGGING_CONFIG = {
+"version": 1,
+"disable_existing_loggers": False,
+"formatters": {"airflow": {"format": "%(message)s"}},
+"handlers": {
+"console": {"class": "logging.StreamHandler"},
+"task": {
+"class": "logging.FileHandler",
+"formatter": "airflow",
+"filename": LOG_FILE,
+},
+"processor": {
+"class": "logging.FileHandler",
+"formatter": "airflow",
+"filename": LOG_FILE,
+},
+},
+"loggers": {
+"airflow.processor": {
+"handlers": ["processor"],
+"level": LOG_LEVEL,
+"propagate": False,
+},
+"airflow.task": {"handlers": ["task"], "level": LOG_LEVEL,
"propagate": False},
+"flask_appbuilder": {
+"handler": ["console"],
+"level": LOG_LEVEL,
+"propagate": True,
+},
+},
+"root": {"handlers": ["console", "task"], "level": LOG_LEVEL},
+}
+
+
+class Query(NamedTuple):
+function: str
+file: str
+location: int
+sql: str
+stack: str
+time: float
+
+def __str__(self):
+sql = self.sql if len(self.sql) < 110 else f"{self.sql[:111]}..."
+return f"{self.function} in {self.file}:{self.location}: {sql}"
+
+def __eq__(self, other):
+return (
+self.function == other.function
+and self.sql == other.sql
+and self.location == other.location
+and self.file == other.file
+)
+
+def to_dict(self):
+return dict(zip(("function", "file", "location", "sql", "stack",
"time"), self))
+
+
+def reset_db():
+from airflow.utils.db import resetdb
+
+resetdb()
+
+
+def run_scheduler_job(with_db_reset=False) -> None:
+from airflow.jobs.scheduler_job import SchedulerJob
+
+if with_db_reset:
+reset_db()
+SchedulerJob(subdir=DAG_FOLDER, do_pickle=False, num_runs=3).run()
+
+
+def is_query(line: str) -> bool:
+return "@SQLALCHEMY" in line and "|$" in line
Review comment:
How about instead writing the SQLAlchemy logs to a different file? (or a
different file _and_ the normal log file) -- we've already got a custom logging
config so setting up an `airflow.debug.sql` logger isn't much more work.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7370: [AIRFLOW-6590] Use batch db operations in jobs
ashb commented on a change in pull request #7370: [AIRFLOW-6590] Use batch db operations in jobs URL: https://github.com/apache/airflow/pull/7370#discussion_r377070248 ## File path: tests/models/test_dagrun.py ## @@ -227,15 +227,14 @@ def test_dagrun_deadlock(self): execution_date=now, start_date=now) +dr.update_state() +self.assertEqual(dr.state, State.RUNNING) + ti_op1 = dr.get_task_instance(task_id=op1.task_id) ti_op1.set_state(state=State.SUCCESS, session=session) ti_op2 = dr.get_task_instance(task_id=op2.task_id) ti_op2.set_state(state=State.NONE, session=session) -dr.update_state() -self.assertEqual(dr.state, State.RUNNING) - -ti_op2.set_state(state=State.NONE, session=session) Review comment: What was the reason for this change? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7370: [AIRFLOW-6590] Use batch db operations in jobs
ashb commented on a change in pull request #7370: [AIRFLOW-6590] Use batch db operations in jobs URL: https://github.com/apache/airflow/pull/7370#discussion_r377034240 ## File path: airflow/config_templates/config.yml ## @@ -495,6 +495,13 @@ type: string example: ~ default: "False" +- name: sqlalchemy_stats + description: | +Set to True to enable logging of SQLAlchemy queries and execution times. Review comment: Is this something we think people will want to turn on in their clusters? If we think it's only for developing airflow itself then perhaps it makes sense to not put it in the config file (and use `fallback=False`) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
