[jira] [Commented] (AIRFLOW-3830) Remove DagBag from /dag_details
[
https://issues.apache.org/jira/browse/AIRFLOW-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783024#comment-16783024
]
ASF GitHub Bot commented on AIRFLOW-3830:
-
ffinfo commented on pull request #4831: [AIRFLOW-3830] Remove DagBag from
/dag_details
URL: https://github.com/apache/airflow/pull/4831
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title.
- https://issues.apache.org/jira/browse/AIRFLOW-3830
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Remove DagBag from /dag_details
> ---
>
> Key: AIRFLOW-3830
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3830
> Project: Apache Airflow
> Issue Type: Improvement
> Components: webserver
>Reporter: Peter van 't Hof
>Assignee: Peter van 't Hof
>Priority: Major
> Fix For: 2.0.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] XD-DENG commented on issue #4801: [AIRFLOW-XXX] Unpin cryptography (2.6.1 fixes issue in 2.6)
XD-DENG commented on issue #4801: [AIRFLOW-XXX] Unpin cryptography (2.6.1 fixes issue in 2.6) URL: https://github.com/apache/airflow/pull/4801#issuecomment-469137865 @feng-tao actually no bug fixes in 2.6 or 2.6.1. Pinning the upper bound earlier in https://github.com/apache/airflow/pull/4800 was only due to wrong wheel build in 2.6, not breaking change, so I thought it may be good to unpin it given 2.6.1 works well. It's ok to me to keep the upper bound as well. Please feel free to close this PR in that case. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feng-tao commented on issue #4648: [AIRFLOW-3274] Add run_as_user and fs_group options for Kubernetes
feng-tao commented on issue #4648: [AIRFLOW-3274] Add run_as_user and fs_group options for Kubernetes URL: https://github.com/apache/airflow/pull/4648#issuecomment-469137465 cc @dimberman in case he has other comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feng-tao commented on issue #4648: [AIRFLOW-3274] Add run_as_user and fs_group options for Kubernetes
feng-tao commented on issue #4648: [AIRFLOW-3274] Add run_as_user and fs_group options for Kubernetes URL: https://github.com/apache/airflow/pull/4648#issuecomment-469137373 @pgagnon , LGTM, we could merge the pr once you rebase with master, This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ffinfo opened a new pull request #4831: [AIRFLOW-3830] Remove DagBag from /dag_details
ffinfo opened a new pull request #4831: [AIRFLOW-3830] Remove DagBag from
/dag_details
URL: https://github.com/apache/airflow/pull/4831
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title.
- https://issues.apache.org/jira/browse/AIRFLOW-3830
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Assigned] (AIRFLOW-3830) Remove DagBag from /dag_details
[ https://issues.apache.org/jira/browse/AIRFLOW-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter van 't Hof reassigned AIRFLOW-3830: - Assignee: Peter van 't Hof > Remove DagBag from /dag_details > --- > > Key: AIRFLOW-3830 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3830 > Project: Apache Airflow > Issue Type: Improvement > Components: webserver >Reporter: Peter van 't Hof >Assignee: Peter van 't Hof >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4004) Dataflow jobs only launched in us-central1 region
Wajid Khattak created AIRFLOW-4004:
--
Summary: Dataflow jobs only launched in us-central1 region
Key: AIRFLOW-4004
URL: https://issues.apache.org/jira/browse/AIRFLOW-4004
Project: Apache Airflow
Issue Type: Bug
Components: Dataflow, hooks
Affects Versions: 1.10.2
Environment: OS = Windows 7 Enterprise, Python = 2.7.12
Reporter: Wajid Khattak
Dataflow jobs can only be launched in us-central1 region. Raeson seems to be
that for launching jobs the REST endpoint
"https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.templates/launch;
is called as below in gcp_dataflow_hook.py:
{code:java}
...
request = service.projects().locations().templates().launch(
projectId=variables['project'],
location=variables['region'],
gcsPath=dataflow_template,
body=body
)
...
{code}
However, for checking the progress of the launched job, the REST endpoint
"https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.jobs/get;
is called as below in gcp_dataflow_hook.py:
{code:java}
...
def _get_job(self):
if self._job_id:
job = self._dataflow.projects().locations().jobs().get(
projectId=self._project_number,
location=self._job_location,
jobId=self._job_id).execute(num_retries=5)
elif self._job_name:
job = self._get_job_id_from_name()
else:
raise Exception('Missing both dataflow job ID and name.')
...{code}
The simple fix is to use the correct REST endpoint for launching jobs i.e
"https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.templates/launch;
so that the job is launched in the correct region as specified in the launch
parameters.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] feng-tao commented on issue #4801: [AIRFLOW-XXX] Unpin cryptography (2.6.1 fixes issue in 2.6)
feng-tao commented on issue #4801: [AIRFLOW-XXX] Unpin cryptography (2.6.1 fixes issue in 2.6) URL: https://github.com/apache/airflow/pull/4801#issuecomment-469123637 @XD-DENG , is there any bug fixes we need in the library 2.6.1 or 2.6 version? I prefer to pin the upper bound for now given they just broke the dependency recently. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feng-tao edited a comment on issue #4801: [AIRFLOW-XXX] Unpin cryptography (2.6.1 fixes issue in 2.6)
feng-tao edited a comment on issue #4801: [AIRFLOW-XXX] Unpin cryptography (2.6.1 fixes issue in 2.6) URL: https://github.com/apache/airflow/pull/4801#issuecomment-469123637 @XD-DENG , is there any bug fixes we need in the library 2.6.1 or 2.6 version? If not, I prefer to pin the upper bound for now given they just broke the dependency recently. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-4001) Update docs about how to run tests
[ https://issues.apache.org/jira/browse/AIRFLOW-4001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782972#comment-16782972 ] ASF GitHub Bot commented on AIRFLOW-4001: - feng-tao commented on pull request #4826: [AIRFLOW-4001] Update docs about how to run tests URL: https://github.com/apache/airflow/pull/4826 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Update docs about how to run tests > -- > > Key: AIRFLOW-4001 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4001 > Project: Apache Airflow > Issue Type: Bug >Reporter: Chao-Han Tsai >Assignee: Chao-Han Tsai >Priority: Major > > The steps in [CONTRIBUTING.md | > https://github.com/apache/airflow/blob/master/CONTRIBUTING.md ] about how to > run tests are out-dated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4001) Update docs about how to run tests
[ https://issues.apache.org/jira/browse/AIRFLOW-4001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4001. --- Resolution: Fixed > Update docs about how to run tests > -- > > Key: AIRFLOW-4001 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4001 > Project: Apache Airflow > Issue Type: Bug >Reporter: Chao-Han Tsai >Assignee: Chao-Han Tsai >Priority: Major > > The steps in [CONTRIBUTING.md | > https://github.com/apache/airflow/blob/master/CONTRIBUTING.md ] about how to > run tests are out-dated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] feng-tao merged pull request #4826: [AIRFLOW-4001] Update docs about how to run tests
feng-tao merged pull request #4826: [AIRFLOW-4001] Update docs about how to run tests URL: https://github.com/apache/airflow/pull/4826 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-4001) Update docs about how to run tests
[ https://issues.apache.org/jira/browse/AIRFLOW-4001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782973#comment-16782973 ] ASF subversion and git services commented on AIRFLOW-4001: -- Commit bbe711640f573b5db865f3c314ce8e11e9f66475 in airflow's branch refs/heads/master from Chao-Han Tsai [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=bbe7116 ] [AIRFLOW-4001] Update docs about how to run tests (#4826) fix docs > Update docs about how to run tests > -- > > Key: AIRFLOW-4001 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4001 > Project: Apache Airflow > Issue Type: Bug >Reporter: Chao-Han Tsai >Assignee: Chao-Han Tsai >Priority: Major > > The steps in [CONTRIBUTING.md | > https://github.com/apache/airflow/blob/master/CONTRIBUTING.md ] about how to > run tests are out-dated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] feng-tao commented on a change in pull request #4826: [AIRFLOW-4001] Update docs about how to run tests
feng-tao commented on a change in pull request #4826: [AIRFLOW-4001] Update docs about how to run tests URL: https://github.com/apache/airflow/pull/4826#discussion_r261925598 ## File path: CONTRIBUTING.md ## @@ -160,18 +160,17 @@ There are three ways to setup an Apache Airflow development environment. ```bash docker-compose -f scripts/ci/docker-compose.yml run airflow-testing bash # From the container - pip install -e .[devel] - # Run all the tests with python and mysql through tox - pip install tox - tox -e py35-backend_mysql + export TOX_ENV=py27-backend_mysql-env_docker + /app/scripts/ci/run-ci.sh ``` If you wish to run individual tests inside of Docker environment you can do as follows: ```bash -# From the container (with your desired environment) with druid hook -tox -e py35-backend_mysql -- tests/hooks/test_druid_hook.py Review comment: thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] xianping commented on a change in pull request #2256: [AIRFLOW-1153] Fix issue ' params ' don't pass to HiveOperator execution context
xianping commented on a change in pull request #2256: [AIRFLOW-1153] Fix issue
' params ' don't pass to HiveOperator execution context
URL: https://github.com/apache/airflow/pull/2256#discussion_r261925058
##
File path: airflow/utils/operator_helpers.py
##
@@ -43,7 +43,8 @@ def context_to_airflow_vars(context,
in_env_var_format=False):
:type in_env_var_format: bool
:return task_instance context as dict.
"""
-params = dict()
+params = context.get('params')
Review comment:
@feng-tao , can u tell add UT in which folder/path?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] xianping commented on a change in pull request #2256: [AIRFLOW-1153] Fix issue ' params ' don't pass to HiveOperator execution context
xianping commented on a change in pull request #2256: [AIRFLOW-1153] Fix issue
' params ' don't pass to HiveOperator execution context
URL: https://github.com/apache/airflow/pull/2256#discussion_r261924870
##
File path: airflow/utils/operator_helpers.py
##
@@ -43,7 +43,8 @@ def context_to_airflow_vars(context,
in_env_var_format=False):
:type in_env_var_format: bool
:return task_instance context as dict.
"""
-params = dict()
+params = context.get('params')
Review comment:
agree, you have a better defensive version
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] XD-DENG commented on issue #4801: [AIRFLOW-XXX] Unpin cryptography (2.6.1 fixes issue in 2.6)
XD-DENG commented on issue #4801: [AIRFLOW-XXX] Unpin cryptography (2.6.1 fixes issue in 2.6) URL: https://github.com/apache/airflow/pull/4801#issuecomment-469119232 Hi @feng-tao @Fokko a gentle ping. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] jmcarp opened a new pull request #4830: [AIRFLOW-3929] Use anchor tag for dag filter link.
jmcarp opened a new pull request #4830: [AIRFLOW-3929] Use anchor tag for dag
filter link.
URL: https://github.com/apache/airflow/pull/4830
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3929
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement
Proposal([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Replace a lingering button with an anchor to finish AIRFLOW-3929.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-3929) Modal buttons should be links
[
https://issues.apache.org/jira/browse/AIRFLOW-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782963#comment-16782963
]
ASF GitHub Bot commented on AIRFLOW-3929:
-
jmcarp commented on pull request #4830: [AIRFLOW-3929] Use anchor tag for dag
filter link.
URL: https://github.com/apache/airflow/pull/4830
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3929
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement
Proposal([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Replace a lingering button with an anchor to finish AIRFLOW-3929.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Modal buttons should be links
> -
>
> Key: AIRFLOW-3929
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3929
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: Josh Carp
>Assignee: Josh Carp
>Priority: Trivial
> Fix For: 2.0.0, 1.10.3
>
>
> Buttons in the dag view modal aren't links, so users can't copy the url they
> refer to, open the url in a new tab, etc. Instead, we change
> `window.location` when a user clicks the button. We should use tags here
> instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] milton0825 commented on a change in pull request #4826: [AIRFLOW-4001] Update docs about how to run tests
milton0825 commented on a change in pull request #4826: [AIRFLOW-4001] Update docs about how to run tests URL: https://github.com/apache/airflow/pull/4826#discussion_r261922058 ## File path: CONTRIBUTING.md ## @@ -160,18 +160,17 @@ There are three ways to setup an Apache Airflow development environment. ```bash docker-compose -f scripts/ci/docker-compose.yml run airflow-testing bash # From the container - pip install -e .[devel] - # Run all the tests with python and mysql through tox - pip install tox - tox -e py35-backend_mysql + export TOX_ENV=py27-backend_mysql-env_docker + /app/scripts/ci/run-ci.sh ``` If you wish to run individual tests inside of Docker environment you can do as follows: ```bash -# From the container (with your desired environment) with druid hook -tox -e py35-backend_mysql -- tests/hooks/test_druid_hook.py Review comment: There is some additional setup in the `run-ci.sh` not captured here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] milton0825 commented on a change in pull request #4826: [AIRFLOW-4001] Update docs about how to run tests
milton0825 commented on a change in pull request #4826: [AIRFLOW-4001] Update docs about how to run tests URL: https://github.com/apache/airflow/pull/4826#discussion_r261921938 ## File path: CONTRIBUTING.md ## @@ -160,18 +160,17 @@ There are three ways to setup an Apache Airflow development environment. ```bash docker-compose -f scripts/ci/docker-compose.yml run airflow-testing bash # From the container - pip install -e .[devel] - # Run all the tests with python and mysql through tox - pip install tox - tox -e py35-backend_mysql + export TOX_ENV=py27-backend_mysql-env_docker + /app/scripts/ci/run-ci.sh ``` If you wish to run individual tests inside of Docker environment you can do as follows: ```bash -# From the container (with your desired environment) with druid hook -tox -e py35-backend_mysql -- tests/hooks/test_druid_hook.py Review comment: This step and the step above does not work anymore. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] pgagnon commented on issue #4648: [AIRFLOW-3274] Add run_as_user and fs_group options for Kubernetes
pgagnon commented on issue #4648: [AIRFLOW-3274] Add run_as_user and fs_group options for Kubernetes URL: https://github.com/apache/airflow/pull/4648#issuecomment-469113268 @ashb Apologies for the mention and double apologies if you aren't the right person for this. This PR has been sitting for a little while now and I'd just like to get a reviewer with write access interested . This PR is important because it allows KubernetesExecutor to work on clusters with restrictive pod security policies in place. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] pgagnon commented on a change in pull request #4772: [AIRFLOW-3937] KubernetesPodOperator support for envFrom configMapRef…
pgagnon commented on a change in pull request #4772: [AIRFLOW-3937] KubernetesPodOperator support for envFrom configMapRef… URL: https://github.com/apache/airflow/pull/4772#discussion_r261918489 ## File path: airflow/contrib/kubernetes/secret.py ## @@ -39,3 +39,10 @@ def __init__(self, deploy_type, deploy_target, secret, key): self.deploy_target = deploy_target self.secret = secret self.key = key + + +class SecretObject: Review comment: It's not clear to me why this class is useful if it only contains a single `name` member. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-4003) Extract all example scripts to separate files in doc
[
https://issues.apache.org/jira/browse/AIRFLOW-4003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Tong updated AIRFLOW-4003:
---
Description:
By extracting example python script, we could have unittest on them and make
sure correction easily.
{quote}
Files can be stored in the {{airflow/example_dags}} directory or
{{airflow/contrib/example_dags}} directory. Files from this directory can be
automatically tested to confirm their correctness.
Example:
{code}
.. literalinclude:: ../../airflow/example_dags/example_python_operator.py
:language: python
:start-after: [START howto_operator_python_kwargs]
:end-before: [END howto_operator_python_kwargs]
{code}
Source:
[https://raw.githubusercontent.com/apache/airflow/master/docs/howto/operator.rst]
Other scripts are stored in py files.
{quote}
was:
By extracting example python script, we could have unittest on them and make
sure correction easily.
{quote}
Files can be stored in the {{airflow/example_dags}} directory or
{{airflow/contrib/example_dags}} directory. Files from this directory can be
automatically tested to confirm their correctness.
Example:
{{.. literalinclude:: ../../airflow/example_dags/example_python_operator.py
:language: python :start-after: [START howto_operator_python_kwargs]
:end-before: [END howto_operator_python_kwargs] }}
Source:
[https://raw.githubusercontent.com/apache/airflow/master/docs/howto/operator.rst]
Other scripts are stored in py files.
{quote}
> Extract all example scripts to separate files in doc
>
>
> Key: AIRFLOW-4003
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4003
> Project: Apache Airflow
> Issue Type: Improvement
> Components: Documentation
>Affects Versions: 1.10.2
>Reporter: Chen Tong
>Assignee: Chen Tong
>Priority: Major
> Fix For: 1.10.3
>
>
> By extracting example python script, we could have unittest on them and make
> sure correction easily.
>
> {quote}
> Files can be stored in the {{airflow/example_dags}} directory or
> {{airflow/contrib/example_dags}} directory. Files from this directory can be
> automatically tested to confirm their correctness.
> Example:
> {code}
> .. literalinclude:: ../../airflow/example_dags/example_python_operator.py
> :language: python
> :start-after: [START howto_operator_python_kwargs]
> :end-before: [END howto_operator_python_kwargs]
> {code}
> Source:
> [https://raw.githubusercontent.com/apache/airflow/master/docs/howto/operator.rst]
> Other scripts are stored in py files.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-4003) Extract all example scripts to separate files in doc
Chen Tong created AIRFLOW-4003:
--
Summary: Extract all example scripts to separate files in doc
Key: AIRFLOW-4003
URL: https://issues.apache.org/jira/browse/AIRFLOW-4003
Project: Apache Airflow
Issue Type: Improvement
Components: Documentation
Affects Versions: 1.10.2
Reporter: Chen Tong
Assignee: Chen Tong
Fix For: 1.10.3
By extracting example python script, we could have unittest on them and make
sure correction easily.
{quote}
Files can be stored in the {{airflow/example_dags}} directory or
{{airflow/contrib/example_dags}} directory. Files from this directory can be
automatically tested to confirm their correctness.
Example:
{{.. literalinclude:: ../../airflow/example_dags/example_python_operator.py
:language: python :start-after: [START howto_operator_python_kwargs]
:end-before: [END howto_operator_python_kwargs] }}
Source:
[https://raw.githubusercontent.com/apache/airflow/master/docs/howto/operator.rst]
Other scripts are stored in py files.
{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (AIRFLOW-3966) Correct default bigquery_conn_id in BigQueryTableSensor
[ https://issues.apache.org/jira/browse/AIRFLOW-3966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Yuan closed AIRFLOW-3966. -- Resolution: Won't Fix This will be fixed as part of [AIRFLOW-3987|https://issues.apache.org/jira/browse/AIRFLOW-3987] > Correct default bigquery_conn_id in BigQueryTableSensor > --- > > Key: AIRFLOW-3966 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3966 > Project: Apache Airflow > Issue Type: Bug >Affects Versions: 1.10.2 >Reporter: Ryan Yuan >Assignee: Ryan Yuan >Priority: Major > > In BigQueryTableSensor, the default value of bigquery_conn_id is > 'bigquery_default_conn', which should be 'bigquery_default' instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3326) High Sierra Complaining 'in progress in another thread when fork() was called'
[
https://issues.apache.org/jira/browse/AIRFLOW-3326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782914#comment-16782914
]
Ryan Yuan commented on AIRFLOW-3326:
[~AetherUnbound] It's v2.7.15
> High Sierra Complaining 'in progress in another thread when fork() was called'
> --
>
> Key: AIRFLOW-3326
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3326
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.10.0
> Environment: macOS High Sierra 10.13.6 (17G65)
>Reporter: Ryan Yuan
>Priority: Blocker
>
> Inside the plugins folder, I have a hook that is a child class of
> BigQueryHook.
> {code:java}
> // code
> from airflow.contrib.hooks.bigquery_hook import BigQueryHook
> class BQHook(BigQueryHook):
> pass{code}
> When I run the airflow server, it keeps throwing messages complaining 'in
> progress in another thread when fork() was called', and I can't use the web
> server UI at all.
> {code:java}
> // messages from terminal
> objc[15098]: +[__NSPlaceholderDate initialize] may have been in progress in
> another thread when fork() was called.
> objc[15098]: +[__NSPlaceholderDate initialize] may have been in progress in
> another thread when fork() was called. We cannot safely call it or ignore it
> in the fork() child process. Crashing instead. Set a breakpoint on
> objc_initializeAfterForkError to debug.
> [2018-11-12 14:03:40 +1100] [15102] [INFO] Booting worker with pid: 15102
> [2018-11-12 14:03:40,792] {__init__.py:51} INFO - Using executor
> SequentialExecutor
> [2018-11-12 14:03:40,851] {base_hook.py:83} INFO - Using connection to:
> https://custom-data-z00100-dev.appspot.com/
> objc[15099]: +[__NSPlaceholderDate initialize] may have been in progress in
> another thread when fork() was called.
> objc[15099]: +[__NSPlaceholderDate initialize] may have been in progress in
> another thread when fork() was called. We cannot safely call it or ignore it
> in the fork() child process. Crashing instead. Set a breakpoint on
> objc_initializeAfterForkError to debug.
> [2018-11-12 14:03:40 +1100] [15103] [INFO] Booting worker with pid: 15103
> [2018-11-12 14:03:40,902] {base_hook.py:83} INFO - Using connection to:
> https://custom-data-z00100-dev.appspot.com/
> objc[15101]: +[__NSPlaceholderDate initialize] may have been in progress in
> another thread when fork() was called.
> objc[15101]: +[__NSPlaceholderDate initialize] may have been in progress in
> another thread when fork() was called. We cannot safely call it or ignore it
> in the fork() child process. Crashing instead. Set a breakpoint on
> objc_initializeAfterForkError to debug.
> [2018-11-12 14:03:40 +1100] [15104] [INFO] Booting worker with pid: 15104
> [2018-11-12 14:03:40,948] {base_hook.py:83} INFO - Using connection to:
> https://custom-data-z00100-dev.appspot.com/
> objc[15100]: +[__NSPlaceholderDate initialize] may have been in progress in
> another thread when fork() was called.
> objc[15100]: +[__NSPlaceholderDate initialize] may have been in progress in
> another thread when fork() was called. We cannot safely call it or ignore it
> in the fork() child process. Crashing instead. Set a breakpoint on
> objc_initializeAfterForkError to debug.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] feng-tao commented on a change in pull request #4826: [AIRFLOW-4001] Update docs about how to run tests
feng-tao commented on a change in pull request #4826: [AIRFLOW-4001] Update docs about how to run tests URL: https://github.com/apache/airflow/pull/4826#discussion_r261899455 ## File path: CONTRIBUTING.md ## @@ -160,18 +160,17 @@ There are three ways to setup an Apache Airflow development environment. ```bash docker-compose -f scripts/ci/docker-compose.yml run airflow-testing bash # From the container - pip install -e .[devel] - # Run all the tests with python and mysql through tox - pip install tox - tox -e py35-backend_mysql + export TOX_ENV=py27-backend_mysql-env_docker + /app/scripts/ci/run-ci.sh ``` If you wish to run individual tests inside of Docker environment you can do as follows: ```bash -# From the container (with your desired environment) with druid hook -tox -e py35-backend_mysql -- tests/hooks/test_druid_hook.py Review comment: does this line doesn't work? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #4829: [AIRFLOW-3993] Add tests for salesforce hook
mik-laj commented on a change in pull request #4829: [AIRFLOW-3993] Add tests
for salesforce hook
URL: https://github.com/apache/airflow/pull/4829#discussion_r261888913
##
File path: airflow/contrib/hooks/salesforce_hook.py
##
@@ -18,127 +18,108 @@
# under the License.
#
"""
-This module contains a Salesforce Hook
-which allows you to connect to your Salesforce instance,
-retrieve data from it, and write that data to a file
-for other uses.
+This module contains a Salesforce Hook which allows you to connect to your
Salesforce instance,
+retrieve data from it, and write that data to a file for other uses.
-NOTE: this hook also relies on the simple_salesforce package:
-https://github.com/simple-salesforce/simple-salesforce
+NOTE: this hook also relies on the simple_salesforce package:
+ https://github.com/simple-salesforce/simple-salesforce
"""
-from simple_salesforce import Salesforce
-from airflow.hooks.base_hook import BaseHook
-
-import json
+import time
import pandas as pd
-import time
+from simple_salesforce import Salesforce
+from airflow.hooks.base_hook import BaseHook
from airflow.utils.log.logging_mixin import LoggingMixin
class SalesforceHook(BaseHook):
-def __init__(
-self,
-conn_id,
-*args,
-**kwargs
-):
+def __init__(self, conn_id):
"""
-Create new connection to Salesforce
-and allows you to pull data out of SFDC and save it to a file.
+Create new connection to Salesforce and allows you to pull data out of
SFDC and save it to a file.
+
+You can then use that file with other Airflow operators to move the
data into another data source.
-You can then use that file with other
-Airflow operators to move the data into another data source
+:param conn_id: the name of the connection that has the parameters we
need to connect to Salesforce.
+The connection should be type `http` and include a
user's security token
+in the `Extras` field.
+:type conn_id: str
-:param conn_id: the name of the connection that has the parameters
-we need to connect to Salesforce.
-The connection should be type `http` and include a
-user's security token in the `Extras` field.
.. note::
For the HTTP connection type, you can include a
JSON structure in the `Extras` field.
We need a user's security token to connect to Salesforce.
So we define it in the `Extras` field as:
`{"security_token":"YOUR_SECURITY_TOKEN"}`
+
"""
+super(SalesforceHook, self).__init__(conn_id)
self.conn_id = conn_id
-self._args = args
-self._kwargs = kwargs
-
-# get the connection parameters
-self.connection = self.get_connection(conn_id)
-self.extras = self.connection.extra_dejson
+self.salesforce = None
def sign_in(self):
Review comment:
```suggestion
def get_conn(self):
```
It is recommended that the `get_conn` method return the API library. Return
should be cached.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #4829: [AIRFLOW-3993] Add tests for salesforce hook
mik-laj commented on issue #4829: [AIRFLOW-3993] Add tests for salesforce hook URL: https://github.com/apache/airflow/pull/4829#issuecomment-469067789 Are all your changes backward compatible? If not, you must add a note in the `UPDATING.rst` file. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #4829: [AIRFLOW-3993] Add tests for salesforce hook
mik-laj commented on a change in pull request #4829: [AIRFLOW-3993] Add tests
for salesforce hook
URL: https://github.com/apache/airflow/pull/4829#discussion_r261888913
##
File path: airflow/contrib/hooks/salesforce_hook.py
##
@@ -18,127 +18,108 @@
# under the License.
#
"""
-This module contains a Salesforce Hook
-which allows you to connect to your Salesforce instance,
-retrieve data from it, and write that data to a file
-for other uses.
+This module contains a Salesforce Hook which allows you to connect to your
Salesforce instance,
+retrieve data from it, and write that data to a file for other uses.
-NOTE: this hook also relies on the simple_salesforce package:
-https://github.com/simple-salesforce/simple-salesforce
+NOTE: this hook also relies on the simple_salesforce package:
+ https://github.com/simple-salesforce/simple-salesforce
"""
-from simple_salesforce import Salesforce
-from airflow.hooks.base_hook import BaseHook
-
-import json
+import time
import pandas as pd
-import time
+from simple_salesforce import Salesforce
+from airflow.hooks.base_hook import BaseHook
from airflow.utils.log.logging_mixin import LoggingMixin
class SalesforceHook(BaseHook):
-def __init__(
-self,
-conn_id,
-*args,
-**kwargs
-):
+def __init__(self, conn_id):
"""
-Create new connection to Salesforce
-and allows you to pull data out of SFDC and save it to a file.
+Create new connection to Salesforce and allows you to pull data out of
SFDC and save it to a file.
+
+You can then use that file with other Airflow operators to move the
data into another data source.
-You can then use that file with other
-Airflow operators to move the data into another data source
+:param conn_id: the name of the connection that has the parameters we
need to connect to Salesforce.
+The connection should be type `http` and include a
user's security token
+in the `Extras` field.
+:type conn_id: str
-:param conn_id: the name of the connection that has the parameters
-we need to connect to Salesforce.
-The connection should be type `http` and include a
-user's security token in the `Extras` field.
.. note::
For the HTTP connection type, you can include a
JSON structure in the `Extras` field.
We need a user's security token to connect to Salesforce.
So we define it in the `Extras` field as:
`{"security_token":"YOUR_SECURITY_TOKEN"}`
+
"""
+super(SalesforceHook, self).__init__(conn_id)
self.conn_id = conn_id
-self._args = args
-self._kwargs = kwargs
-
-# get the connection parameters
-self.connection = self.get_connection(conn_id)
-self.extras = self.connection.extra_dejson
+self.salesforce = None
def sign_in(self):
Review comment:
```suggestion
def get_conn(self):
```
It is recommended that the `get_conn` method return the API library. The
instance should be cached.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #4829: [AIRFLOW-3993] Add tests for salesforce hook
mik-laj commented on a change in pull request #4829: [AIRFLOW-3993] Add tests for salesforce hook URL: https://github.com/apache/airflow/pull/4829#discussion_r261888743 ## File path: airflow/contrib/hooks/salesforce_hook.py ## @@ -18,127 +18,108 @@ # under the License. # """ -This module contains a Salesforce Hook -which allows you to connect to your Salesforce instance, -retrieve data from it, and write that data to a file -for other uses. +This module contains a Salesforce Hook which allows you to connect to your Salesforce instance, +retrieve data from it, and write that data to a file for other uses. -NOTE: this hook also relies on the simple_salesforce package: -https://github.com/simple-salesforce/simple-salesforce +NOTE: this hook also relies on the simple_salesforce package: Review comment: ```suggestion .. note:: this hook also relies on the simple_salesforce package: ``` Reference: https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#directive-note This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (AIRFLOW-3822) Airflow on Kubernetes, KubernetesPodOperator doesn't stop a task after timeout
[ https://issues.apache.org/jira/browse/AIRFLOW-3822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Mateus Pires closed AIRFLOW-3822. Resolution: Fixed was fixed by https://github.com/apache/airflow/pull/4663 > Airflow on Kubernetes, KubernetesPodOperator doesn't stop a task after timeout > -- > > Key: AIRFLOW-3822 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3822 > Project: Apache Airflow > Issue Type: Bug >Reporter: Daniel Mateus Pires >Assignee: Daniel Mateus Pires >Priority: Major > > Airflow with KubernetesExecutor starts a "watcher pod" which controls the > lifecycle of the actual Operator instance, when a timeout occurs it happens > on the watcher pod which dies without killing the Operator instance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3822) Airflow on Kubernetes, KubernetesPodOperator doesn't stop a task after timeout
[ https://issues.apache.org/jira/browse/AIRFLOW-3822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782859#comment-16782859 ] ASF GitHub Bot commented on AIRFLOW-3822: - dmateusp commented on pull request #4663: [AIRFLOW-3822] Delete KubernetesPodOperator pod on kill URL: https://github.com/apache/airflow/pull/4663 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Airflow on Kubernetes, KubernetesPodOperator doesn't stop a task after timeout > -- > > Key: AIRFLOW-3822 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3822 > Project: Apache Airflow > Issue Type: Bug >Reporter: Daniel Mateus Pires >Assignee: Daniel Mateus Pires >Priority: Major > > Airflow with KubernetesExecutor starts a "watcher pod" which controls the > lifecycle of the actual Operator instance, when a timeout occurs it happens > on the watcher pod which dies without killing the Operator instance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] dmateusp commented on issue #4663: [AIRFLOW-3822] Delete KubernetesPodOperator pod on kill
dmateusp commented on issue #4663: [AIRFLOW-3822] Delete KubernetesPodOperator pod on kill URL: https://github.com/apache/airflow/pull/4663#issuecomment-469062183 This was actually solved by https://github.com/apache/airflow/pull/4218 ! I only realised once I rebased, I originally noticed the bug on my production cluster so it did not have the change This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dmateusp closed pull request #4663: [AIRFLOW-3822] Delete KubernetesPodOperator pod on kill
dmateusp closed pull request #4663: [AIRFLOW-3822] Delete KubernetesPodOperator pod on kill URL: https://github.com/apache/airflow/pull/4663 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #4792: [AIRFLOW-3659] Create Google Cloud Transfer Service Operators
potiuk commented on a change in pull request #4792: [AIRFLOW-3659] Create Google Cloud Transfer Service Operators URL: https://github.com/apache/airflow/pull/4792#discussion_r261885856 ## File path: airflow/contrib/hooks/gcp_api_base_hook.py ## @@ -188,6 +190,21 @@ def inner_wrapper(self, *args, **kwargs): fallback_to_default_project_id = staticmethod(fallback_to_default_project_id) +@staticmethod +def catch_http_exception(func): Review comment: This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3993) Add tests for SalesforceHook
[
https://issues.apache.org/jira/browse/AIRFLOW-3993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782853#comment-16782853
]
ASF GitHub Bot commented on AIRFLOW-3993:
-
feluelle commented on pull request #4829: [AIRFLOW-3993] Add tests for
salesforce hook
URL: https://github.com/apache/airflow/pull/4829
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3993
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement
Proposal([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
This PR mainly adds tests, but also..
- refactors code
- updates docs
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add tests for SalesforceHook
>
>
> Key: AIRFLOW-3993
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3993
> Project: Apache Airflow
> Issue Type: Test
>Reporter: Felix Uellendall
>Assignee: Felix Uellendall
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] feluelle opened a new pull request #4829: [AIRFLOW-3993] Add tests for salesforce hook
feluelle opened a new pull request #4829: [AIRFLOW-3993] Add tests for
salesforce hook
URL: https://github.com/apache/airflow/pull/4829
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3993
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement
Proposal([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
This PR mainly adds tests, but also..
- refactors code
- updates docs
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #4792: [AIRFLOW-3659] Create Google Cloud Transfer Service Operators
mik-laj commented on a change in pull request #4792: [AIRFLOW-3659] Create Google Cloud Transfer Service Operators URL: https://github.com/apache/airflow/pull/4792#discussion_r261454998 ## File path: airflow/contrib/example_dags/example_gcp_transfer.py ## @@ -0,0 +1,254 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +""" +Example Airflow DAG that demonstrates interactions with Google Cloud Transfer. + +This DAG relies on the following OS environment variables + +* GCP_PROJECT_ID - Google Cloud Project to use for the Google Cloud Transfer Service. +* GCP_DESCRIPTION - Description of transfer job +* GCP_TRANSFER_SOURCE_AWS_BUCKET - Amazon Web Services Storage bucket from which files are copied. +* GCP_TRANSFER_FIRST_TARGET_BUCKET - Google Cloud Storage bucket to which files are copied from AWS. + It is also a source bucket in next step +* GCP_TRANSFER_SECOND_TARGET_BUCKET - Google Cloud Storage bucket bucket to which files are copied +* WAIT_FOR_OPERATION_POKE_INTERVAL - interval of what to check the status of the operation Review comment: It's a parameter for `BaseSensorOperator`. We have no special recommendations for values. We have added it only to be able to set a lower value, which will allow the operator to perform properly with a smaller set of data. I will expand the description of this variable. Reference: https://incubator-airflow.readthedocs.io/en/airflow-1075/code.html#airflow.operators.sensors.BaseSensorOperator This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #4792: [AIRFLOW-3659] Create Google Cloud Transfer Service Operators
mik-laj commented on a change in pull request #4792: [AIRFLOW-3659] Create Google Cloud Transfer Service Operators URL: https://github.com/apache/airflow/pull/4792#discussion_r261883966 ## File path: airflow/contrib/hooks/gcp_api_base_hook.py ## @@ -188,6 +190,21 @@ def inner_wrapper(self, *args, **kwargs): fallback_to_default_project_id = staticmethod(fallback_to_default_project_id) +@staticmethod +def catch_http_exception(func): Review comment: I found one more small difference. If you throw an AirflowException exception and the task has been marked externally as success/failed, executing is considered to be successfully completed - no log entry, etc. Reference: https://github.com/apache/airflow/blob/0982715a61653d21ac5ff9c1cc26e456ff713607/airflow/models/__init__.py#L1415-L1420 https://github.com/apache/airflow/commit/284dbdb60ab1fec027dea4871e3013a4727f6041 https://github.com/apache/airflow/commit/b2e1753f5b74ad1b6e0889f7b784ce69623c95ce This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #4792: [AIRFLOW-3659] Create Google Cloud Transfer Service Operators
mik-laj commented on a change in pull request #4792: [AIRFLOW-3659] Create Google Cloud Transfer Service Operators URL: https://github.com/apache/airflow/pull/4792#discussion_r261883966 ## File path: airflow/contrib/hooks/gcp_api_base_hook.py ## @@ -188,6 +190,21 @@ def inner_wrapper(self, *args, **kwargs): fallback_to_default_project_id = staticmethod(fallback_to_default_project_id) +@staticmethod +def catch_http_exception(func): Review comment: I found one more small difference. If you throw an AirflowException exception and the task has been marked externally as success/failed, executing is considered to be successfully completed - no log entry, etc. Reference: https://github.com/apache/airflow/blob/0982715a61653d21ac5ff9c1cc26e456ff713607/airflow/models/__init__.py#L1415-L1420 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #4828: [AIRFLOW-4002] Optionally open debugger on errors in airflow test.
codecov-io commented on issue #4828: [AIRFLOW-4002] Optionally open debugger on errors in airflow test. URL: https://github.com/apache/airflow/pull/4828#issuecomment-469055338 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4828?src=pr=h1) Report > Merging [#4828](https://codecov.io/gh/apache/airflow/pull/4828?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/45d24e79eab98589b1b0509e920811cbf778048b?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `44.44%`. [](https://codecov.io/gh/apache/airflow/pull/4828?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4828 +/- ## == - Coverage 74.49% 74.49% -0.01% == Files 450 450 Lines 2899629002 +6 == + Hits2160221604 +2 - Misses 7394 7398 +4 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4828?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/4828/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `66.9% <44.44%> (-0.32%)` | :arrow_down: | | [airflow/contrib/operators/ssh\_operator.py](https://codecov.io/gh/apache/airflow/pull/4828/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zc2hfb3BlcmF0b3IucHk=) | `83.54% <0%> (+1.26%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4828?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4828?src=pr=footer). Last update [45d24e7...3f788fb](https://codecov.io/gh/apache/airflow/pull/4828?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #4828: [AIRFLOW-4002] Optionally open debugger on errors in airflow test.
codecov-io commented on issue #4828: [AIRFLOW-4002] Optionally open debugger on errors in airflow test. URL: https://github.com/apache/airflow/pull/4828#issuecomment-469055337 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4828?src=pr=h1) Report > Merging [#4828](https://codecov.io/gh/apache/airflow/pull/4828?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/45d24e79eab98589b1b0509e920811cbf778048b?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `44.44%`. [](https://codecov.io/gh/apache/airflow/pull/4828?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4828 +/- ## == - Coverage 74.49% 74.49% -0.01% == Files 450 450 Lines 2899629002 +6 == + Hits2160221604 +2 - Misses 7394 7398 +4 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4828?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/4828/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `66.9% <44.44%> (-0.32%)` | :arrow_down: | | [airflow/contrib/operators/ssh\_operator.py](https://codecov.io/gh/apache/airflow/pull/4828/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zc2hfb3BlcmF0b3IucHk=) | `83.54% <0%> (+1.26%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4828?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4828?src=pr=footer). Last update [45d24e7...3f788fb](https://codecov.io/gh/apache/airflow/pull/4828?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-4002) Optionally start debugger on uncaught exceptions in `airflow test`
Josh Carp created AIRFLOW-4002: -- Summary: Optionally start debugger on uncaught exceptions in `airflow test` Key: AIRFLOW-4002 URL: https://issues.apache.org/jira/browse/AIRFLOW-4002 Project: Apache Airflow Issue Type: Improvement Reporter: Josh Carp When testing a new task with `airflow test`, it would be useful to optionally open a `pdb` session on uncaught exceptions in order to debug the issue. I propose adding a `–post_mortem` flag to `airflow test` to enable this behavior. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4002) Optionally start debugger on uncaught exceptions in `airflow test`
[
https://issues.apache.org/jira/browse/AIRFLOW-4002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782819#comment-16782819
]
ASF GitHub Bot commented on AIRFLOW-4002:
-
jmcarp commented on pull request #4828: [AIRFLOW-4002] Optionally open debugger
on errors in airflow test.
URL: https://github.com/apache/airflow/pull/4828
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-4002
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement
Proposal([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Optionally start debugger on uncaught exceptions in `airflow test`
> --
>
> Key: AIRFLOW-4002
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4002
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: Josh Carp
>Priority: Trivial
>
> When testing a new task with `airflow test`, it would be useful to optionally
> open a `pdb` session on uncaught exceptions in order to debug the issue. I
> propose adding a `–post_mortem` flag to `airflow test` to enable this
> behavior.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] jmcarp opened a new pull request #4828: [AIRFLOW-4002] Optionally open debugger on errors in airflow test.
jmcarp opened a new pull request #4828: [AIRFLOW-4002] Optionally open debugger
on errors in airflow test.
URL: https://github.com/apache/airflow/pull/4828
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-4002
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement
Proposal([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #4827: [AIRFLOW-XXX] Add contribute guide, updating to docs
codecov-io edited a comment on issue #4827: [AIRFLOW-XXX] Add contribute guide, updating to docs URL: https://github.com/apache/airflow/pull/4827#issuecomment-469043251 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4827?src=pr=h1) Report > Merging [#4827](https://codecov.io/gh/apache/airflow/pull/4827?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/45d24e79eab98589b1b0509e920811cbf778048b?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/4827?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4827 +/- ## === Coverage 74.49% 74.49% === Files 450 450 Lines 2899628996 === Hits2160221602 Misses 7394 7394 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4827?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4827?src=pr=footer). Last update [45d24e7...9e0f399](https://codecov.io/gh/apache/airflow/pull/4827?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] amichai07 commented on a change in pull request #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run
amichai07 commented on a change in pull request #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run URL: https://github.com/apache/airflow/pull/4751#discussion_r261877321 ## File path: airflow/jobs.py ## @@ -930,7 +932,20 @@ def _process_task_instances(self, dag, queue, session=None): run.dag = dag # todo: preferably the integrity check happens at dag collection time run.verify_integrity(session=session) -run.update_state(session=session) +finished_tasks = ( +session +.query(TI.task_id, TI.state + ) +.filter( +TI.dag_id == run.dag_id, +TI.execution_date == run.execution_date, Review comment: Yes, I will fix it to be `run.get_task_instances(state=State.finished(), session=session)` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] amichai07 commented on a change in pull request #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run
amichai07 commented on a change in pull request #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run URL: https://github.com/apache/airflow/pull/4751#discussion_r261877204 ## File path: airflow/jobs.py ## @@ -930,7 +932,20 @@ def _process_task_instances(self, dag, queue, session=None): run.dag = dag # todo: preferably the integrity check happens at dag collection time run.verify_integrity(session=session) -run.update_state(session=session) +finished_tasks = ( +session +.query(TI.task_id, TI.state + ) +.filter( +TI.dag_id == run.dag_id, +TI.execution_date == run.execution_date, +TI.state.in_([ +State.SUCCESS, State.FAILED, +State.UPSTREAM_FAILED, State.SKIPPED]) Review comment: It seems that it should include also UPSTREAM_FAILED, the behaviour dealing with finished tasks is quiet similar in most cases, I will run the tests and we will see :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] amichai07 commented on a change in pull request #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run
amichai07 commented on a change in pull request #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run URL: https://github.com/apache/airflow/pull/4751#discussion_r261877155 ## File path: airflow/ti_deps/deps/trigger_rule_dep.py ## @@ -49,33 +49,49 @@ def _get_dep_statuses(self, ti, session, dep_context): yield self._passing_status(reason="The task had a dummy trigger rule set.") return -# TODO(unknown): this query becomes quite expensive with dags that have many -# tasks. It should be refactored to let the task report to the dag run and get the -# aggregates from there. -qry = ( -session -.query( -func.coalesce(func.sum( -case([(TI.state == State.SUCCESS, 1)], else_=0)), 0), -func.coalesce(func.sum( -case([(TI.state == State.SKIPPED, 1)], else_=0)), 0), -func.coalesce(func.sum( -case([(TI.state == State.FAILED, 1)], else_=0)), 0), -func.coalesce(func.sum( -case([(TI.state == State.UPSTREAM_FAILED, 1)], else_=0)), 0), -func.count(TI.task_id), +successes, skipped, failed, upstream_failed, done = 0, 0, 0, 0, 0 +if dep_context.finished_tasks is None: +qry = ( +session +.query( +func.coalesce(func.sum( +case([(TI.state == State.SUCCESS, 1)], else_=0)), 0), +func.coalesce(func.sum( +case([(TI.state == State.SKIPPED, 1)], else_=0)), 0), +func.coalesce(func.sum( +case([(TI.state == State.FAILED, 1)], else_=0)), 0), +func.coalesce(func.sum( +case([(TI.state == State.UPSTREAM_FAILED, 1)], else_=0)), 0), +func.count(TI.task_id), Review comment: Yes, almost the same here, I will add UPSTREAM_FAILED and we will see what happen This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] amichai07 commented on a change in pull request #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run
amichai07 commented on a change in pull request #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run URL: https://github.com/apache/airflow/pull/4751#discussion_r261877174 ## File path: airflow/models/__init__.py ## @@ -4872,11 +4872,12 @@ def get_previous_scheduled_dagrun(self, session=None): ).first() @provide_session -def update_state(self, session=None): +def update_state(self, session=None, finished_tasks=None): """ Determines the overall state of the DagRun based on the state of its TaskInstances. +:param finished_tasks: The finished tasks collected ordered by dagrun as a column (task_name, state) Review comment: It is necessary for the dependency check of all unfinished tasks(4915), and we have it already so why not. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #4827: [AIRFLOW-XXX] Add contribute guide, updating to docs
codecov-io commented on issue #4827: [AIRFLOW-XXX] Add contribute guide, updating to docs URL: https://github.com/apache/airflow/pull/4827#issuecomment-469043251 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4827?src=pr=h1) Report > Merging [#4827](https://codecov.io/gh/apache/airflow/pull/4827?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/45d24e79eab98589b1b0509e920811cbf778048b?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/4827?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4827 +/- ## == - Coverage 74.49% 74.49% -0.01% == Files 450 450 Lines 2899628996 == - Hits2160221601 -1 - Misses 7394 7395 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4827?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/4827/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=) | `92.59% <0%> (-0.06%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4827?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4827?src=pr=footer). Last update [45d24e7...9e0f399](https://codecov.io/gh/apache/airflow/pull/4827?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #4827: [AIRFLOW-XXX] Add contribute guide, updating to docs
mik-laj commented on issue #4827: [AIRFLOW-XXX] Add contribute guide, updating to docs URL: https://github.com/apache/airflow/pull/4827#issuecomment-469040808 If someone wants to see changes, the following commands will be useful: ```bash colordiff <(git show apache/master:CHANGELOG.txt) <(git show HEAD:CHANGELOG.rst) colordiff <(git show apache/master:CONTRIBUTING.md) <(git show HEAD:CONTRIBUTING.rst) colordiff <(git show apache/master:UPDATING.md) <(git show HEAD:UPDATING.rst) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #4827: [AIRFLOW-XXX] Add contribute guide, updating to docs
mik-laj commented on a change in pull request #4827: [AIRFLOW-XXX] Add contribute guide, updating to docs URL: https://github.com/apache/airflow/pull/4827#discussion_r261875135 ## File path: CHANGELOG.rst ## @@ -1,3 +1,23 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +..http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. + +Changeslog Review comment: Yes. I have already noticed it and updated it This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #4827: [AIRFLOW-XXX] Add contribute guide, updating to docs
feluelle commented on a change in pull request #4827: [AIRFLOW-XXX] Add contribute guide, updating to docs URL: https://github.com/apache/airflow/pull/4827#discussion_r261873698 ## File path: CHANGELOG.rst ## @@ -1,3 +1,23 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +..http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. + +Changeslog Review comment: typo? :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj opened a new pull request #4827: [AIRFLOW-XXX] Add contribute guide, updating to docs
mik-laj opened a new pull request #4827: [AIRFLOW-XXX] Add contribute guide,
updating to docs
URL: https://github.com/apache/airflow/pull/4827
Make sure you have checked _all_ steps below.
### Jira
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
### Description
Preview: http://wiggly-ring.surge.sh/contributing.html
This change adds two new pages to the documentation.
* Developer’s Guide
* Updating
Reasons for making changes:
* creates space for another document that was discussed on the dev list
https://lists.apache.org/thread.html/af7637c61acbb877dc68f337b633b2c34a43f91a2ebafc61208daf0c@%3Cdev.airflow.apache.org%3E
* creates one source of knowledge. The user has all the materials available
in one place.
* historical versions of documents become available in the compiled version
- https://readthedocs.org/projects/airflow/
* unifies the documentation format in repo. Previously some files were in
txt, others in markdown.
* linking between documents was difficult. It is now possible to refer to
this document from the documentation. See:
http://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#directive-versionchanged
Other changes:
* Some files lacked a license header.
* Rename CHANGELOG.txt to CHANGELOG.rst. The content of the file pointed to
a different format than the file name.
CC: @potiuk @fenglu-g
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
No applicable
### Code Quality
No applicable
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] XD-DENG commented on issue #4637: [AIRFLOW-3793] Decommission configuration items for Flask-Admin web UI & related codes
XD-DENG commented on issue #4637: [AIRFLOW-3793] Decommission configuration items for Flask-Admin web UI & related codes URL: https://github.com/apache/airflow/pull/4637#issuecomment-469027034 Hi @ashb , @feng-tao , @Fokko , a gentle ping on this. Have rebased earlier today to ensure no conflict. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] PaulW edited a comment on issue #4636: [AIRFLOW-3737] Kubernetes executor cannot handle long dag/task names
PaulW edited a comment on issue #4636: [AIRFLOW-3737] Kubernetes executor
cannot handle long dag/task names
URL: https://github.com/apache/airflow/pull/4636#issuecomment-469023138
So I hit the same issue as the last time we tried to trim/hash labels & not
store the raw dag_id & task_id as an annotation, in that the following
exception is raised upon the pod deletion, when the `_change_state()` function
is called at the end:
```[2019-03-01 18:02:04,081] {{kubernetes_executor.py:679}} INFO - Changing
state of (('kubernetesexecutorworkerlaunchtest', 'workerpodsubdagtest',
datetime.datetime(2019, 3, 1, 17, 45, tzinfo=tzlocal()), 1), None,
'kubernetesexecutorworkerlaunchtestworkerpodsubdagtest-c77976ffb1e54b6788612ac949fb31e2',
'102168655') to None
[2019-03-01 18:02:04,093] {{kubernetes_executor.py:692}} INFO - Deleted pod:
('kubernetesexecutorworkerlaunchtest', 'workerpodsubdagtest',
datetime.datetime(2019, 3, 1, 17, 45, tzinfo=tzlocal()), 1)
[2019-03-01 18:02:04,093] {{kubernetes_executor.py:695}} DEBUG - Could not
find key: ('kubernetesexecutorworkerlaunchtest', 'workerpodsubdagtest',
datetime.datetime(2019, 3, 1, 17, 45, tzinfo=tzlocal()), 1)
[2019-03-01 18:02:04,094] {{kubernetes_executor.py:293}} INFO - Event:
kubernetesexecutorworkerlaunchtestworkerpodsubdagtest-c77976ffb1e54b6788612ac949fb31e2
had an event of type MODIFIED
[2019-03-01 18:02:04,094] {{kubernetes_executor.py:330}} INFO - Event:
kubernetesexecutorworkerlaunchtestworkerpodsubdagtest-c77976ffb1e54b6788612ac949fb31e2
Succeeded
[2019-03-01 18:02:04,097] {{kubernetes_executor.py:293}} INFO - Event:
kubernetesexecutorworkerlaunchtestworkerpodsubdagtest-c77976ffb1e54b6788612ac949fb31e2
had an event of type DELETED
[2019-03-01 18:02:04,097] {{kubernetes_executor.py:330}} INFO - Event:
kubernetesexecutorworkerlaunchtestworkerpodsubdagtest-c77976ffb1e54b6788612ac949fb31e2
Succeeded
[2019-03-01 18:02:04,185] {{dag_processing.py:587}} INFO - Terminating
manager process: 639
[2019-03-01 18:02:04,185] {{dag_processing.py:592}} INFO - Waiting up to 5s
for manager process to exit...
[2019-03-01 18:02:04,207] {{settings.py:201}} DEBUG - Disposing DB
connection pool (PID 4983)
[2019-03-01 18:02:04,256] {{settings.py:201}} DEBUG - Disposing DB
connection pool (PID 5010)
[2019-03-01 18:02:04,320] {{jobs.py:1513}} INFO - Exited execute loop
[2019-03-01 18:02:04,438] {{cli_action_loggers.py:81}} DEBUG - Calling
callbacks: []
[2019-03-01 18:02:04,445] {{settings.py:201}} DEBUG - Disposing DB
connection pool (PID 1)
Traceback (most recent call last):
File "/usr/bin/airflow", line 32, in
args.func(args)
File "/usr/lib/python3.6/site-packages/airflow/utils/cli.py", line 74, in
wrapper
return f(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/airflow/bin/cli.py", line 991, in
scheduler
job.run()
File "/usr/lib/python3.6/site-packages/airflow/jobs.py", line 202, in run
self._execute()
File "/usr/lib/python3.6/site-packages/airflow/jobs.py", line 1510, in
_execute
self._execute_helper()
File "/usr/lib/python3.6/site-packages/airflow/jobs.py", line 1596, in
_execute_helper
self.executor.heartbeat()
File
"/usr/lib/python3.6/site-packages/airflow/executors/base_executor.py", line
150, in heartbeat
self.sync()
File
"/usr/lib/python3.6/site-packages/airflow/contrib/executors/kubernetes_executor.py",
line 680, in sync
self._change_state(key, state, pod_id)
File
"/usr/lib/python3.6/site-packages/airflow/contrib/executors/kubernetes_executor.py",
line 703, in _change_state
execution_date=ex_time
File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line
3046, in one
raise orm_exc.NoResultFound("No row was found for one()")
sqlalchemy.orm.exc.NoResultFound: No row was found for one()
```
The process seems to be:
`KubernetesExecutor.sync()` -> `AirflowKubernetesScheduler.sync()` ->
`AirflowKubernetesScheduler.process_watcher_task()`
`process_watcher_task()` then calls `_labels_to_key()` which extracts the
labels from the pod & stores them to `key`. These labels are the
truncated/modified ones.
`KubernetesExecutor.sync()` resumes, and calls
`KubernetesExecutor._change_state()` and passes `key` through to it.
`_change_state` then deletes the pod which ran the dag/task, and using the
`dag_id` and `task_id` values returned previously from `_labels_to_key()` and
stored within `key`, then tries to query the db.
``` (dag_id, task_id, ex_time, try_number) = key
with create_session() as session:
item = session.query(TaskInstance).filter_by(
dag_id=dag_id,
task_id=task_id,
execution_date=ex_time
).one()
if state:
item.state = state
[GitHub] [airflow] PaulW edited a comment on issue #4636: [AIRFLOW-3737] Kubernetes executor cannot handle long dag/task names
PaulW edited a comment on issue #4636: [AIRFLOW-3737] Kubernetes executor
cannot handle long dag/task names
URL: https://github.com/apache/airflow/pull/4636#issuecomment-469023138
So I hit the same issue as the last time we tried to trim/hash labels & not
store the raw dag_id & task_id as an annotation, in that the following
exception is raised upon the pod deletion, when the `_change_state()` function
is called at the end:
```[2019-03-01 18:02:04,081] {{kubernetes_executor.py:679}} INFO - Changing
state of (('kubernetesexecutorworkerlaunchtest', 'workerpodsubdagtest',
datetime.datetime(2019, 3, 1, 17, 45, tzinfo=tzlocal()), 1), None,
'kubernetesexecutorworkerlaunchtestworkerpodsubdagtest-c77976ffb1e54b6788612ac949fb31e2',
'102168655') to None
[2019-03-01 18:02:04,093] {{kubernetes_executor.py:692}} INFO - Deleted pod:
('kubernetesexecutorworkerlaunchtest', 'workerpodsubdagtest',
datetime.datetime(2019, 3, 1, 17, 45, tzinfo=tzlocal()), 1)
[2019-03-01 18:02:04,093] {{kubernetes_executor.py:695}} DEBUG - Could not
find key: ('kubernetesexecutorworkerlaunchtest', 'workerpodsubdagtest',
datetime.datetime(2019, 3, 1, 17, 45, tzinfo=tzlocal()), 1)
[2019-03-01 18:02:04,094] {{kubernetes_executor.py:293}} INFO - Event:
kubernetesexecutorworkerlaunchtestworkerpodsubdagtest-c77976ffb1e54b6788612ac949fb31e2
had an event of type MODIFIED
[2019-03-01 18:02:04,094] {{kubernetes_executor.py:330}} INFO - Event:
kubernetesexecutorworkerlaunchtestworkerpodsubdagtest-c77976ffb1e54b6788612ac949fb31e2
Succeeded
[2019-03-01 18:02:04,097] {{kubernetes_executor.py:293}} INFO - Event:
kubernetesexecutorworkerlaunchtestworkerpodsubdagtest-c77976ffb1e54b6788612ac949fb31e2
had an event of type DELETED
[2019-03-01 18:02:04,097] {{kubernetes_executor.py:330}} INFO - Event:
kubernetesexecutorworkerlaunchtestworkerpodsubdagtest-c77976ffb1e54b6788612ac949fb31e2
Succeeded
[2019-03-01 18:02:04,185] {{dag_processing.py:587}} INFO - Terminating
manager process: 639
[2019-03-01 18:02:04,185] {{dag_processing.py:592}} INFO - Waiting up to 5s
for manager process to exit...
[2019-03-01 18:02:04,207] {{settings.py:201}} DEBUG - Disposing DB
connection pool (PID 4983)
[2019-03-01 18:02:04,256] {{settings.py:201}} DEBUG - Disposing DB
connection pool (PID 5010)
[2019-03-01 18:02:04,320] {{jobs.py:1513}} INFO - Exited execute loop
[2019-03-01 18:02:04,438] {{cli_action_loggers.py:81}} DEBUG - Calling
callbacks: []
[2019-03-01 18:02:04,445] {{settings.py:201}} DEBUG - Disposing DB
connection pool (PID 1)
Traceback (most recent call last):
File "/usr/bin/airflow", line 32, in
args.func(args)
File "/usr/lib/python3.6/site-packages/airflow/utils/cli.py", line 74, in
wrapper
return f(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/airflow/bin/cli.py", line 991, in
scheduler
job.run()
File "/usr/lib/python3.6/site-packages/airflow/jobs.py", line 202, in run
self._execute()
File "/usr/lib/python3.6/site-packages/airflow/jobs.py", line 1510, in
_execute
self._execute_helper()
File "/usr/lib/python3.6/site-packages/airflow/jobs.py", line 1596, in
_execute_helper
self.executor.heartbeat()
File
"/usr/lib/python3.6/site-packages/airflow/executors/base_executor.py", line
150, in heartbeat
self.sync()
File
"/usr/lib/python3.6/site-packages/airflow/contrib/executors/kubernetes_executor.py",
line 680, in sync
self._change_state(key, state, pod_id)
File
"/usr/lib/python3.6/site-packages/airflow/contrib/executors/kubernetes_executor.py",
line 703, in _change_state
execution_date=ex_time
File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line
3046, in one
raise orm_exc.NoResultFound("No row was found for one()")
sqlalchemy.orm.exc.NoResultFound: No row was found for one()
```
The process seems to be:
`KubernetesExecutor.sync()` -> `AirflowKubernetesScheduler.sync()` ->
`AirflowKubernetesScheduler.process_watcher_task()`
`process_watcher_task()` then calls `_labels_to_key()` which extracts the
labels from the pod & stores them to `key`. These labels are the
truncated/modified ones.
`KubernetesExecutor.sync()` resumes, and calls
`KubernetesExecutor._change_state()` and passes `key` through to it.
`_change_state` then deletes the pod which ran the dag/task, and using the
`dag_id` and `task_id` values returned previously from `_labels_to_key()` and
stored within `key`, then tries to query the db.
``` (dag_id, task_id, ex_time, try_number) = key
with create_session() as session:
item = session.query(TaskInstance).filter_by(
dag_id=dag_id,
task_id=task_id,
execution_date=ex_time
).one()
if state:
item.state = state
[GitHub] [airflow] PaulW commented on issue #4636: [AIRFLOW-3737] Kubernetes executor cannot handle long dag/task names
PaulW commented on issue #4636: [AIRFLOW-3737] Kubernetes executor cannot
handle long dag/task names
URL: https://github.com/apache/airflow/pull/4636#issuecomment-469023138
So I hit the same issue as the last time we tried to trim/hash labels & not
store the raw dag_id & task_id as an annotation, in that the following
exception is raised upon the pod deletion, when the `_change_state()` function
is called at the end:
```[2019-03-01 18:02:04,081] {{kubernetes_executor.py:679}} INFO - Changing
state of (('kubernetesexecutorworkerlaunchtest', 'workerpodsubdagtest',
datetime.datetime(2019, 3, 1, 17, 45, tzinfo=tzlocal()), 1), None,
'kubernetesexecutorworkerlaunchtestworkerpodsubdagtest-c77976ffb1e54b6788612ac949fb31e2',
'102168655') to None
[2019-03-01 18:02:04,093] {{kubernetes_executor.py:692}} INFO - Deleted pod:
('kubernetesexecutorworkerlaunchtest', 'workerpodsubdagtest',
datetime.datetime(2019, 3, 1, 17, 45, tzinfo=tzlocal()), 1)
[2019-03-01 18:02:04,093] {{kubernetes_executor.py:695}} DEBUG - Could not
find key: ('kubernetesexecutorworkerlaunchtest', 'workerpodsubdagtest',
datetime.datetime(2019, 3, 1, 17, 45, tzinfo=tzlocal()), 1)
[2019-03-01 18:02:04,094] {{kubernetes_executor.py:293}} INFO - Event:
kubernetesexecutorworkerlaunchtestworkerpodsubdagtest-c77976ffb1e54b6788612ac949fb31e2
had an event of type MODIFIED
[2019-03-01 18:02:04,094] {{kubernetes_executor.py:330}} INFO - Event:
kubernetesexecutorworkerlaunchtestworkerpodsubdagtest-c77976ffb1e54b6788612ac949fb31e2
Succeeded
[2019-03-01 18:02:04,097] {{kubernetes_executor.py:293}} INFO - Event:
kubernetesexecutorworkerlaunchtestworkerpodsubdagtest-c77976ffb1e54b6788612ac949fb31e2
had an event of type DELETED
[2019-03-01 18:02:04,097] {{kubernetes_executor.py:330}} INFO - Event:
kubernetesexecutorworkerlaunchtestworkerpodsubdagtest-c77976ffb1e54b6788612ac949fb31e2
Succeeded
[2019-03-01 18:02:04,185] {{dag_processing.py:587}} INFO - Terminating
manager process: 639
[2019-03-01 18:02:04,185] {{dag_processing.py:592}} INFO - Waiting up to 5s
for manager process to exit...
[2019-03-01 18:02:04,207] {{settings.py:201}} DEBUG - Disposing DB
connection pool (PID 4983)
[2019-03-01 18:02:04,256] {{settings.py:201}} DEBUG - Disposing DB
connection pool (PID 5010)
[2019-03-01 18:02:04,320] {{jobs.py:1513}} INFO - Exited execute loop
[2019-03-01 18:02:04,438] {{cli_action_loggers.py:81}} DEBUG - Calling
callbacks: []
[2019-03-01 18:02:04,445] {{settings.py:201}} DEBUG - Disposing DB
connection pool (PID 1)
Traceback (most recent call last):
File "/usr/bin/airflow", line 32, in
args.func(args)
File "/usr/lib/python3.6/site-packages/airflow/utils/cli.py", line 74, in
wrapper
return f(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/airflow/bin/cli.py", line 991, in
scheduler
job.run()
File "/usr/lib/python3.6/site-packages/airflow/jobs.py", line 202, in run
self._execute()
File "/usr/lib/python3.6/site-packages/airflow/jobs.py", line 1510, in
_execute
self._execute_helper()
File "/usr/lib/python3.6/site-packages/airflow/jobs.py", line 1596, in
_execute_helper
self.executor.heartbeat()
File
"/usr/lib/python3.6/site-packages/airflow/executors/base_executor.py", line
150, in heartbeat
self.sync()
File
"/usr/lib/python3.6/site-packages/airflow/contrib/executors/kubernetes_executor.py",
line 680, in sync
self._change_state(key, state, pod_id)
File
"/usr/lib/python3.6/site-packages/airflow/contrib/executors/kubernetes_executor.py",
line 703, in _change_state
execution_date=ex_time
File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line
3046, in one
raise orm_exc.NoResultFound("No row was found for one()")
sqlalchemy.orm.exc.NoResultFound: No row was found for one()```
The process seems to be:
`KubernetesExecutor.sync()` -> `AirflowKubernetesScheduler.sync()` ->
`AirflowKubernetesScheduler.process_watcher_task()`
`process_watcher_task()` then calls `_labels_to_key()` which extracts the
labels from the pod & stores them to `key`. These labels are the
truncated/modified ones.
`KubernetesExecutor.sync()` resumes, and calls
`KubernetesExecutor._change_state()` and passes `key` through to it.
`_change_state` then deletes the pod which ran the dag/task, and using the
`dag_id` and `task_id` values returned previously from `_labels_to_key()` and
stored within `key`, then tries to query the db.
``` (dag_id, task_id, ex_time, try_number) = key
with create_session() as session:
item = session.query(TaskInstance).filter_by(
dag_id=dag_id,
task_id=task_id,
execution_date=ex_time
).one()
if state:
item.state = state
[GitHub] [airflow] adamatrais commented on a change in pull request #4245: [AIRFLOW-3404] Add Amazon SES support
adamatrais commented on a change in pull request #4245: [AIRFLOW-3404] Add
Amazon SES support
URL: https://github.com/apache/airflow/pull/4245#discussion_r261865484
##
File path: airflow/contrib/utils/ses.py
##
@@ -0,0 +1,94 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import logging
+import os
+import boto3
+
+from email.mime.application import MIMEApplication
+from email.mime.multipart import MIMEMultipart
+from email.mime.text import MIMEText
+from email.utils import formatdate
+
+from airflow import configuration
+
+
+def send_email(to, subject, html_content, files=None, dryrun=False, cc=None,
+ bcc=None, mime_subtype='mixed', sandbox_mode=False, **kwargs):
+"""
+Send an email with html content using Amazon SES.
+
+To use this plugin:
+0. include ses subpackage as part of your Airflow installation, e.g.,
+pip install apache-airflow[ses]
+1. update [email] backend in airflow.cfg, i.e.,
+[email]
+email_backend = airflow.contrib.utils.ses.send_email
+2. configure SES specific setting in airflow.cfg, e.g.:
+[ses]
+aws_region = eu-west-1
Review comment:
It should be region as you're calling:
aws_region = configuration.get('ses', 'REGION')
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] RosterIn commented on issue #4802: [AIRFLOW-3978] - Add TIME, BINARY, VARBINARY to MySqlToGoogleCloudStorageOperator
RosterIn commented on issue #4802: [AIRFLOW-3978] - Add TIME, BINARY, VARBINARY to MySqlToGoogleCloudStorageOperator URL: https://github.com/apache/airflow/pull/4802#issuecomment-469015885 I tested my changed only for the `TIME ` field and it doesn't work either. ``` File "/NewOperators/mysql_to_gcs_mymodification.py", line 105, in execute files_to_upload = self._write_local_data_files(cursor) File "/NewOperators/mysql_to_gcs_mymodification.py", line 151, in _write_local_data_files s = json.dumps(row_dict) File "/usr/lib/python2.7/json/__init__.py", line 244, in dumps return _default_encoder.encode(obj) File "/usr/lib/python2.7/json/encoder.py", line 207, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode return _iterencode(o, 0) File "/usr/lib/python2.7/json/encoder.py", line 184, in default raise TypeError(repr(o) + " is not JSON serializable") TypeError: datetime.timedelta(0, 86385) is not JSON serializable ``` I'm closing this PR as I don't have time to refactor / investigate this. If any one has the time to fix this operator you are welcome to take this. Currently the operator doesn't work for column types : TIME, BINARY, VARBINARY and I assume that there are more types that aren't supported. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kristiany edited a comment on issue #4819: [AIRFLOW-3997] Add accessor for optional variables
kristiany edited a comment on issue #4819: [AIRFLOW-3997] Add accessor for optional variables URL: https://github.com/apache/airflow/pull/4819#issuecomment-469012571 @RosterIn Agree. `try_get` is a better name EDIT: Changed function name from `optional` to `try_get` and also updated the doc change in this PR, as well as the PR body. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kristiany commented on issue #4819: [AIRFLOW-3997] Add accessor for optional variables
kristiany commented on issue #4819: [AIRFLOW-3997] Add accessor for optional variables URL: https://github.com/apache/airflow/pull/4819#issuecomment-469012571 @RosterIn Agree. `try_get` is a better name This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] RosterIn commented on issue #4819: [AIRFLOW-3997] Add accessor for optional variables
RosterIn commented on issue #4819: [AIRFLOW-3997] Add accessor for optional variables URL: https://github.com/apache/airflow/pull/4819#issuecomment-469004096 I think `get_soft()` or `try_get()` might be a better name than `optional()` The try_get() idea is taken from the `CAST `function of `SQL`: https://database.guide/cast-vs-try_cast-in-sql-server-whats-the-difference/ which has very similar functionality to what this PR is about. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] RosterIn commented on issue #4802: [AIRFLOW-3978] - Add TIME, BINARY, VARBINARY to MySqlToGoogleCloudStorageOperator
RosterIn commented on issue #4802: [AIRFLOW-3978] - Add TIME, BINARY, VARBINARY to MySqlToGoogleCloudStorageOperator URL: https://github.com/apache/airflow/pull/4802#issuecomment-469000730 @XD-DENG @feng-tao If you can please let me know what is needed to add support for : TIME, BINARY, VARBINARY types. I don't have time to refactor this whole operator so I'm looking for a working patch here... @methane This operator gets a SQL query and transfer data from MySQL to Google Storage (Json file). so according to your answer I might be able to avoid the issue completly if the query will convert the problematic types to simple types as this is what the lib does anyway (?) so if my table is: ``` CREATE TABLE `my_table` ( `id` bigint(10) unsigned NOT NULL AUTO_INCREMENT, `config_id` binary(8) NOT NULL, `my_ip` varbinary(16) NOT NULL, PRIMARY KEY (`id`) ) ``` and my query will be: ``` select id, CAST(config_id AS varchar()), CAST(my_ip AS varchar()) from my_table ``` It should work doesn't it? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
