[GitHub] tedmiston commented on issue #3656: [AIRFLOW-2803] Fix all ESLint issues

[GitBox]

tedmiston commented on issue #3656: [AIRFLOW-2803] Fix all ESLint issues
URL: 
https://github.com/apache/incubator-airflow/pull/3656#issuecomment-415647080
 
 
   @verdan Please see the revised diff / new squashed commit, and review when 
you have a chance.
   
   There were some opinionated decisions to make here as far as which warnings 
/ errors to fix via refactoring vs which were more complicated and worthy of 
disabling via a linter comment... there's a decent amount of tech debt in the 
front end JS.  I started off doing more refactoring, but eventually saw the 
size of this diff growing and growing and switched to only refactoring what was 
absolutely required for right now.
   
   Some notes:
   
   - This diff overall now resolves something like ~1500–2500 linter errors in 
total.  In some places I have used disable linter calls if refactoring the code 
to pass felt cumbersome or like a bit much.  I think these few long tail errors 
could be cleaned up with refactoring post-PR.
   - I have an example of refactoring building the long query string with 
template strings here - 
https://github.com/apache/incubator-airflow/pull/3656/commits/b65388a7b772fa7b40140d77ef25c45ad1c475c9.
  Since the PR grew pretty huge already, I'm thinking of maybe pulling it out 
of this PR and refactoring the others requests separately if this style pattern 
feels like an improvement.  Rewriting this also made me realize we have a 
dormant bug where some query params aren't being encoded which is extra 
motivation to migrate this part into a separate PR and keep scope here focused 
on linting.
   - The original 
[eslint-plugin-jinja](https://github.com/alexkuz/eslint-plugin-jinja) is not 
maintained and 
[doesn't](https://github.com/alexkuz/eslint-plugin-jinja/issues/3) 
[work](https://github.com/alexkuz/eslint-plugin-jinja/issues/4) with current 
version of ESLint, however, the 
[jupyterlab/eslint-plugin-jinja](https://github.com/jupyterlab/eslint-plugin-jinja)
 fork does, so I went with that one.
   - Apparently there's an issue with `eslint --fix` when using the 
eslint-plugin-jinja plugin / some plugins where it just doesn't work but also 
doesn't given an error.  "Unfortunately, auto-fix is not supported on plugins 
with processors (i.e., plugins which transform files to be ready for linting in 
JS)." (https://github.com/eslint/eslint/issues/7456).
   - My workaround for this was to temporarily comment out the plugin to run 
`--fix` which will fix some things but also break some things (which the plugin 
would prevent) inside Jinja, but since it can auto fix many errors, it was 
still helpful.  Then when done, uncomment the plugin, fix any auto errors by 
hand, and re-run lint.  [I'm considering adding this to our docs since it was 
unexpected.]
   - I have 2 FIXMEs outstanding - 1 in graph.html, 1 in tree.html where the 
linter is convinced the lines should not end with a semicolon (check the 
error).  I haven't figured out how to solve these yet if you have  any advice.
   - Aligned all outermost vertical indentation to match level of script tags.
   - Ensured consistent indentation for script tags nested inside Jinja blocks.
   - Ensured newlines surrounding every function definition.
   - Added blank newline after Apache license text where missing for 
consistency.
   - Added missing closing  tag in circles.html.
   - Needs tested from the GUI.
   - Also just realized I forgot to revert the use of ES6 syntax for anonymous 
functions, so I'll have to fix that.  Otherwise, I'm hoping this is very close 
to done.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] gerardo commented on issue #3797: [AIRFLOW-2952] Splits CI into k8s + docker-compose

[GitBox]

gerardo commented on issue #3797: [AIRFLOW-2952] Splits CI into k8s + 
docker-compose
URL: 
https://github.com/apache/incubator-airflow/pull/3797#issuecomment-415643427
 
 
   @dimberman given the Kubernetes ci scripts runs outside docker, [this 
line](https://github.com/apache/incubator-airflow/pull/3797/files#diff-0134d0a946dd857a50631ec1dd8e1aceR27)
 should be `sudo chown -R travis.travis . $HOME/.wheelhouse/ $HOME/.cache/pip` 
instead.
   
   I think we can expose minikube as just another service container inside the 
docker-compose setup, but for the sake of getting the K8S tests back up, it 
looks good.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] gerardo commented on issue #3797: [AIRFLOW-2952] Splits CI into k8s + docker-compose

[GitBox]

gerardo commented on issue #3797: [AIRFLOW-2952] Splits CI into k8s + 
docker-compose
URL: 
https://github.com/apache/incubator-airflow/pull/3797#issuecomment-415640654
 
 
   @dimberman I'll have a look now


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-2953) Installing postgres and gcp_api doesn't install PostgresToGoogleCloudStorageOperator

[robert kelly (JIRA)]

robert kelly created AIRFLOW-2953:
-

 Summary: Installing postgres and gcp_api doesn't install 
PostgresToGoogleCloudStorageOperator
 Key: AIRFLOW-2953
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2953
 Project: Apache Airflow
  Issue Type: Bug
Reporter: robert kelly


I have gone through the installation process several times in python2 and 
python3.   I'm wondering how to handle this without manually copying the class.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] XD-DENG commented on a change in pull request #3793: [AIRFLOW-2948] Arg check & better doc - SSHOperator & SFTPOperator

[GitBox]

XD-DENG commented on a change in pull request #3793: [AIRFLOW-2948] Arg check & 
better doc - SSHOperator & SFTPOperator
URL: https://github.com/apache/incubator-airflow/pull/3793#discussion_r212488052
 
 

 ##
 File path: airflow/contrib/operators/sftp_operator.py
 ##
 @@ -77,13 +81,19 @@ def __init__(self,
 def execute(self, context):
 file_msg = None
 try:
-if self.ssh_conn_id and not self.ssh_hook:
-self.ssh_hook = SSHHook(ssh_conn_id=self.ssh_conn_id)
+if self.ssh_conn_id:
+if self.ssh_hook:
 
 Review comment:
   Agree on this.
   
   I will add the instance check and update tests for this (to test whether 
`ssh_hook` of wrong instance can be captured).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] XD-DENG commented on a change in pull request #3793: [AIRFLOW-2948] Arg check & better doc - SSHOperator & SFTPOperator

[GitBox]

XD-DENG commented on a change in pull request #3793: [AIRFLOW-2948] Arg check & 
better doc - SSHOperator & SFTPOperator
URL: https://github.com/apache/incubator-airflow/pull/3793#discussion_r212487903
 
 

 ##
 File path: airflow/contrib/operators/ssh_operator.py
 ##
 @@ -68,14 +72,20 @@ def __init__(self,
 
 def execute(self, context):
 try:
-if self.ssh_conn_id and not self.ssh_hook:
-self.ssh_hook = SSHHook(ssh_conn_id=self.ssh_conn_id,
-timeout=self.timeout)
+if self.ssh_conn_id:
+if self.ssh_hook:
+self.log.info("ssh_conn_id is ignored when ssh_hook is 
provided.")
+else:
+self.ssh_hook = SSHHook(ssh_conn_id=self.ssh_conn_id,
+timeout=self.timeout)
 
 if not self.ssh_hook:
 
 Review comment:
   The same: we need to guard the situation in which neither `ssh_conn_id` nor 
`ssh_hook` is provided.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] XD-DENG commented on a change in pull request #3793: [AIRFLOW-2948] Arg check & better doc - SSHOperator & SFTPOperator

[GitBox]

XD-DENG commented on a change in pull request #3793: [AIRFLOW-2948] Arg check & 
better doc - SSHOperator & SFTPOperator
URL: https://github.com/apache/incubator-airflow/pull/3793#discussion_r212488849
 
 

 ##
 File path: airflow/contrib/operators/sftp_operator.py
 ##
 @@ -77,13 +81,19 @@ def __init__(self,
 def execute(self, context):
 file_msg = None
 try:
-if self.ssh_conn_id and not self.ssh_hook:
-self.ssh_hook = SSHHook(ssh_conn_id=self.ssh_conn_id)
+if self.ssh_conn_id:
+if self.ssh_hook:
+self.log.info("ssh_conn_id is ignored when ssh_hook is 
provided.")
+else:
+self.ssh_hook = SSHHook(ssh_conn_id=self.ssh_conn_id)
 
 Review comment:
   I have noticed this as well. I'm following the existing implementation.
   
   `timeout` was added for `SSHOperator` in PR 
https://github.com/apache/incubator-airflow/pull/3666 . I'm not sure if there 
is any good reason for which `timeout` was added for `SSHOperator` but not 
added for the `SSHHook` in `SFTPOperator`. Giving I'm not confident about the 
reason behind, I don't intend to make change on this in this PR as well.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] XD-DENG commented on a change in pull request #3793: [AIRFLOW-2948] Arg check & better doc - SSHOperator & SFTPOperator

[GitBox]

XD-DENG commented on a change in pull request #3793: [AIRFLOW-2948] Arg check & 
better doc - SSHOperator & SFTPOperator
URL: https://github.com/apache/incubator-airflow/pull/3793#discussion_r212488052
 
 

 ##
 File path: airflow/contrib/operators/sftp_operator.py
 ##
 @@ -77,13 +81,19 @@ def __init__(self,
 def execute(self, context):
 file_msg = None
 try:
-if self.ssh_conn_id and not self.ssh_hook:
-self.ssh_hook = SSHHook(ssh_conn_id=self.ssh_conn_id)
+if self.ssh_conn_id:
+if self.ssh_hook:
 
 Review comment:
   Agree on this.
   
   I will add the instance check and update tests for this (to check whether 
`ssh_hook` of wrong instance can be captured).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] XD-DENG commented on a change in pull request #3793: [AIRFLOW-2948] Arg check & better doc - SSHOperator & SFTPOperator

[GitBox]

XD-DENG commented on a change in pull request #3793: [AIRFLOW-2948] Arg check & 
better doc - SSHOperator & SFTPOperator
URL: https://github.com/apache/incubator-airflow/pull/3793#discussion_r212487903
 
 

 ##
 File path: airflow/contrib/operators/ssh_operator.py
 ##
 @@ -68,14 +72,20 @@ def __init__(self,
 
 def execute(self, context):
 try:
-if self.ssh_conn_id and not self.ssh_hook:
-self.ssh_hook = SSHHook(ssh_conn_id=self.ssh_conn_id,
-timeout=self.timeout)
+if self.ssh_conn_id:
+if self.ssh_hook:
+self.log.info("ssh_conn_id is ignored when ssh_hook is 
provided.")
+else:
+self.ssh_hook = SSHHook(ssh_conn_id=self.ssh_conn_id,
+timeout=self.timeout)
 
 if not self.ssh_hook:
 
 Review comment:
   The same.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] XD-DENG commented on a change in pull request #3793: [AIRFLOW-2948] Arg check & better doc - SSHOperator & SFTPOperator

[GitBox]

XD-DENG commented on a change in pull request #3793: [AIRFLOW-2948] Arg check & 
better doc - SSHOperator & SFTPOperator
URL: https://github.com/apache/incubator-airflow/pull/3793#discussion_r212487852
 
 

 ##
 File path: airflow/contrib/operators/sftp_operator.py
 ##
 @@ -77,13 +81,19 @@ def __init__(self,
 def execute(self, context):
 file_msg = None
 try:
-if self.ssh_conn_id and not self.ssh_hook:
-self.ssh_hook = SSHHook(ssh_conn_id=self.ssh_conn_id)
+if self.ssh_conn_id:
+if self.ssh_hook:
+self.log.info("ssh_conn_id is ignored when ssh_hook is 
provided.")
+else:
+self.ssh_hook = SSHHook(ssh_conn_id=self.ssh_conn_id)
 
 if not self.ssh_hook:
 
 Review comment:
   This line is still necessary: what if neither `ssh_conn_id` nor `ssh_hook` 
is provided?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2951) dag_run end_date Null after a dag is finished

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590912#comment-16590912
 ] 

ASF GitHub Bot commented on AIRFLOW-2951:
-

YingboWang opened a new pull request #3798: [AIRFLOW-2951] Update dag_run table 
end_date when state change
URL: https://github.com/apache/incubator-airflow/pull/3798
 
 
   The existing airflow only change dag_run table end_date value when
   a user teminate a dag in web UI. The end_date will not be updated
   if airflow detected a dag finished and updated its state.
   
   This commit add end_date update in DagRun's set_state function to
   make up tho problem mentioned above.
   
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOWAIRFLOW-2951/) issues and 
references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-2951
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [x] Any finished dag should have a valid `end_date` in `dag_run` table, 
however this feature was not fully implemented. The `end_date` only exists when 
user try to mark a dag as success or failed from UI. This PR change the 
`DagRun.set_state()` function to set `end_date` when a `DagRun` state is 
changed and redirect existing dag_run state assignment to `set_state` function. 
It also fix UI to make sure a cleared dag will reset `end_date` to NULL.
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   tests/models.py:DagRunTest.test_dagrun_set_state_end_date
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Code Quality
   
   - [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   
   @yrqls21


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> dag_run end_date Null after a dag is finished
> -
>
> Key: AIRFLOW-2951
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2951
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: DagRun
>Reporter: Yingbo Wang
>Assignee: Yingbo Wang
>Priority: Major
>
> dag_run table should have an end_date updated when a dag is finished. 
> Currently only user activated dag termination request coming from UI may 
> change the "end_date" in dag_run table. All scheduled dags that are 
> automatically running by airflow will leave a NULL value after they fall into 
> a "success" or "failed" state. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] YingboWang opened a new pull request #3798: [AIRFLOW-2951] Update dag_run table end_date when state change

[GitBox]

YingboWang opened a new pull request #3798: [AIRFLOW-2951] Update dag_run table 
end_date when state change
URL: https://github.com/apache/incubator-airflow/pull/3798
 
 
   The existing airflow only change dag_run table end_date value when
   a user teminate a dag in web UI. The end_date will not be updated
   if airflow detected a dag finished and updated its state.
   
   This commit add end_date update in DagRun's set_state function to
   make up tho problem mentioned above.
   
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOWAIRFLOW-2951/) issues and 
references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-2951
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [x] Any finished dag should have a valid `end_date` in `dag_run` table, 
however this feature was not fully implemented. The `end_date` only exists when 
user try to mark a dag as success or failed from UI. This PR change the 
`DagRun.set_state()` function to set `end_date` when a `DagRun` state is 
changed and redirect existing dag_run state assignment to `set_state` function. 
It also fix UI to make sure a cleared dag will reset `end_date` to NULL.
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   tests/models.py:DagRunTest.test_dagrun_set_state_end_date
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Code Quality
   
   - [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   
   @yrqls21


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] dimberman commented on issue #3797: [AIRFLOW-2952] Splits CI into k8s + docker-compose

[GitBox]

dimberman commented on issue #3797: [AIRFLOW-2952] Splits CI into k8s + 
docker-compose
URL: 
https://github.com/apache/incubator-airflow/pull/3797#issuecomment-415601935
 
 
   
https://github.com/apache/incubator-airflow/commit/ede67299c4b2b64fc0a72a25838ba29682463d81#diff-354f30a63fb0907d4ad57269548329e3R43
 
   
   Looks like this might not be so simple. Getting errors based on:
   ```IOError: [Errno 13] Permission denied: 
u'/home/travis/.wheelhouse/bleach-2.1.4-py2.py3-none-any.whl'```
   
   @gerardo  any idea how to get around these permission issues?
   
   I have to run to an event but will revisit later tonight.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] dimberman commented on issue #3797: [AIRFLOW-2952] Splits CI into k8s + docker-compose

[GitBox]

dimberman commented on issue #3797: [AIRFLOW-2952] Splits CI into k8s + 
docker-compose
URL: 
https://github.com/apache/incubator-airflow/pull/3797#issuecomment-415597921
 
 
   @bolkedebruin @Fokko @kaxil  PTAL. I'm going to try to get this all working 
tonight (though I assume the tests should pass based on the travis running on 
my branch). 
   
   Kubernetes tests are silently running non-kubernetes airflow tests. This 
means that it will show up as passing as long as the non-kubernetes tests pass.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] dimberman commented on issue #3792: [AIRFLOW-2947] Added Helm chart for Kubernetes executor

[GitBox]

dimberman commented on issue #3792: [AIRFLOW-2947] Added Helm chart for 
Kubernetes executor
URL: 
https://github.com/apache/incubator-airflow/pull/3792#issuecomment-415596492
 
 
   Ok so that was caused by the recent dockerized ci pipeline. I'm addressing 
the issue here https://github.com/apache/incubator-airflow/pull/3797


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2938) Invalid 'extra' field in connection can raise an AttributeError when attempting to edit

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590896#comment-16590896
 ] 

ASF GitHub Bot commented on AIRFLOW-2938:
-

feng-tao closed pull request #3785: [AIRFLOW-2938] Handle improperly formatted 
extra field in connection …
URL: https://github.com/apache/incubator-airflow/pull/3785
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/airflow/hooks/http_hook.py b/airflow/hooks/http_hook.py
index c449fe0c15..caa89d3142 100644
--- a/airflow/hooks/http_hook.py
+++ b/airflow/hooks/http_hook.py
@@ -70,7 +70,11 @@ def get_conn(self, headers=None):
 if conn.login:
 session.auth = (conn.login, conn.password)
 if conn.extra:
-session.headers.update(conn.extra_dejson)
+try:
+session.headers.update(conn.extra_dejson)
+except TypeError:
+self.log.warn('Connection to {} has invalid extra 
field.'.format(
+conn.host))
 if headers:
 session.headers.update(headers)
 
diff --git a/airflow/www_rbac/views.py b/airflow/www_rbac/views.py
index 3ab872d5b1..d011724cc6 100644
--- a/airflow/www_rbac/views.py
+++ b/airflow/www_rbac/views.py
@@ -1922,6 +1922,11 @@ def prefill_form(self, form, pk):
 except Exception:
 d = {}
 
+if not hasattr(d, 'get'):
+logging.warning('extra field for {} is not iterable'.format(
+form.data.get('conn_id', '')))
+return
+
 for field in self.extra_fields:
 value = d.get(field, '')
 if value:


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Invalid 'extra' field in connection can raise an AttributeError when 
> attempting to edit
> ---
>
> Key: AIRFLOW-2938
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2938
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: William Pursell
>Assignee: William Pursell
>Priority: Minor
>
> When the extra field of a connection is not an iterable json object, 
> attempting to edit the connection can lead to a stack trace "Ooops": 
> AttributeError: 'NoneType' object has no attribute 'get'



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] feng-tao closed pull request #3785: [AIRFLOW-2938] Handle improperly formatted extra field in connection …

[GitBox]

feng-tao closed pull request #3785: [AIRFLOW-2938] Handle improperly formatted 
extra field in connection …
URL: https://github.com/apache/incubator-airflow/pull/3785
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/airflow/hooks/http_hook.py b/airflow/hooks/http_hook.py
index c449fe0c15..caa89d3142 100644
--- a/airflow/hooks/http_hook.py
+++ b/airflow/hooks/http_hook.py
@@ -70,7 +70,11 @@ def get_conn(self, headers=None):
 if conn.login:
 session.auth = (conn.login, conn.password)
 if conn.extra:
-session.headers.update(conn.extra_dejson)
+try:
+session.headers.update(conn.extra_dejson)
+except TypeError:
+self.log.warn('Connection to {} has invalid extra 
field.'.format(
+conn.host))
 if headers:
 session.headers.update(headers)
 
diff --git a/airflow/www_rbac/views.py b/airflow/www_rbac/views.py
index 3ab872d5b1..d011724cc6 100644
--- a/airflow/www_rbac/views.py
+++ b/airflow/www_rbac/views.py
@@ -1922,6 +1922,11 @@ def prefill_form(self, form, pk):
 except Exception:
 d = {}
 
+if not hasattr(d, 'get'):
+logging.warning('extra field for {} is not iterable'.format(
+form.data.get('conn_id', '')))
+return
+
 for field in self.extra_fields:
 value = d.get(field, '')
 if value:


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] dimberman opened a new pull request #3797: [AIRFLOW-2952] Splits CI into k8s + docker-compose

[GitBox]

dimberman opened a new pull request #3797: [AIRFLOW-2952] Splits CI into k8s + 
docker-compose
URL: https://github.com/apache/incubator-airflow/pull/3797
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-2952
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   Since using docker-compose for everything was causing k8s integration
   tests to die silently, this will determine whether a CI test is in k8s
   or docker-compose mode
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2952) Dockerized CI pipeline has silently broken integration testing for KubernetesExecutor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590891#comment-16590891
 ] 

ASF GitHub Bot commented on AIRFLOW-2952:
-

dimberman opened a new pull request #3797: [AIRFLOW-2952] Splits CI into k8s + 
docker-compose
URL: https://github.com/apache/incubator-airflow/pull/3797
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-2952
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   Since using docker-compose for everything was causing k8s integration
   tests to die silently, this will determine whether a CI test is in k8s
   or docker-compose mode
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Dockerized CI pipeline has silently broken integration testing for 
> KubernetesExecutor
> -
>
> Key: AIRFLOW-2952
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2952
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Daniel Imberman
>Assignee: Daniel Imberman
>Priority: Blocker
>
> [~gcuriel] [~bolke] [~Fokko]
> Looking at all recent builds the new CI pipeline is silently reverting the 
> kubernetes tests to the normal airflow tests.
> Before
> https://travis-ci.org/apache/incubator-airflow/jobs/418914949#L1007
> After:
> [https://travis-ci.org/apache/incubator-airflow/jobs/419062412#L4970]
> This means that kubernetes builds will pass without actually testing on a 
> kubernetes cluster.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2952) Dockerized CI pipeline has silently broken integration testing for KubernetesExecutor

[Daniel Imberman (JIRA)]

Daniel Imberman created AIRFLOW-2952:


 Summary: Dockerized CI pipeline has silently broken integration 
testing for KubernetesExecutor
 Key: AIRFLOW-2952
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2952
 Project: Apache Airflow
  Issue Type: Bug
Reporter: Daniel Imberman
Assignee: Daniel Imberman


[~gcuriel] [~bolke] [~Fokko]

Looking at all recent builds the new CI pipeline is silently reverting the 
kubernetes tests to the normal airflow tests.

Before
https://travis-ci.org/apache/incubator-airflow/jobs/418914949#L1007
After:
[https://travis-ci.org/apache/incubator-airflow/jobs/419062412#L4970]

This means that kubernetes builds will pass without actually testing on a 
kubernetes cluster.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] dimberman commented on issue #3792: [AIRFLOW-2947] Added Helm chart for Kubernetes executor

[GitBox]

dimberman commented on issue #3792: [AIRFLOW-2947] Added Helm chart for 
Kubernetes executor
URL: 
https://github.com/apache/incubator-airflow/pull/3792#issuecomment-415588605
 
 
   Wait. No. This is on the master branch. Thats... not good. Going to 
investigate further.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] dimberman commented on issue #3792: [AIRFLOW-2947] Added Helm chart for Kubernetes executor

[GitBox]

dimberman commented on issue #3792: [AIRFLOW-2947] Added Helm chart for 
Kubernetes executor
URL: 
https://github.com/apache/incubator-airflow/pull/3792#issuecomment-415587425
 
 
   @jpds It looks like you removed the kube/deploy folder but it's not removed 
from the setup_kubernetes script 
https://github.com/apache/incubator-airflow/blob/5bd5a7bad638241c01ac1ddd0fe8f8c7a3e95d27/scripts/ci/kubernetes/setup_kubernetes.sh.
 
   
   It looks like this PR has screwed up the kubernetes tests and now they're 
just running the normal airflow tests. 
   
   https://travis-ci.org/apache/incubator-airflow/jobs/419688983


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] tswast commented on a change in pull request #3733: [AIRFLOW-491] Add cache parameter in BigQuery query method - with 'api_resource_configs'

[GitBox]

tswast commented on a change in pull request #3733: [AIRFLOW-491] Add cache 
parameter in BigQuery query method - with 'api_resource_configs'
URL: https://github.com/apache/incubator-airflow/pull/3733#discussion_r212471376
 
 

 ##
 File path: airflow/contrib/hooks/bigquery_hook.py
 ##
 @@ -587,15 +603,22 @@ def run_query(self,
 if use_legacy_sql is None:
 use_legacy_sql = self.use_legacy_sql
 
-configuration = {
-'query': {
+configuration = deepcopy(api_resource_configs)
+
+query_default =  {
 'query': sql,
 'useLegacySql': use_legacy_sql,
 'maximumBillingTier': maximum_billing_tier,
 'maximumBytesBilled': maximum_bytes_billed,
 'priority': priority
 }
-}
+
+if not 'query' in configuration:
+configuration['query'] = query_default
+else:
+for param in query_default:
+if param not in configuration['query']:
 
 Review comment:
   I don't like that we might silently discard input values such as `sql` with 
this `if` statement. Maybe we should raise an error if the key is already 
defined in `configuration` and the desired default value is not `None`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] dimberman commented on a change in pull request #3782: [AIRFLOW-2936] Use official Python images as base image for Docker

[GitBox]

dimberman commented on a change in pull request #3782: [AIRFLOW-2936] Use 
official Python images as base image for Docker
URL: https://github.com/apache/incubator-airflow/pull/3782#discussion_r212471258
 
 

 ##
 File path: scripts/ci/kubernetes/docker/airflow-init.sh
 ##
 @@ -17,9 +17,10 @@
 #  specific language governing permissions and limitations  *
 #  under the License.
 
-cd /usr/local/lib/python2.7/dist-packages/airflow && \
-cp -R example_dags/* /root/airflow/dags/ && \
+set -e
+
+cd /usr/local/lib/python3.7/site-packages/airflow/ && \
+cp -R example_dags/* /home/airflow/dags/ && \
 airflow initdb && \
 alembic upgrade heads && \
-(airflow create_user -u airflow -l airflow -f jon -e airf...@apache.org -r 
Admin -p airflow || true) && \
-echo "retrieved from mount" > /root/test_volume/test.txt
 
 Review comment:
   Why did you remove this line? Isn't this necessary for the volume mounting 
tests?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] tswast commented on a change in pull request #3733: [AIRFLOW-491] Add cache parameter in BigQuery query method - with 'api_resource_configs'

[GitBox]

tswast commented on a change in pull request #3733: [AIRFLOW-491] Add cache 
parameter in BigQuery query method - with 'api_resource_configs'
URL: https://github.com/apache/incubator-airflow/pull/3733#discussion_r212469303
 
 

 ##
 File path: airflow/contrib/hooks/bigquery_hook.py
 ##
 @@ -656,6 +668,21 @@ def run_query(self,
 configuration['query'][
 'schemaUpdateOptions'] = schema_update_options
 
+if not api_resource_configs:
+api_resource_configs = self.api_resource_configs
+
+if 'configuration' in api_resource_configs and isinstance(
 
 Review comment:
   Yes. Since `api_resource_configs` basically _is_ `configuration`, a straight 
`deepcopy` should work as a first step.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] xnuinside commented on a change in pull request #3733: [AIRFLOW-491] Add cache parameter in BigQuery query method - with 'api_resource_configs'

[GitBox]

xnuinside commented on a change in pull request #3733: [AIRFLOW-491] Add cache 
parameter in BigQuery query method - with 'api_resource_configs'
URL: https://github.com/apache/incubator-airflow/pull/3733#discussion_r212469186
 
 

 ##
 File path: airflow/contrib/hooks/bigquery_hook.py
 ##
 @@ -656,6 +668,21 @@ def run_query(self,
 configuration['query'][
 'schemaUpdateOptions'] = schema_update_options
 
+if not api_resource_configs:
+api_resource_configs = self.api_resource_configs
+
+if 'configuration' in api_resource_configs and isinstance(
 
 Review comment:
   check pls


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] xnuinside commented on a change in pull request #3733: [AIRFLOW-491] Add cache parameter in BigQuery query method - with 'api_resource_configs'

[GitBox]

xnuinside commented on a change in pull request #3733: [AIRFLOW-491] Add cache 
parameter in BigQuery query method - with 'api_resource_configs'
URL: https://github.com/apache/incubator-airflow/pull/3733#discussion_r212466137
 
 

 ##
 File path: airflow/contrib/hooks/bigquery_hook.py
 ##
 @@ -656,6 +668,21 @@ def run_query(self,
 configuration['query'][
 'schemaUpdateOptions'] = schema_update_options
 
+if not api_resource_configs:
+api_resource_configs = self.api_resource_configs
+
+if 'configuration' in api_resource_configs and isinstance(
 
 Review comment:
   Do you mean to do a copy of api_config in configuration at first and then 
add other params to configuration if they do not exist? Am I right understand? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] xnuinside commented on a change in pull request #3733: [AIRFLOW-491] Add cache parameter in BigQuery query method - with 'api_resource_configs'

[GitBox]

xnuinside commented on a change in pull request #3733: [AIRFLOW-491] Add cache 
parameter in BigQuery query method - with 'api_resource_configs'
URL: https://github.com/apache/incubator-airflow/pull/3733#discussion_r212466137
 
 

 ##
 File path: airflow/contrib/hooks/bigquery_hook.py
 ##
 @@ -656,6 +668,21 @@ def run_query(self,
 configuration['query'][
 'schemaUpdateOptions'] = schema_update_options
 
+if not api_resource_configs:
+api_resource_configs = self.api_resource_configs
+
+if 'configuration' in api_resource_configs and isinstance(
 
 Review comment:
   Do you mean first do a copy of api_config in configuration and then add 
other params to configuration if they do not exist? Am I right understand? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] Fokko commented on issue #3780: [AIRFLOW-2933] Enable Codecov on Docker-CI Build

[GitBox]

Fokko commented on issue #3780: [AIRFLOW-2933] Enable Codecov on Docker-CI Build
URL: 
https://github.com/apache/incubator-airflow/pull/3780#issuecomment-415578973
 
 
   So the coverage report gets uploaded: 
https://codecov.io/gh/apache/incubator-airflow/commit/c1566951dcecb91a2a739700e2b84f3dcee02c6b
   
   not sure why the notification is not posted to Github. Apart from that, I 
would like to see to get this merged so we get the reporting back on master.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] tswast commented on a change in pull request #3733: [AIRFLOW-491] Add cache parameter in BigQuery query method - with 'api_resource_configs'

[GitBox]

tswast commented on a change in pull request #3733: [AIRFLOW-491] Add cache 
parameter in BigQuery query method - with 'api_resource_configs'
URL: https://github.com/apache/incubator-airflow/pull/3733#discussion_r212463435
 
 

 ##
 File path: airflow/contrib/hooks/bigquery_hook.py
 ##
 @@ -656,6 +668,21 @@ def run_query(self,
 configuration['query'][
 'schemaUpdateOptions'] = schema_update_options
 
+if not api_resource_configs:
+api_resource_configs = self.api_resource_configs
+
+if 'configuration' in api_resource_configs and isinstance(
 
 Review comment:
   I'm having a hard time understanding this logic with all the nested loops 
and type checking.
   
   I think it'd be easier if we started off by setting `configuration = 
copy.deepcopy(api_resource_configs)` just before 
https://github.com/apache/incubator-airflow/blob/e933ee8e52e0449d88bac0c83f6ee06729988343/airflow/contrib/hooks/bigquery_hook.py#L602
   
   We should warn anywhere where we set a `key` by one of the other arguments 
but it was already present in `api_resource_configs`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Assigned] (AIRFLOW-2237) Airflow SLA is not checked for subdag tasks

[Tomas Zubiri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tomas Zubiri reassigned AIRFLOW-2237:
-

Assignee: Tomas Zubiri

> Airflow SLA is not checked for subdag tasks
> ---
>
> Key: AIRFLOW-2237
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2237
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: scheduler
>Affects Versions: 1.8.0, 1.9.0
>Reporter: barak schoster
>Assignee: Tomas Zubiri
>Priority: Major
>
> If a task is inside a subdag it's SLA parameters are ignored and not handled 
> by the scheduler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] feng-tao commented on a change in pull request #3785: [AIRFLOW-2938] Handle improperly formatted extra field in connection …

[GitBox]

feng-tao commented on a change in pull request #3785: [AIRFLOW-2938] Handle 
improperly formatted extra field in connection …
URL: https://github.com/apache/incubator-airflow/pull/3785#discussion_r212455107
 
 

 ##
 File path: airflow/www_rbac/views.py
 ##
 @@ -1923,7 +1923,10 @@ def prefill_form(self, form, pk):
 d = {}
 
 for field in self.extra_fields:
-value = d.get(field, '')
+try:
+value = d.get(field, '')
+except Exception:
 
 Review comment:
   lgtm


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao commented on issue #3790: [AIRFLOW-2994] Fix command status check in Qubole Check operator

[GitBox]

feng-tao commented on issue #3790: [AIRFLOW-2994] Fix command status check in 
Qubole Check operator
URL: 
https://github.com/apache/incubator-airflow/pull/3790#issuecomment-415564978
 
 
   It seems python3 fails. Do you know if it is related?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao commented on a change in pull request #3796: [AIRFLOW-2824] - Add config to disable default conn creation

[GitBox]

feng-tao commented on a change in pull request #3796: [AIRFLOW-2824] - Add 
config to disable default conn creation
URL: https://github.com/apache/incubator-airflow/pull/3796#discussion_r212450745
 
 

 ##
 File path: airflow/utils/db.py
 ##
 @@ -286,6 +284,16 @@ def initdb(rbac=False):
 conn_id='cassandra_default', conn_type='cassandra',
 host='cassandra', port=9042))
 
+
+def initdb(rbac=False):
+session = settings.Session()
+
+from airflow import models
 
 Review comment:
   is there a reason we put `airflow import models` as local import not global 
import?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2824) Disable loading of default connections via airflow config

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590785#comment-16590785
 ] 

ASF GitHub Bot commented on AIRFLOW-2824:
-

andscoop opened a new pull request #3796: [AIRFLOW-2824] - Add config to 
disable default conn creation
URL: https://github.com/apache/incubator-airflow/pull/3796
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-2824
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   This PR gives users a way to avoid recreating default connections when 
running initDB via the airflow.cfg file
   
   Before merging - I wanted feedback from the maintainers on whether or not 
they feel this is the way to approach resolving this issue.
   
   ### Tests
   
   - [] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Disable loading of default connections via airflow config
> -
>
> Key: AIRFLOW-2824
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2824
> Project: Apache Airflow
>  Issue Type: Wish
>Reporter: Felix Uellendall
>Priority: Major
>
> I would love to have a variable I can set in the airflow.cfg, like the DAG 
> examples have, to not load the default connections.
> Either by using {{load_examples}} that is already 
> [there|https://github.com/apache/incubator-airflow/blob/dfa7b26ddaca80ee8fd9915ee9f6eac50fac77f6/airflow/config_templates/default_airflow.cfg#L128]
>  for loading dag examples or by a new one like {{load_default_connections}} 
> to check if the user wants to have it or not.
> The implementation of the default connections starts 
> [here|https://github.com/apache/incubator-airflow/blob/9e1d8ee837ea2c23e828d070b6a72a6331d98602/airflow/utils/db.py#L94]
> Let me know what you guys think of it, pls. :)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work started] (AIRFLOW-2951) dag_run end_date Null after a dag is finished

[Yingbo Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on AIRFLOW-2951 started by Yingbo Wang.

> dag_run end_date Null after a dag is finished
> -
>
> Key: AIRFLOW-2951
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2951
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: DagRun
>Reporter: Yingbo Wang
>Assignee: Yingbo Wang
>Priority: Major
>
> dag_run table should have an end_date updated when a dag is finished. 
> Currently only user activated dag termination request coming from UI may 
> change the "end_date" in dag_run table. All scheduled dags that are 
> automatically running by airflow will leave a NULL value after they fall into 
> a "success" or "failed" state. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2951) dag_run end_date Null after a dag is finished

[Yingbo Wang (JIRA)]

Yingbo Wang created AIRFLOW-2951:


 Summary: dag_run end_date Null after a dag is finished
 Key: AIRFLOW-2951
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2951
 Project: Apache Airflow
  Issue Type: Improvement
  Components: DagRun
Reporter: Yingbo Wang
Assignee: Yingbo Wang


dag_run table should have an end_date updated when a dag is finished. 

Currently only user activated dag termination request coming from UI may change 
the "end_date" in dag_run table. All scheduled dags that are automatically 
running by airflow will leave a NULL value after they fall into a "success" or 
"failed" state. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] andscoop opened a new pull request #3796: [AIRFLOW-2824] - Add config to disable default conn creation

[GitBox]

andscoop opened a new pull request #3796: [AIRFLOW-2824] - Add config to 
disable default conn creation
URL: https://github.com/apache/incubator-airflow/pull/3796
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-2824
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   This PR gives users a way to avoid recreating default connections when 
running initDB via the airflow.cfg file
   
   Before merging - I wanted feedback from the maintainers on whether or not 
they feel this is the way to approach resolving this issue.
   
   ### Tests
   
   - [] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2932) GoogleCloudStorageHook - allow compression of file

[Iuliia Volkova (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590781#comment-16590781
 ] 

Iuliia Volkova commented on AIRFLOW-2932:
-

[~kaxilnaik] and [~tswast] what do you think about this task? I can implement 
it this flag - gzip=False by default and if it's True - when compress file 
before upload.

> GoogleCloudStorageHook - allow compression of file
> --
>
> Key: AIRFLOW-2932
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2932
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: jack
>Priority: Major
> Fix For: 1.10.1
>
>
> The  *upload*{color:#55}({color}_bucket_{color:#55}, 
> {color}_object_{color:#55}, {color}_filename_{color:#55}, 
> {color}_mime_type='application/octet-stream'_{color:#55}){color} function 
> allows to upload file from local disk.
> The google cloud support GZIP and BigQuery can read GZIP files. So, most 
> people upload compressed files in order to save space.
> It would be nice if the upload function would be able to make the compression 
> on it's own (if asked by the user). This will save the trouble of having to 
> compress the file by ourselves.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] xnuinside commented on a change in pull request #3733: [AIRFLOW-491] Add cache parameter in BigQuery query method - with 'api_resource_configs'

[GitBox]

xnuinside commented on a change in pull request #3733: [AIRFLOW-491] Add cache 
parameter in BigQuery query method - with 'api_resource_configs'
URL: https://github.com/apache/incubator-airflow/pull/3733#discussion_r212444862
 
 

 ##
 File path: tests/contrib/hooks/test_bigquery_hook.py
 ##
 @@ -281,6 +281,18 @@ def test_run_query_sql_dialect_override(self, 
run_with_config):
 args, kwargs = run_with_config.call_args
 self.assertIs(args[0]['query']['useLegacySql'], bool_val)
 
+@mock.patch.object(hook.BigQueryBaseCursor, 'run_with_configuration')
+def test_api_resource_configs(self, run_with_config):
+for bool_val in [True, False]:
+cursor = hook.BigQueryBaseCursor(mock.Mock(), "project_id")
+cursor.run_query('query',
+ api_resource_configs={
+ 'configuration':
+ {'query': {'useQueryCache': bool_val}}})
+
+args, kwargs = run_with_config.call_args
+self.assertIs(args[0]['query']['useQueryCache'], bool_val)
 
 Review comment:
   and added check that default values exist and were not delete


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao commented on a change in pull request #3793: [AIRFLOW-2948] Arg check & better doc - SSHOperator & SFTPOperator

[GitBox]

feng-tao commented on a change in pull request #3793: [AIRFLOW-2948] Arg check 
& better doc - SSHOperator & SFTPOperator
URL: https://github.com/apache/incubator-airflow/pull/3793#discussion_r212443735
 
 

 ##
 File path: airflow/contrib/operators/sftp_operator.py
 ##
 @@ -77,13 +81,19 @@ def __init__(self,
 def execute(self, context):
 file_msg = None
 try:
-if self.ssh_conn_id and not self.ssh_hook:
-self.ssh_hook = SSHHook(ssh_conn_id=self.ssh_conn_id)
+if self.ssh_conn_id:
+if self.ssh_hook:
 
 Review comment:
   And I wonder we should check the self.ssh_hook is an instance for SSHHook.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] xnuinside commented on a change in pull request #3733: [AIRFLOW-491] Add cache parameter in BigQuery query method - with 'api_resource_configs'

[GitBox]

xnuinside commented on a change in pull request #3733: [AIRFLOW-491] Add cache 
parameter in BigQuery query method - with 'api_resource_configs'
URL: https://github.com/apache/incubator-airflow/pull/3733#discussion_r212443568
 
 

 ##
 File path: tests/contrib/hooks/test_bigquery_hook.py
 ##
 @@ -281,6 +281,18 @@ def test_run_query_sql_dialect_override(self, 
run_with_config):
 args, kwargs = run_with_config.call_args
 self.assertIs(args[0]['query']['useLegacySql'], bool_val)
 
+@mock.patch.object(hook.BigQueryBaseCursor, 'run_with_configuration')
+def test_api_resource_configs(self, run_with_config):
+for bool_val in [True, False]:
+cursor = hook.BigQueryBaseCursor(mock.Mock(), "project_id")
+cursor.run_query('query',
+ api_resource_configs={
+ 'configuration':
+ {'query': {'useQueryCache': bool_val}}})
+
+args, kwargs = run_with_config.call_args
+self.assertIs(args[0]['query']['useQueryCache'], bool_val)
 
 Review comment:
   resolved with changing the logic in upper comment


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao commented on a change in pull request #3795: [AIRFLOW-2949] Add syntax highlight for single quote strings

[GitBox]

feng-tao commented on a change in pull request #3795: [AIRFLOW-2949] Add syntax 
highlight for single quote strings
URL: https://github.com/apache/incubator-airflow/pull/3795#discussion_r212443447
 
 

 ##
 File path: airflow/www/static/main.css
 ##
 @@ -262,3 +262,4 @@ div.square {
 .sc { color: #BA2121 } /* Literal.String.Char */
 .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */
 .s2 { color: #BA2121 } /* Literal.String.Double */
+.s1 { color: #BA2121 } /* Literal.String.Single */
 
 Review comment:
   Given we are about to retire old UI, do you want to modify the new rbac 
UI(www_rbac). FYI, the new UI is managed by npm, so I assume you need to modify 
the npm package?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao commented on a change in pull request #3791: Adding King.com to the list of companies.

[GitBox]

feng-tao commented on a change in pull request #3791: Adding King.com to the 
list of companies.
URL: https://github.com/apache/incubator-airflow/pull/3791#discussion_r212442372
 
 

 ##
 File path: README.md
 ##
 @@ -184,6 +184,7 @@ Currently **officially** using Airflow:
 1. [JobTeaser](https://www.jobteaser.com) 
[[@stefani75](https://github.com/stefani75) &  
[@knil-sama](https://github.com/knil-sama)]
 1. [Kalibrr](https://www.kalibrr.com/) 
[[@charlesverdad](https://github.com/charlesverdad)]
 1. [Karmic](https://karmiclabs.com) [[@hyw](https://github.com/hyw)]
+1. [King.com](https://www.king.com)
 
 Review comment:
   could you follow the other format(e.g [Karmic](https://karmiclabs.com) 
[[@hyw](https://github.com/hyw)])?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao commented on a change in pull request #3793: [AIRFLOW-2948] Arg check & better doc - SSHOperator & SFTPOperator

[GitBox]

feng-tao commented on a change in pull request #3793: [AIRFLOW-2948] Arg check 
& better doc - SSHOperator & SFTPOperator
URL: https://github.com/apache/incubator-airflow/pull/3793#discussion_r212441692
 
 

 ##
 File path: airflow/contrib/operators/ssh_operator.py
 ##
 @@ -68,14 +72,20 @@ def __init__(self,
 
 def execute(self, context):
 try:
-if self.ssh_conn_id and not self.ssh_hook:
-self.ssh_hook = SSHHook(ssh_conn_id=self.ssh_conn_id,
-timeout=self.timeout)
+if self.ssh_conn_id:
+if self.ssh_hook:
+self.log.info("ssh_conn_id is ignored when ssh_hook is 
provided.")
+else:
+self.ssh_hook = SSHHook(ssh_conn_id=self.ssh_conn_id,
+timeout=self.timeout)
 
 if not self.ssh_hook:
 
 Review comment:
   same


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao commented on a change in pull request #3793: [AIRFLOW-2948] Arg check & better doc - SSHOperator & SFTPOperator

[GitBox]

feng-tao commented on a change in pull request #3793: [AIRFLOW-2948] Arg check 
& better doc - SSHOperator & SFTPOperator
URL: https://github.com/apache/incubator-airflow/pull/3793#discussion_r212441652
 
 

 ##
 File path: airflow/contrib/operators/sftp_operator.py
 ##
 @@ -77,13 +81,19 @@ def __init__(self,
 def execute(self, context):
 file_msg = None
 try:
-if self.ssh_conn_id and not self.ssh_hook:
-self.ssh_hook = SSHHook(ssh_conn_id=self.ssh_conn_id)
+if self.ssh_conn_id:
+if self.ssh_hook:
+self.log.info("ssh_conn_id is ignored when ssh_hook is 
provided.")
+else:
+self.ssh_hook = SSHHook(ssh_conn_id=self.ssh_conn_id)
 
 if not self.ssh_hook:
 
 Review comment:
   in this case do we still need this line?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao commented on a change in pull request #3793: [AIRFLOW-2948] Arg check & better doc - SSHOperator & SFTPOperator

[GitBox]

feng-tao commented on a change in pull request #3793: [AIRFLOW-2948] Arg check 
& better doc - SSHOperator & SFTPOperator
URL: https://github.com/apache/incubator-airflow/pull/3793#discussion_r212442049
 
 

 ##
 File path: airflow/contrib/operators/sftp_operator.py
 ##
 @@ -77,13 +81,19 @@ def __init__(self,
 def execute(self, context):
 file_msg = None
 try:
-if self.ssh_conn_id and not self.ssh_hook:
-self.ssh_hook = SSHHook(ssh_conn_id=self.ssh_conn_id)
+if self.ssh_conn_id:
+if self.ssh_hook:
+self.log.info("ssh_conn_id is ignored when ssh_hook is 
provided.")
+else:
+self.ssh_hook = SSHHook(ssh_conn_id=self.ssh_conn_id)
 
 Review comment:
   why don't we set timeout for this hook but the one in SSHOperator?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2949) Syntax Highlight for Single Quote

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590723#comment-16590723
 ] 

ASF GitHub Bot commented on AIRFLOW-2949:
-

tzulberti-jampp opened a new pull request #3795: [AIRFLOW-2949] Add syntax 
highlight for single quote strings
URL: https://github.com/apache/incubator-airflow/pull/3795
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   Added .s1 style class for single quoted strings on main.css to have color 
highlight for single quote strings. 
   
   ### Tests
   
   There are no tests for the CSS selectos on main.css
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [X] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Syntax Highlight for Single Quote
> -
>
> Key: AIRFLOW-2949
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2949
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Tomas Zulberti
>Priority: Major
> Attachments: image-2018-08-23-16-16-59-375.png
>
>
> When checking the code of any DAG, there is a highlight for double quote 
> strings but there isn't any for single quote strings. pygments generate a 
> special css class but there is no color asigned for them
>  
> !image-2018-08-23-16-16-59-375.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] tzulberti-jampp opened a new pull request #3795: [AIRFLOW-2949] Add syntax highlight for single quote strings

[GitBox]

tzulberti-jampp opened a new pull request #3795: [AIRFLOW-2949] Add syntax 
highlight for single quote strings
URL: https://github.com/apache/incubator-airflow/pull/3795
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   Added .s1 style class for single quoted strings on main.css to have color 
highlight for single quote strings. 
   
   ### Tests
   
   There are no tests for the CSS selectos on main.css
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [X] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-2950) Running Airflow behind a proxy

[Shivakumar Gopalakrishnan (JIRA)]

Shivakumar Gopalakrishnan created AIRFLOW-2950:
--

 Summary: Running Airflow behind a proxy
 Key: AIRFLOW-2950
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2950
 Project: Apache Airflow
  Issue Type: Bug
Reporter: Shivakumar Gopalakrishnan


Airflow with a Celary+SQS configuration does not work behind a proxy

it will be nice to add variables http_proxy, https_proxy & no_proxy as part of 
the core configuration so that they can be used internally



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2949) Syntax Highlight for Single Quote

[Tomas Zulberti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tomas Zulberti updated AIRFLOW-2949:

Summary: Syntax Highlight for Single Quote  (was: Syntax highlight for 
single quote)

> Syntax Highlight for Single Quote
> -
>
> Key: AIRFLOW-2949
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2949
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Tomas Zulberti
>Priority: Major
> Attachments: image-2018-08-23-16-16-59-375.png
>
>
> When checking the code of any DAG, there is a highlight for double quote 
> strings but there isn't any for single quote strings. pygments generate a 
> special css class but there is no color asigned for them
>  
> !image-2018-08-23-16-16-59-375.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2949) Syntax highlight for single quote

[Tomas Zulberti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tomas Zulberti updated AIRFLOW-2949:

Summary: Syntax highlight for single quote  (was: Syntax Highligt for 
single quote)

> Syntax highlight for single quote
> -
>
> Key: AIRFLOW-2949
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2949
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Tomas Zulberti
>Priority: Major
> Attachments: image-2018-08-23-16-16-59-375.png
>
>
> When checking the code of any DAG, there is a highlight for double quote 
> strings but there isn't any for single quote strings. pygments generate a 
> special css class but there is no color asigned for them
>  
> !image-2018-08-23-16-16-59-375.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2949) Syntax Highligt for single quote

[Tomas Zulberti (JIRA)]

Tomas Zulberti created AIRFLOW-2949:
---

 Summary: Syntax Highligt for single quote
 Key: AIRFLOW-2949
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2949
 Project: Apache Airflow
  Issue Type: Improvement
Reporter: Tomas Zulberti
 Attachments: image-2018-08-23-16-16-59-375.png

When checking the code of any DAG, there is a highlight for double quote 
strings but there isn't any for single quote strings. pygments generate a 
special css class but there is no color asigned for them

 

!image-2018-08-23-16-16-59-375.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] feng-tao closed pull request #3794: [AIRFLOW-XXX] Added G Adventures to Users

[GitBox]

feng-tao closed pull request #3794: [AIRFLOW-XXX] Added G Adventures to Users
URL: https://github.com/apache/incubator-airflow/pull/3794
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/README.md b/README.md
index 34296c1cd3..606f2da745 100644
--- a/README.md
+++ b/README.md
@@ -154,6 +154,7 @@ Currently **officially** using Airflow:
 1. [Flipp](https://www.flipp.com) 
[[@sethwilsonwishabi](https://github.com/sethwilsonwishabi)]
 1. [FreshBooks](https://github.com/freshbooks) 
[[@DinoCow](https://github.com/DinoCow)]
 1. [Fundera](https://fundera.com) 
[[@andyxhadji](https://github.com/andyxhadji)]
+1. [G Adventures](https://gadventures.com) 
[[@samuelmullin](https://github.com/samuelmullin)]
 1. [GameWisp](https://gamewisp.com) [[@tjbiii](https://github.com/TJBIII) & 
[@theryanwalls](https://github.com/theryanwalls)]
 1. [Gentner Lab](http://github.com/gentnerlab) 
[[@neuromusic](https://github.com/neuromusic)]
 1. [Glassdoor](https://github.com/Glassdoor) 
[[@syvineckruyk](https://github.com/syvineckruyk)]


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao commented on a change in pull request #3786: [AIRFLOW-XXX] Note on min_file_process_interval

[GitBox]

feng-tao commented on a change in pull request #3786: [AIRFLOW-XXX] Note on 
min_file_process_interval
URL: https://github.com/apache/incubator-airflow/pull/3786#discussion_r212398092
 
 

 ##
 File path: airflow/config_templates/default_airflow.cfg
 ##
 @@ -432,7 +432,8 @@ scheduler_heartbeat_sec = 5
 # -1 indicates to run continuously (see also num_runs)
 run_duration = -1
 
-# after how much time (seconds) a new DAGs should be picked up from the 
filesystem
+# After how much time (seconds) a new DAGs should be picked up from the 
filesystem.
+# Note that this is also increases interval between new tasks being scheduled
 
 Review comment:
   nit:  `Note that this also increases the interval between new tasks being 
scheduled`
   
   And could you help to elaborate on why it is the case?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] samuelmullin opened a new pull request #3794: [AIRFLOW-XXX] Added G Adventures to Users

[GitBox]

samuelmullin opened a new pull request #3794: [AIRFLOW-XXX] Added G Adventures 
to Users
URL: https://github.com/apache/incubator-airflow/pull/3794
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [X] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [X] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [X] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [X] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [X] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [X] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] wrp commented on a change in pull request #3785: [AIRFLOW-2938] Handle improperly formatted extra field in connection …

[GitBox]

wrp commented on a change in pull request #3785: [AIRFLOW-2938] Handle 
improperly formatted extra field in connection …
URL: https://github.com/apache/incubator-airflow/pull/3785#discussion_r212363527
 
 

 ##
 File path: airflow/www_rbac/views.py
 ##
 @@ -1923,7 +1923,10 @@ def prefill_form(self, form, pk):
 d = {}
 
 for field in self.extra_fields:
-value = d.get(field, '')
+try:
+value = d.get(field, '')
+except Exception:
 
 Review comment:
   The extra field is user generated, so must be treated with care.  The 
particular problem I'm seeing is when it is the string "null", and then 
clicking on the edit button fails.  I've just added a logging.warning, but it's 
not clear to me if the message is accurate since this code path is probably 
used with forms that do not have a 'conn_id' fieldbut an ambiguous warning 
message may be better than the current user experience.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] gsemet commented on issue #3792: [AIRFLOW-2947] Added Helm chart for Kubernetes executor

[GitBox]

gsemet commented on issue #3792: [AIRFLOW-2947] Added Helm chart for Kubernetes 
executor
URL: 
https://github.com/apache/incubator-airflow/pull/3792#issuecomment-415460295
 
 
   @ashb I do not see what your remark has to do with Airflow, to what I see 
and use in other charts in this repository, they almost all follow the same 
structure, and this airflow chart looks like every other. If you want to deploy 
this chart without Tiller, you can do `helm template` and then `kubectl apply`. 
Or use Helm in client mode only.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] dimberman commented on issue #3792: [AIRFLOW-2947] Added Helm chart for Kubernetes executor

[GitBox]

dimberman commented on issue #3792: [AIRFLOW-2947] Added Helm chart for 
Kubernetes executor
URL: 
https://github.com/apache/incubator-airflow/pull/3792#issuecomment-415454497
 
 
   I think we're in a similar mind about the current state of helm. I think
   this'll be useful for people wanting to test airflow out or build quick
   POCs, but hopefully for production they use something like the airflow
   operator.
   
   On Thu, Aug 23, 2018, 8:05 AM Ash Berlin-Taylor 
   wrote:
   
   > Yeah, I'm waiting quite eagerly for Helm3. I'm not a fan of this right
   > now, but I also don't feel that strongly about it, certainly not enough to
   > veto or anything like that.
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or mute the thread
   > 

   > .
   >
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ashb commented on issue #3792: [AIRFLOW-2947] Added Helm chart for Kubernetes executor

[GitBox]

ashb commented on issue #3792: [AIRFLOW-2947] Added Helm chart for Kubernetes 
executor
URL: 
https://github.com/apache/incubator-airflow/pull/3792#issuecomment-415451730
 
 
   Yeah, I'm waiting quite eagerly for Helm3. I'm not a fan of this right now, 
but I also don't feel that strongly about it, certainly not enough to veto or 
anything like that.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] tedmiston commented on issue #3656: [WIP][AIRFLOW-2803] Fix all ESLint issues

[GitBox]

tedmiston commented on issue #3656: [WIP][AIRFLOW-2803] Fix all ESLint issues
URL: 
https://github.com/apache/incubator-airflow/pull/3656#issuecomment-415451613
 
 
   @verdan Yes! I'll push up my new work on this today.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2948) Arg checking & better doc for SSHOperator and SFTPOperator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590346#comment-16590346
 ] 

ASF GitHub Bot commented on AIRFLOW-2948:
-

XD-DENG opened a new pull request #3793: [AIRFLOW-2948] Arg check & better doc 
- SSHOperator & SFTPOperator
URL: https://github.com/apache/incubator-airflow/pull/3793
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-2948
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   There may be different combinations of arguments, and some processings are 
being done 'silently', while users may not be fully aware of them.
   
   For example
   - User only needs to provide either `ssh_hook` or `ssh_conn_id`, while this 
is not clear in doc
   - if both provided, `ssh_conn_id` will be ignored.
   - if `remote_host` is provided, it will replace the `remote_host` which 
wasndefined in `ssh_hook` or predefined in the connection of `ssh_conn_id`
   
   These should be documented clearly to ensure it's transparent to the users.
   
   log.info() should also be used to remind users and provide clear logs.
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Arg checking & better doc for SSHOperator and SFTPOperator
> --
>
> Key: AIRFLOW-2948
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2948
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: operators
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Minor
>
> There may be different combinations of arguments, and some processings are 
> being done 'silently', while users may not be fully aware of them.
>  
>  For example
>  - User only needs to provide either `ssh_hook` or `ssh_conn_id`, while this 
> is not clear in doc
>  - if both provided, `ssh_conn_id` will be ignored.
>  - if `remote_host` is provided, it will replace the `remote_host` which 
> wasndefined in `ssh_hook` or predefined in the connection of `ssh_conn_id`
>  
>  These should be documented clearly to ensure it's transparent to the users.
>  
>  log.info() should also be used to remind users and provide clear logs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] XD-DENG opened a new pull request #3793: [AIRFLOW-2948] Arg check & better doc - SSHOperator & SFTPOperator

[GitBox]

XD-DENG opened a new pull request #3793: [AIRFLOW-2948] Arg check & better doc 
- SSHOperator & SFTPOperator
URL: https://github.com/apache/incubator-airflow/pull/3793
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-2948
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   There may be different combinations of arguments, and some processings are 
being done 'silently', while users may not be fully aware of them.
   
   For example
   - User only needs to provide either `ssh_hook` or `ssh_conn_id`, while this 
is not clear in doc
   - if both provided, `ssh_conn_id` will be ignored.
   - if `remote_host` is provided, it will replace the `remote_host` which 
wasndefined in `ssh_hook` or predefined in the connection of `ssh_conn_id`
   
   These should be documented clearly to ensure it's transparent to the users.
   
   log.info() should also be used to remind users and provide clear logs.
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-2948) Arg checking & better doc for SSHOperator and SFTPOperator

[Xiaodong DENG (JIRA)]

Xiaodong DENG created AIRFLOW-2948:
--

 Summary: Arg checking & better doc for SSHOperator and SFTPOperator
 Key: AIRFLOW-2948
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2948
 Project: Apache Airflow
  Issue Type: Improvement
  Components: operators
Reporter: Xiaodong DENG
Assignee: Xiaodong DENG


There may be different combinations of arguments, and some processings are 
being done 'silently', while users may not be fully aware of them.
 
 For example
 - User only needs to provide either `ssh_hook` or `ssh_conn_id`, while this is 
not clear in doc
 - if both provided, `ssh_conn_id` will be ignored.
 - if `remote_host` is provided, it will replace the `remote_host` which 
wasndefined in `ssh_hook` or predefined in the connection of `ssh_conn_id`
 
 These should be documented clearly to ensure it's transparent to the users.
 
 log.info() should also be used to remind users and provide clear logs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] dimberman commented on issue #3792: [AIRFLOW-2947] Added Helm chart for Kubernetes executor

[GitBox]

dimberman commented on issue #3792: [AIRFLOW-2947] Added Helm chart for 
Kubernetes executor
URL: 
https://github.com/apache/incubator-airflow/pull/3792#issuecomment-415448465
 
 
   @ashb I agree that helms super-user tiller is a really bad security model
   but there are best practices users can take to use it safely. It's also the
   lowest barrier to entry and a lot of people in the k8s world have been
   asking for a chart.
   
   Also worth mentioning that helm 3 will completely remove the tiller.
   
   https://sweetcode.io/a-first-look-at-the-helm-3-plan/
   
   On Thu, Aug 23, 2018, 7:47 AM Jon Davies  wrote:
   
   > @ashb  That's not a hard-requirement from Helm,
   > it's completely possible to deploy Tiller in a namespace.
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or mute the thread
   > 

   > .
   >
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ashb commented on issue #3792: [AIRFLOW-2947] Added Helm chart for Kubernetes executor

[GitBox]

ashb commented on issue #3792: [AIRFLOW-2947] Added Helm chart for Kubernetes 
executor
URL: 
https://github.com/apache/incubator-airflow/pull/3792#issuecomment-415446816
 
 
   I'm also not a fan of that model as it's a resource hog ;)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] jpds edited a comment on issue #3792: [AIRFLOW-2947] Added Helm chart for Kubernetes executor

[GitBox]

jpds edited a comment on issue #3792: [AIRFLOW-2947] Added Helm chart for 
Kubernetes executor
URL: 
https://github.com/apache/incubator-airflow/pull/3792#issuecomment-415445252
 
 
   @ashb That's not a hard-requirement from Helm, it's completely possible to 
deploy Tiller in a namespace with only access to that namespace.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] jpds commented on issue #3792: [AIRFLOW-2947] Added Helm chart for Kubernetes executor

[GitBox]

jpds commented on issue #3792: [AIRFLOW-2947] Added Helm chart for Kubernetes 
executor
URL: 
https://github.com/apache/incubator-airflow/pull/3792#issuecomment-415445252
 
 
   @ashb That's not a hard-requirement from Helm, it's completely possible to 
deploy Tiller in a namespace.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ashb commented on issue #3792: [AIRFLOW-2947] Added Helm chart for Kubernetes executor

[GitBox]

ashb commented on issue #3792: [AIRFLOW-2947] Added Helm chart for Kubernetes 
executor
URL: 
https://github.com/apache/incubator-airflow/pull/3792#issuecomment-415444761
 
 
   This is a -1 from me - I don't like Helm's requirement to be run as an 
all-powerful user in the cluster.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] jpds commented on issue #3792: [AIRFLOW-2947] Added Helm chart for Kubernetes executor

[GitBox]

jpds commented on issue #3792: [AIRFLOW-2947] Added Helm chart for Kubernetes 
executor
URL: 
https://github.com/apache/incubator-airflow/pull/3792#issuecomment-415443337
 
 
   @dimberman @gsemet


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2947) Example Kubernetes manifests should be a Helm chart

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590329#comment-16590329
 ] 

ASF GitHub Bot commented on AIRFLOW-2947:
-

jpds opened a new pull request #3792: [AIRFLOW-2947] Added Helm chart for 
Kubernetes executor
URL: https://github.com/apache/incubator-airflow/pull/3792
 
 
   ### Jira
   
   - [X] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-2947
   
   ### Description
   
   - [X] Here are some details about my PR, including screenshots of any UI 
changes:
   
   Added Helm chart for Airflow.
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   This probably should run helm lint, happy to add it if someone can show me 
how to integrate it into Travis CI.
   
   ### Commits
   
   - [X] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Example Kubernetes manifests should be a Helm chart
> ---
>
> Key: AIRFLOW-2947
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2947
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Jon Davies
>Assignee: Jon Davies
>Priority: Major
>
> The files under scripts/ci/kubernetes/kube/ are manifests and not an actual 
> Helm chart.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] jpds opened a new pull request #3792: [AIRFLOW-2947] Added Helm chart for Kubernetes executor

[GitBox]

jpds opened a new pull request #3792: [AIRFLOW-2947] Added Helm chart for 
Kubernetes executor
URL: https://github.com/apache/incubator-airflow/pull/3792
 
 
   ### Jira
   
   - [X] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-2947
   
   ### Description
   
   - [X] Here are some details about my PR, including screenshots of any UI 
changes:
   
   Added Helm chart for Airflow.
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   This probably should run helm lint, happy to add it if someone can show me 
how to integrate it into Travis CI.
   
   ### Commits
   
   - [X] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-2947) Example Kubernetes manifests should be a Helm chart

[Jon Davies (JIRA)]

Jon Davies created AIRFLOW-2947:
---

 Summary: Example Kubernetes manifests should be a Helm chart
 Key: AIRFLOW-2947
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2947
 Project: Apache Airflow
  Issue Type: Bug
Reporter: Jon Davies
Assignee: Jon Davies


The files under scripts/ci/kubernetes/kube/ are manifests and not an actual 
Helm chart.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nathadfield opened a new pull request #3791: Adding King.com to the list of companies.

[GitBox]

nathadfield opened a new pull request #3791: Adding King.com to the list of 
companies.
URL: https://github.com/apache/incubator-airflow/pull/3791
 
 
   Adding King.com to the list of Airflow users.
   
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] msumit closed pull request #3789: Add Arquivei to companies list

[GitBox]

msumit closed pull request #3789: Add Arquivei to companies list
URL: https://github.com/apache/incubator-airflow/pull/3789
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/README.md b/README.md
index 11e25a7bdf..34296c1cd3 100644
--- a/README.md
+++ b/README.md
@@ -102,6 +102,7 @@ Currently **officially** using Airflow:
 1. [AltX](https://www.getaltx.com/about) 
[[@pedromduarte](https://github.com/pedromduarte)]
 1. [Apigee](https://apigee.com) [[@btallman](https://github.com/btallman)]
 1. [ARGO Labs](http://www.argolabs.org) [[California Data 
Collaborative](https://github.com/California-Data-Collaborative)]
+1. [Arquivei](https://www.arquivei.com.br/) 
[[@arquivei](https://github.com/arquivei)]
 1. [Astronomer](http://www.astronomer.io) 
[[@schnie](https://github.com/schnie), 
[@andscoop](https://github.com/andscoop), 
[@tedmiston](https://github.com/tedmiston), 
[@benjamingregory](https://github.com/benjamingregory)]
 1. [Auth0](https://auth0.com) [[@sicarul](https://github.com/sicarul)]
 1. [Away](https://awaytravel.com) [[@trunsky](https://github.com/trunsky)]


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-2946) Connection times out on airflow worker

[Avik Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Avik Aggarwal updated AIRFLOW-2946:
---
Description: 
Hi 

I have Airflow cluster setup running Celery executors with Postgresql installed 
on same machine as webserver and scheduler.

After sometime, remote worker shows error 'Connection timed out' and Airflow 
queues number of configured tasks in pool in queue and flow hungs up there 
until queue tasks are deleted manually after stopping the scheduler service.

 

Logs:

[2018-08-23 13:44:03,954: ERROR/MainProcess] Pool callback raised exception: 
OperationalError('(psycopg2.OperationalError) could not connect to server: 
Connection timed out\n\tIs the server running on host "" and 
accepting\n\tTCP/IP connections on port 5432?\n',)
 Traceback (most recent call last):
 File "/home/ubuntu/.local/lib/python2.7/site-packages/billiard/pool.py", line 
1747, in safe_apply_callback
 fun(*args, **kwargs)
 File 
"/home/ubuntu/.local/lib/python2.7/site-packages/celery/worker/request.py", 
line 367, in on_failure
 self.id, exc, request=self, store_result=self.store_errors,
 File 
"/home/ubuntu/.local/lib/python2.7/site-packages/celery/backends/base.py", line 
157, in mark_as_failure
 traceback=traceback, request=request)
 File 
"/home/ubuntu/.local/lib/python2.7/site-packages/celery/backends/base.py", line 
322, in store_result
 request=request, **kwargs)
 File 
"/home/ubuntu/.local/lib/python2.7/site-packages/celery/backends/database/__init__.py",
 line 53, in _inner
 return fun(*args, **kwargs)
 File 
"/home/ubuntu/.local/lib/python2.7/site-packages/celery/backends/database/__init__.py",
 line 105, in _store_result
 session = self.ResultSession()
 File 
"/home/ubuntu/.local/lib/python2.7/site-packages/celery/backends/database/__init__.py",
 line 99, in ResultSession
 **self.engine_options)
 File 
"/home/ubuntu/.local/lib/python2.7/site-packages/celery/backends/database/session.py",
 line 60, in session_factory
 self.prepare_models(engine)
 File 
"/home/ubuntu/.local/lib/python2.7/site-packages/celery/backends/database/session.py",
 line 55, in prepare_models
 ResultModelBase.metadata.create_all(engine)
 File 
"/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", 
line 4005, in create_all
 tables=tables)
 File 
"/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", 
line 1939, in _run_visitor
 with self._optional_conn_ctx_manager(connection) as conn:
 File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
 return self.gen.next()
 File 
"/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", 
line 1932, in _optional_conn_ctx_manager
 with self.contextual_connect() as conn:
 File 
"/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", 
line 2123, in contextual_connect
 self._wrap_pool_connect(self.pool.connect, None),
 File 
"/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", 
line 2162, in _wrap_pool_connect
 e, dialect, self)
 File 
"/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", 
line 1476, in _handle_dbapi_exception_noconnection
 exc_info
 File 
"/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", 
line 265, in raise_from_cause
 reraise(type(exception), exception, tb=exc_tb, cause=cause)
 File 
"/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", 
line 2158, in _wrap_pool_connect
 return fn()
 File "/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/pool.py", 
line 403, in connect
 return _ConnectionFairy._checkout(self)
 File "/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/pool.py", 
line 791, in _checkout
 fairy = _ConnectionRecord.checkout(pool)
 File "/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/pool.py", 
line 532, in checkout
 rec = pool._do_get()
 File "/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/pool.py", 
line 1287, in _do_get
 return self._create_connection()
 File "/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/pool.py", 
line 350, in _create_connection
 return _ConnectionRecord(self)
 File "/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/pool.py", 
line 477, in __init__
 self.__connect(first_connect_check=True)
 File "/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/pool.py", 
line 674, in __connect
 connection = pool._invoke_creator(self)
 File 
"/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py",
 line 106, in connect
 return dialect.connect(*cargs, **cparams)
 File 
"/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", 
line 412, in connect
 return self.dbapi.connect(*cargs, **cparams)
 File "/home/ubuntu/.local/lib/python2.7/site-packages/psycopg2/__init__.py", 
line 130, in connect
 conn = _connect(dsn, connection_factory=connection_factory, **kwasync)

[jira] [Created] (AIRFLOW-2946) Connection times out on airflow worker

[Avik Aggarwal (JIRA)]

Avik Aggarwal created AIRFLOW-2946:
--

 Summary: Connection times out on airflow worker
 Key: AIRFLOW-2946
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2946
 Project: Apache Airflow
  Issue Type: Bug
  Components: celery, executor, worker
Affects Versions: 1.10.0
 Environment: ubuntu 16.04, AWS EC2 
Reporter: Avik Aggarwal


Hi 

I have Airflow cluster setup running Celery executors with Postgresql installed 
on same machine as webserver and scheduler.

After sometime, remote worker shows error 'Connection timed out' and Airflow 
queues number of configured tasks in pool in queue and flow hungs up there 
until queue tasks are deleted manually after stopping the scheduler service.

 

Logs:

[2018-08-23 13:44:03,954: ERROR/MainProcess] Pool callback raised exception: 
OperationalError('(psycopg2.OperationalError) could not connect to server: 
Connection timed out\n\tIs the server running on host "34.232.109.233" and 
accepting\n\tTCP/IP connections on port 5432?\n',)
Traceback (most recent call last):
 File "/home/ubuntu/.local/lib/python2.7/site-packages/billiard/pool.py", line 
1747, in safe_apply_callback
 fun(*args, **kwargs)
 File 
"/home/ubuntu/.local/lib/python2.7/site-packages/celery/worker/request.py", 
line 367, in on_failure
 self.id, exc, request=self, store_result=self.store_errors,
 File 
"/home/ubuntu/.local/lib/python2.7/site-packages/celery/backends/base.py", line 
157, in mark_as_failure
 traceback=traceback, request=request)
 File 
"/home/ubuntu/.local/lib/python2.7/site-packages/celery/backends/base.py", line 
322, in store_result
 request=request, **kwargs)
 File 
"/home/ubuntu/.local/lib/python2.7/site-packages/celery/backends/database/__init__.py",
 line 53, in _inner
 return fun(*args, **kwargs)
 File 
"/home/ubuntu/.local/lib/python2.7/site-packages/celery/backends/database/__init__.py",
 line 105, in _store_result
 session = self.ResultSession()
 File 
"/home/ubuntu/.local/lib/python2.7/site-packages/celery/backends/database/__init__.py",
 line 99, in ResultSession
 **self.engine_options)
 File 
"/home/ubuntu/.local/lib/python2.7/site-packages/celery/backends/database/session.py",
 line 60, in session_factory
 self.prepare_models(engine)
 File 
"/home/ubuntu/.local/lib/python2.7/site-packages/celery/backends/database/session.py",
 line 55, in prepare_models
 ResultModelBase.metadata.create_all(engine)
 File 
"/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", 
line 4005, in create_all
 tables=tables)
 File 
"/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", 
line 1939, in _run_visitor
 with self._optional_conn_ctx_manager(connection) as conn:
 File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
 return self.gen.next()
 File 
"/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", 
line 1932, in _optional_conn_ctx_manager
 with self.contextual_connect() as conn:
 File 
"/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", 
line 2123, in contextual_connect
 self._wrap_pool_connect(self.pool.connect, None),
 File 
"/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", 
line 2162, in _wrap_pool_connect
 e, dialect, self)
 File 
"/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", 
line 1476, in _handle_dbapi_exception_noconnection
 exc_info
 File 
"/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", 
line 265, in raise_from_cause
 reraise(type(exception), exception, tb=exc_tb, cause=cause)
 File 
"/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", 
line 2158, in _wrap_pool_connect
 return fn()
 File "/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/pool.py", 
line 403, in connect
 return _ConnectionFairy._checkout(self)
 File "/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/pool.py", 
line 791, in _checkout
 fairy = _ConnectionRecord.checkout(pool)
 File "/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/pool.py", 
line 532, in checkout
 rec = pool._do_get()
 File "/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/pool.py", 
line 1287, in _do_get
 return self._create_connection()
 File "/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/pool.py", 
line 350, in _create_connection
 return _ConnectionRecord(self)
 File "/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/pool.py", 
line 477, in __init__
 self.__connect(first_connect_check=True)
 File "/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/pool.py", 
line 674, in __connect
 connection = pool._invoke_creator(self)
 File 
"/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py",
 line 106, in connect
 return dialect.connect(*cargs, **cparams)
 File 

[GitHub] sakshi2894 opened a new pull request #3790: [AIRFLOW-2994] Fix command status check in Qubole Check operator

[GitBox]

sakshi2894 opened a new pull request #3790: [AIRFLOW-2994] Fix command status 
check in Qubole Check operator
URL: https://github.com/apache/incubator-airflow/pull/3790
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-2994\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-2994
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   Exception message shouldn't contain Qubole command results if the command 
fails. This PR fixes a bug which causes this behaviour.
   
   ### Tests
   
   - [ ] My PR adds the following unit tests:
   `test_execute_assert_query_fail` - Tests that the exception message doesn't 
contain Qubole command id and results if the command has failed.
   
   
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] leonardoam opened a new pull request #3789: Add Arquivei to companies list

[GitBox]

leonardoam opened a new pull request #3789: Add Arquivei to companies list
URL: https://github.com/apache/incubator-airflow/pull/3789
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (AIRFLOW-2859) DateTimes returned from the database are not converted to UTC

[Ash Berlin-Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor resolved AIRFLOW-2859.

Resolution: Fixed

> DateTimes returned from the database are not converted to UTC
> -
>
> Key: AIRFLOW-2859
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2859
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: database
>Reporter: Bolke de Bruin
>Priority: Blocker
> Fix For: 1.10.0
>
>
> This is due to the fact that sqlalchemy-utcdatetime does not convert to UTC 
> when the database returns datetimes with tzinfo.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (AIRFLOW-1748) Task is not being scheduled daily as expected in Airflow

[Ash Berlin-Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor closed AIRFLOW-1748.
--
Resolution: Cannot Reproduce

Can't reproduce this - if you are still having issues a better channel for 
support is the Gitter chat, or the Airflow mailing list.

> Task is not being scheduled daily as expected in Airflow
> 
>
> Key: AIRFLOW-1748
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1748
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DagRun
> Environment: v1.8.0
>Reporter: Di Zhu
>Priority: Major
>
> A dag python file is defined as below:
> ```
> default_args = {
> 'owner': 'airflow',
> 'depends_on_past': False,
> 'start_date': datetime(2017, 10, 22, 11, 25),
> 'email': ['airf...@airflow.com'],
> 'email_on_failure': False,
> 'email_on_retry': False,
> 'retries': 0,
> 'retry_delay': timedelta(minutes=1),
> # 'queue': 'bash_queue',
> # 'pool': 'backfill',
> # 'priority_weight': 10,
> # 'end_date': datetime(2016, 1, 1),
> }
> dag = DAG('test_jason_1', default_args=default_args, 
> schedule_interval="@daily")
> # t1, t2 and t3 are examples of tasks created by instantiating operators
> t1 = BashOperator(
> task_id='print_date',
> bash_command='echo "123" && exit 1',
> dag=dag)```
> Current datetime from command line (same machine as where Airflow is 
> installed) is:
> ```
> [ec2-user@ip-10-0-0-XXX print_date]$ date
> Mon Oct 23 11:24:06 UTC 2017
> ```
> but after 11:25:00, this job is still not scheduled accordingly. Could anyone 
> help correct me if i'm wrong? Thanks a lot!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (AIRFLOW-1926) Mouseover on icons

[Ash Berlin-Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor closed AIRFLOW-1926.
--
Resolution: Duplicate

> Mouseover on icons
> --
>
> Key: AIRFLOW-1926
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1926
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Søren Holbech
>Priority: Minor
>
> I am learning to maintain a DAG using the airflow UI, and the meaning of 
> several of the miniature icons is not obvious to me. It would be nice to have 
> a mouseover on them - as it is now I have to click them to see what they do, 
> and sometimes that causes a lot of problems (such as when I launched a second 
> copy of a running DAG).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AIRFLOW-2867) Airflow Python Code not compatible to coding guidelines and standards

[Ash Berlin-Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor resolved AIRFLOW-2867.

Resolution: Fixed

> Airflow Python Code not compatible to coding guidelines and standards 
> --
>
> Key: AIRFLOW-2867
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2867
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Minor
> Fix For: 2.0.0
>
>
> Some of the Airflow code doesn't conform to python coding guidelines and 
> standards.
> The improvement I have analyzed are below:
> - Dictionary creation should be written by dictionary literal
> - Mutable default argument. Python’s default arguments are evaluated once 
> when the function is defined, not each time the function is called (like it 
> is in say, Ruby). This means that if you use a mutable default argument and 
> mutate it, you will and have mutated that object for all future calls to the 
> function as well.
> - Functions calling sets can be replaced by set literal 
> - Replace list literals
> - Some of the static methods haven't been set static
> - Redundant parentheses



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (AIRFLOW-526) Pin all dependencies versions in setup.py

[Ash Berlin-Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor closed AIRFLOW-526.
-
Resolution: Won't Fix

Closing as WontFix based on discussion on github

> Pin all dependencies versions in setup.py
> -
>
> Key: AIRFLOW-526
> URL: https://issues.apache.org/jira/browse/AIRFLOW-526
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Maxime Beauchemin
>Priority: Major
>
> [AIRFLOW-526] pin all dependencies in setup.py
> Pinning all dependencies in setup.py with help having more homogenous
> installs across environments. The current approach of allowing version
> ranges based on version notation is risky at best and results in bad
> surprises.
> The downside is that we won't get positive bug fixes and security fixes
> for free, but these come at a high cost where a single bad version of a
> package may take your Airflow environment down, without warning.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AIRFLOW-1565) Incompatibility with Flask-Login-0.4.0

[Ash Berlin-Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor resolved AIRFLOW-1565.

   Resolution: Fixed
Fix Version/s: 1.9.0

> Incompatibility with Flask-Login-0.4.0
> --
>
> Key: AIRFLOW-1565
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1565
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.8.2
>Reporter: dud
>Priority: Minor
> Fix For: 1.9.0
>
>
> Hello
> I've notice that Airflow v1.8.2 isn't playing well with flask_login v0.4.0 :
> {code}
> $ pip install --user flask_login==0.4.0
> Collecting flask_login==0.4.0
>   Using cached Flask_Login-0.4.0-py2.py3-none-any.whl
> Requirement already satisfied: Flask in ./.local/lib/python2.7/site-packages 
> (from flask_login==0.4.0)
> Requirement already satisfied: itsdangerous>=0.21 in 
> ./.local/lib/python2.7/site-packages (from Flask->flask_login==0.4.0)
> Requirement already satisfied: Jinja2>=2.4 in 
> ./.local/lib/python2.7/site-packages (from Flask->flask_login==0.4.0)
> Requirement already satisfied: Werkzeug>=0.7 in 
> ./.local/lib/python2.7/site-packages (from Flask->flask_login==0.4.0)
> Requirement already satisfied: click>=2.0 in 
> ./.local/lib/python2.7/site-packages (from Flask->flask_login==0.4.0)
> Requirement already satisfied: MarkupSafe>=0.23 in 
> ./.local/lib/python2.7/site-packages (from 
> Jinja2>=2.4->Flask->flask_login==0.4.0)
> Installing collected packages: flask-login
>   Found existing installation: Flask-Login 0.3.2
> Uninstalling Flask-Login-0.3.2:
>   Successfully uninstalled Flask-Login-0.3.2
> Successfully installed flask-login-0.4.0
> $ airflow webserver
> [2017-09-05 14:30:50,130] {dag_processing.py:627} INFO - Started a process 
> (PID: 22751) to generate tasks for 
> /home/airflow/dags/metadsp/APB_process_table/airflow.py - logging into 
> /var/log/airflow/scheduler/2017-09-05
> [2017-09-05 14:30:50,133] {dag_processing.py:627} INFO - Started a process 
> (PID: 22752) to generate tasks for 
> /home/airflow/dags/metadsp/APN_segment_manager/segment_manager_factory.py - 
> logging into /var/log/airflow/sc
> [2017-09-05 14:30:50,134] {jobs.py:1443} INFO - Heartbeating the executor
> 2017-09-05 14:30:50,192] {__init__.py:57} INFO - Using executor CeleryExecutor
> [2017-09-05 14:30:50,252] {driver.py:120} INFO - Generating grammar tables 
> from /usr/lib/python2.7/lib2to3/Grammar.txt
> [2017-09-05 14:30:50,273] {driver.py:120} INFO - Generating grammar tables 
> from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
> /home/airflow/.local/lib/python2.7/site-packages/airflow/www/app.py:23: 
> FlaskWTFDeprecationWarning: "flask_wtf.CsrfProtect" has been renamed to 
> "CSRFProtect" and will be removed in 1.0.
> csrf = CsrfProtect()
> /home/airflow/.local/lib/python2.7/site-packages/flask_cache/jinja2ext.py:33: 
> ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use 
> flask_cache instead.
> from flask.ext.cache import make_template_fragment_key
>    _
> |__( )_  __/__  /  __
>   /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
> ___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
> _/_/  |_/_/  /_//_//_/  \//|__/
> Traceback (most recent call last):
> File "/home/airflow/bin/airflow", line 28, in 
> args.func(args)
> File "/home/airflow/.local/lib/python2.7/site-packages/airflow/bin/cli.py", 
> line 734, in webserver
> app = cached_app(conf)
> File "/home/airflow/.local/lib/python2.7/site-packages/airflow/www/app.py", 
> line 161, in cached_app
> app = create_app(config)
> File "/home/airflow/.local/lib/python2.7/site-packages/airflow/www/app.py", 
> line 60, in create_app
> from airflow.www import views
> File "/home/airflow/.local/lib/python2.7/site-packages/airflow/www/views.py", 
> line 41, in 
> from flask_login import flash
> ImportError: cannot import name flash
> $ python -c "from flask_login import flash"
> Traceback (most recent call last):
>   File "", line 1, in 
> ImportError: cannot import name flash
> {code}
> -Downgrading to flask_login==0.3.2 fixes the issue.-
> EDIT : Downgrading to flask_login==0.2.11 fixes the issue.
> Due to :
> {quote}
> Version 0.3.0
> -
> Released on September 10th, 2015
> [...]
> - BREAKING: The `is_authenticated`, `is_active`, and `is_anonymous` members 
> of the user class are now properties, not methods. Applications should update 
> their user classes accordingly.
> {quote}
> Airflow isn't working with flask_login >= v0.3.0 either.
> dud



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] wrp commented on a change in pull request #3785: [AIRFLOW-2938] Handle improperly formatted extra field in connection …

[GitBox]

wrp commented on a change in pull request #3785: [AIRFLOW-2938] Handle 
improperly formatted extra field in connection …
URL: https://github.com/apache/incubator-airflow/pull/3785#discussion_r212284424
 
 

 ##
 File path: airflow/www_rbac/views.py
 ##
 @@ -1923,7 +1923,10 @@ def prefill_form(self, form, pk):
 d = {}
 
 for field in self.extra_fields:
-value = d.get(field, '')
+try:
+value = d.get(field, '')
+except Exception:
 
 Review comment:
   Yes.  I'm not sure if we should log something here, but once in that state 
it becomes difficult to edit the extra field from the UI


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ashb closed pull request #3788: [AIRFLOW-XXX] Replaces incorrect env var name in INSTALL

[GitBox]

ashb closed pull request #3788: [AIRFLOW-XXX] Replaces incorrect env var name 
in INSTALL
URL: https://github.com/apache/incubator-airflow/pull/3788
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/INSTALL b/INSTALL
index 596ce25814..b018839ab1 100644
--- a/INSTALL
+++ b/INSTALL
@@ -20,7 +20,7 @@ source my_env/bin/activate
 # To make sure not to install the GPL dependency:
 #   export SLUGIFY_USES_TEXT_UNIDECODE=yes
 # In case you do not mind:
-#   export GPL_UNIDECODE=yes
+#   export AIRFLOW_GPL_UNIDECODE=yes
 
 # [required] building and installing
 # by pip (preferred)


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] PointyShinyBurning opened a new pull request #3788: [AIRFLOW-XXX] Replaces incorrect env var name in INSTALL

[GitBox]

PointyShinyBurning opened a new pull request #3788: [AIRFLOW-XXX] Replaces 
incorrect env var name in INSTALL
URL: https://github.com/apache/incubator-airflow/pull/3788
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [ x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   Fixes one-word mistake in INSTALL
   
   ### Tests
   
   - [ x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   Docs change
   ### Commits
   
   - [x ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [ x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-987) `airflow kerberos` ignores --keytab and --principal arguments

[jack (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590091#comment-16590091
 ] 

jack commented on AIRFLOW-987:
--

[~bolke] is it still an issue?

> `airflow kerberos` ignores --keytab and --principal arguments
> -
>
> Key: AIRFLOW-987
> URL: https://issues.apache.org/jira/browse/AIRFLOW-987
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: security
>Affects Versions: Airflow 1.8
> Environment: 1.8-rc5
>Reporter: Ruslan Dautkhanov
>Assignee: Bolke de Bruin
>Priority: Major
>  Labels: easyfix, kerberos, security
>
> No matter which arguments I pass to `airflow kerberos`, 
> it always executes as `kinit -r 3600m -k -t airflow.keytab -c 
> /tmp/airflow_krb5_ccache airflow`
> So it failes with expected "kinit: Keytab contains no suitable keys for 
> airf...@corp.some.com while getting initial credentials"
> Tried different arguments, -kt and --keytab, here's one of the runs (some 
> lines wrapped for readability):
> {noformat}
> $ airflow kerberos -kt /home/rdautkha/.keytab rdautkha...@corp.some.com
> [2017-03-14 23:50:11,523] {__init__.py:57} INFO - Using executor LocalExecutor
> [2017-03-14 23:50:12,069] {kerberos.py:43} INFO - Reinitting kerberos from 
> keytab: 
> kinit -r 3600m -k -t airflow.keytab -c /tmp/airflow_krb5_ccache airflow
> [2017-03-14 23:50:12,080] {kerberos.py:55} ERROR -
>  Couldn't reinit from keytab! `kinit' exited with 1.
> kinit: Keytab contains no suitable keys for airf...@corp.some.com 
> while getting initial credentials
> {noformat}
> 1.8-rc5



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-438) "airflow initdb" not working

[jack (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590076#comment-16590076
 ] 

jack commented on AIRFLOW-438:
--

There shouldn't be any forced install of packages just because a specific 
example which doesn't even have to be relevant to the user requires this import.

> "airflow initdb" not working
> 
>
> Key: AIRFLOW-438
> URL: https://issues.apache.org/jira/browse/AIRFLOW-438
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Andreas Jung
>Priority: Major
>
> I am receiving this error with a fresh Airflow installation
>  bin/airflow  initdb
> [2016-08-17 15:29:23,831] {__init__.py:36} INFO - Using executor 
> SequentialExecutor
> DB: sqlite:home/ajung/airflow/airflow.db
> [2016-08-17 15:29:24,179] {db.py:222} INFO - Creating tables
> INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
> INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
> ERROR [airflow.models.DagBag] Failed to import: 
> /home/ajung/airflow/lib/python2.7/site-packages/airflow/example_dags/example_twitter_dag.py
> Traceback (most recent call last):
>   File "/home/ajung/airflow/lib/python2.7/site-packages/airflow/models.py", 
> line 247, in process_file
> m = imp.load_source(mod_name, filepath)
>   File 
> "/home/ajung/airflow/lib/python2.7/site-packages/airflow/example_dags/example_twitter_dag.py",
>  line 26, in 
> from airflow.operators import BashOperator, HiveOperator, PythonOperator
> ImportError: cannot import name HiveOperator
> airflow==1.7.1.3
> alembic==0.8.7
> Babel==1.3
> chartkick==0.4.2
> croniter==0.3.12
> dill==0.2.5
> docutils==0.12
> Flask==0.10.1
> Flask-Admin==1.4.0
> Flask-Cache==0.13.1
> Flask-Login==0.2.11
> Flask-WTF==0.12
> funcsigs==0.4
> future==0.15.2
> gunicorn==19.3.0
> itsdangerous==0.24
> Jinja2==2.8
> lockfile==0.12.2
> Mako==1.0.4
> Markdown==2.6.6
> MarkupSafe==0.23
> numpy==1.11.1
> pandas==0.18.1
> Pygments==2.1.3
> python-daemon==2.1.1
> python-dateutil==2.5.3
> python-editor==1.0.1
> pytz==2016.6.1
> requests==2.11.0
> setproctitle==1.1.10
> six==1.10.0
> SQLAlchemy==1.0.14
> thrift==0.9.3
> Werkzeug==0.11.10
> WTForms==2.1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] bolkedebruin commented on a change in pull request #3740: [AIRFLOW-2888] Remove shell=True and bash from task launch

[GitBox]

bolkedebruin commented on a change in pull request #3740: [AIRFLOW-2888] Remove 
shell=True and bash from task launch
URL: https://github.com/apache/incubator-airflow/pull/3740#discussion_r212264469
 
 

 ##
 File path: airflow/executors/celery_executor.py
 ##
 @@ -84,7 +84,7 @@ def execute_async(self, key, command,
 self.log.info("[celery] queuing {key} through celery, "
   "queue={queue}".format(**locals()))
 self.tasks[key] = execute_command.apply_async(
-args=[command], queue=queue)
+args=command, queue=queue)
 
 Review comment:
   Good catch! I do think renaming the parmeter is the right way. 
   
   Update: strike that got why it happens with celery


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-627) Tasks getting Queued when Pool is full sometimes never run

[jack (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590072#comment-16590072
 ] 

jack commented on AIRFLOW-627:
--

[~btall...@gmail.com]

What do you expect to happen when the pool is full?

> Tasks getting Queued when Pool is full sometimes never run
> --
>
> Key: AIRFLOW-627
> URL: https://issues.apache.org/jira/browse/AIRFLOW-627
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: scheduler
>Affects Versions: Airflow 1.8
> Environment: Celery Executor, Master Branch, Postgres
>Reporter: Ben Tallman
>Priority: Major
>
> Log data when this happens:
> [2016-11-14 10:54:04,174] {models.py:168} INFO - Filling up the DagBag from 
> /opt/efs/airflow/dags/crawl_traffic_prod.py
> [2016-11-14 10:54:07,562] {base_hook.py:67} INFO - Using connection to: 
> db.xyz.com
> [2016-11-14 10:54:07,667] {base_hook.py:67} INFO - Using connection to: 
> db.xyz.com
> [2016-11-14 10:54:27,214] {models.py:168} INFO - Filling up the DagBag from 
> /opt/efs/airflow/dags/crawl_traffic_prod.py
> [2016-11-14 10:54:30,150] {base_hook.py:67} INFO - Using connection to: 
> db.xyz.com
> [2016-11-14 10:54:30,311] {base_hook.py:67} INFO - Using connection to: 
> db.xyz.com
> [2016-11-14 10:54:32,438] {models.py:1072} INFO - Dependencies all met for 
>  01:00:00 [queued]>
> [2016-11-14 10:54:32,700] {models.py:1069} WARNING - Dependencies not met for 
>  01:00:00 [queued]>, dependency 'DAG's Pool Has Space' FAILED: Task's pool 
> 'prod_pod_crawler' is full.
> Task Details info:
> depends_on_past   False
> deps  set([, , 
> ])



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] bolkedebruin commented on a change in pull request #3740: [AIRFLOW-2888] Remove shell=True and bash from task launch

[GitBox]

bolkedebruin commented on a change in pull request #3740: [AIRFLOW-2888] Remove 
shell=True and bash from task launch
URL: https://github.com/apache/incubator-airflow/pull/3740#discussion_r212264469
 
 

 ##
 File path: airflow/executors/celery_executor.py
 ##
 @@ -84,7 +84,7 @@ def execute_async(self, key, command,
 self.log.info("[celery] queuing {key} through celery, "
   "queue={queue}".format(**locals()))
 self.tasks[key] = execute_command.apply_async(
-args=[command], queue=queue)
+args=command, queue=queue)
 
 Review comment:
   Good catch! I do think renaming the parmeter is the right way 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2945) Web UI blank in 2.0.0+dev0/master

[Ash Berlin-Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590057#comment-16590057
 ] 

Ash Berlin-Taylor commented on AIRFLOW-2945:


Are you using the RBAC UI, if so then you need to follow 
https://github.com/apache/incubator-airflow/blob/master/CONTRIBUTING.md#setting-up-the-node--npm-javascript-environment-only-for-www_rbac

> Web UI blank in 2.0.0+dev0/master
> -
>
> Key: AIRFLOW-2945
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2945
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: webapp
>Affects Versions: 2.0.0
>Reporter: Jon Davies
>Priority: Major
>
> For some reason, every time I use the master branch for my Airflow docker 
> image, I'm presented with a blank web UI for the DAGs page.
> The only suspicious thing I see in the logs is:
> {code:java}
> testing-airflow/elder-seastar-airflow-web-dcd948f56-vglfg[airflow-web]: 
> 172.20.51.87 - - [23/Aug/2018:10:27:13 +] "GET /static/dist/ HTTP/1.1" 
> 404 3717 ...
> {code}
> I don't have this issue if I use the 1.10.0rc2 tag for my branch base.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2945) Web UI blank in 2.0.0+dev0/master

[Jon Davies (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Davies updated AIRFLOW-2945:

Description: 
For some reason, every time I use the master branch for my Airflow docker 
image, I'm presented with a blank web UI for the DAGs page.

The only suspicious thing I see in the logs is:

{code:java}
testing-airflow/elder-seastar-airflow-web-dcd948f56-vglfg[airflow-web]: 
172.20.51.87 - - [23/Aug/2018:10:27:13 +] "GET /static/dist/ HTTP/1.1" 404 
3717 ...
{code}

I don't have this issue if I use the 1.10.0rc2 tag for my branch base.

  was:
For some reason, every time I use the master branch for my Airflow docker 
image, I'm presented with a blank web UI.

The only suspicious thing I see in the logs is:

{code:java}
testing-airflow/elder-seastar-airflow-web-dcd948f56-vglfg[airflow-web]: 
172.20.51.87 - - [23/Aug/2018:10:27:13 +] "GET /static/dist/ HTTP/1.1" 404 
3717 ...
{code}

I don't have this issue if I use the 1.10.0rc2 tag for my branch base.


> Web UI blank in 2.0.0+dev0/master
> -
>
> Key: AIRFLOW-2945
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2945
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: webapp
>Affects Versions: 2.0.0
>Reporter: Jon Davies
>Priority: Major
>
> For some reason, every time I use the master branch for my Airflow docker 
> image, I'm presented with a blank web UI for the DAGs page.
> The only suspicious thing I see in the logs is:
> {code:java}
> testing-airflow/elder-seastar-airflow-web-dcd948f56-vglfg[airflow-web]: 
> 172.20.51.87 - - [23/Aug/2018:10:27:13 +] "GET /static/dist/ HTTP/1.1" 
> 404 3717 ...
> {code}
> I don't have this issue if I use the 1.10.0rc2 tag for my branch base.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2945) Web UI blank in 2.0.0+dev0/master

[Jon Davies (JIRA)]

Jon Davies created AIRFLOW-2945:
---

 Summary: Web UI blank in 2.0.0+dev0/master
 Key: AIRFLOW-2945
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2945
 Project: Apache Airflow
  Issue Type: Bug
  Components: webapp
Affects Versions: 2.0.0
Reporter: Jon Davies


For some reason, every time I use the master branch for my Airflow docker 
image, I'm presented with a blank web UI.

The only suspicious thing I see in the logs is:

{code:java}
testing-airflow/elder-seastar-airflow-web-dcd948f56-vglfg[airflow-web]: 
172.20.51.87 - - [23/Aug/2018:10:27:13 +] "GET /static/dist/ HTTP/1.1" 404 
3717 ...
{code}

I don't have this issue if I use the 1.10.0rc2 tag for my branch base.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2944) Bug in checking command status in Qubole Check operator

[Sakshi Bansal (JIRA)]

Sakshi Bansal created AIRFLOW-2944:
--

 Summary: Bug in checking command status in Qubole Check operator
 Key: AIRFLOW-2944
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2944
 Project: Apache Airflow
  Issue Type: Bug
Reporter: Sakshi Bansal
Assignee: Sakshi Bansal






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2943) example_trigger_target_dag.py Dag is stuck in running state

[Loveleen Kaur (JIRA)]

Loveleen Kaur created AIRFLOW-2943:
--

 Summary: example_trigger_target_dag.py Dag is stuck in running 
state
 Key: AIRFLOW-2943
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2943
 Project: Apache Airflow
  Issue Type: Bug
Affects Versions: Airflow 1.8
 Environment: Python 2.7
Reporter: Loveleen Kaur
 Attachments: airflow issue.PNG

Hi,
I am trying to implement example_trigger_controller_dag.py and 
example_trigger_target_dag.py (according to my requirements)  but the execution 
time of target dag is approximately 7 hours more than the triggered time i.e 
Triggered time is in PDT but the execution time of dag is showing in UTC. These 
sample examples are giving same issue.  Because of this Dag run after 7 hours 
of the triggered time.

In Issue - Apache AirflowAIRFLOW-43 , it is mentioned - 
If you see the dag run created, but not being scheduled, it is likely because 
you are not running in UTC everywhere.

Airflow dags are running in PDT timezone , so to convert that into UTC, i 
updated cfg file with default_timezone = utc and restarted airflow but still 
getting same issue.

Please help. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2942) pypi not updated to 1.10

[Matthias (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590021#comment-16590021
 ] 

Matthias commented on AIRFLOW-2942:
---

Thanks for the very quick response. 

fixed the wrong date

> pypi not updated to 1.10
> 
>
> Key: AIRFLOW-2942
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2942
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.10
>Reporter: Matthias
>Priority: Major
>
> According to 
> [https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements,] airflow 
> 1.10 is out since August 20th. 
> Pypi is still serving 1.9.0 ([https://pypi.org/project/apache-airflow/)] as 
> of today.
>  
> Installing via `pip install apache-airflow==1.10.0` does not work as that 
> release is not available on pypi.
> output:
> ```
>  Collecting apache-airflow==1.10.0
>  Could not find a version that satisfies the requirement 
> apache-airflow==1.10.0 (from versions: 1.8.1, 1.8.2rc1, 1.8.2, 1.9.0)
>  No matching distribution found for apache-airflow==1.10.0
>  ```
>  
> Fix should be trivial - pushing the new release to pypi.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2942) pypi not updated to 1.10

[Matthias (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias updated AIRFLOW-2942:
--
Description: 
According to 
[https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements,] airflow 
1.10 is out since August 20th. 

Pypi is still serving 1.9.0 ([https://pypi.org/project/apache-airflow/)] as of 
today.

 

Installing via `pip install apache-airflow==1.10.0` does not work as that 
release is not available on pypi.

output:

```
 Collecting apache-airflow==1.10.0
 Could not find a version that satisfies the requirement apache-airflow==1.10.0 
(from versions: 1.8.1, 1.8.2rc1, 1.8.2, 1.9.0)
 No matching distribution found for apache-airflow==1.10.0
 ```

 

Fix should be trivial - pushing the new release to pypi.

  was:
According to 
[https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements,] airflow 
1.10 is out since April 20th. 

Pypi is still serving 1.9.0 ([https://pypi.org/project/apache-airflow/)] as of 
today.

 

Installing via `pip install apache-airflow==1.10.0` does not work as that 
release is not available on pypi.

output:

```
 Collecting apache-airflow==1.10.0
 Could not find a version that satisfies the requirement apache-airflow==1.10.0 
(from versions: 1.8.1, 1.8.2rc1, 1.8.2, 1.9.0)
 No matching distribution found for apache-airflow==1.10.0
```

 

Fix should be trivial - pushing the new release to pypi.


> pypi not updated to 1.10
> 
>
> Key: AIRFLOW-2942
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2942
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.10
>Reporter: Matthias
>Priority: Major
>
> According to 
> [https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements,] airflow 
> 1.10 is out since August 20th. 
> Pypi is still serving 1.9.0 ([https://pypi.org/project/apache-airflow/)] as 
> of today.
>  
> Installing via `pip install apache-airflow==1.10.0` does not work as that 
> release is not available on pypi.
> output:
> ```
>  Collecting apache-airflow==1.10.0
>  Could not find a version that satisfies the requirement 
> apache-airflow==1.10.0 (from versions: 1.8.1, 1.8.2rc1, 1.8.2, 1.9.0)
>  No matching distribution found for apache-airflow==1.10.0
>  ```
>  
> Fix should be trivial - pushing the new release to pypi.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)