date:20180810

XD-DENG commented on issue #3724: [AIRFLOW-2878] Fix www_rbac display issue
URL: 
https://github.com/apache/incubator-airflow/pull/3724#issuecomment-412246789
 
 
   Hi @verdan , FYI, this commit is already reverted via 
https://github.com/apache/incubator-airflow/pull/3737 . Thanks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] XD-DENG edited a comment on issue #3738: [AIRFLOW-2886] Secure Flask SECRET_KEY

XD-DENG edited a comment on issue #3738: [AIRFLOW-2886] Secure Flask SECRET_KEY
URL:
https://github.com/apache/incubator-airflow/pull/3738#issuecomment-412227570

Hi @feng-tao , in cluster mode, given the `.cfg` files are initiated on
different nodes, the SECRET_KEY will be different. That's also why Craig
mentioned it fixes now on a single webserver instance (single worker or
multiple workers), but still having `CSRF` error on a cluster of webservers
([LINK](https://issues.apache.org/jira/browse/AIRFLOW-2866?focusedCommentId=16576717=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16576717)).

I insist in that we should have random SECRET_KEY for the webserver. If we
use the previous way, that is to have a default value in the template, it's
gonna be risky.

If users want to use a cluster of webservers, they need to either
- make sure that SECRET_KEYs are consistent across nodes (to manually
specify). This is similar to what people need to do for `sql_alchemy_conn` if
they're running multiple nodes for webserver.
OR
- apply `ip-hash` strategy for the cluster load-balancing, rather than
`round-robin`.

Please let me know your thoughts?

cc: @Fokko @kaxil

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

With regards,
Apache Git Services

[GitHub] kaxil commented on issue #3733: [WIP] [AIRFLOW-491] Add cache parameter in BigQuery query method

kaxil commented on issue #3733: [WIP] [AIRFLOW-491] Add cache parameter in 
BigQuery query method
URL: 
https://github.com/apache/incubator-airflow/pull/3733#issuecomment-412239088
 
 
   Don't remove any previous parameters, just add support to add new ones


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] XD-DENG edited a comment on issue #3738: [AIRFLOW-2886] Secure Flask SECRET_KEY

XD-DENG edited a comment on issue #3738: [AIRFLOW-2886] Secure Flask SECRET_KEY
URL: 
https://github.com/apache/incubator-airflow/pull/3738#issuecomment-412227570
 
 
   Hi @feng-tao , in cluster mode, given the `.cfg` files are initiated on 
different nodes, the SECRET_KEY will be different. That's also why Craig 
mentioned it fixes now on a single webserver instance (single worker or 
multiple workers), but still having `CSRF` error on a cluster of webservers 
([LINK](https://issues.apache.org/jira/browse/AIRFLOW-2866?focusedCommentId=16576717=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16576717)).
   
   I insist in that we should have random SECRET_KEY for the webserver. If we 
use the previous way, that is to have a default value in the template, it's 
gonna be risky.
   
   If users want to use a cluster of webservers, they need to either 
   - make sure that SECRET_KEYs are consistent across nodes (to manually 
specify)
   or
   - apply `ip-hash` strategy for the cluster load-balancing, rather than 
`round-robin`.
   
   Please let me know your thoughts?
   
   cc: @Fokko @kaxil 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] XD-DENG edited a comment on issue #3738: [AIRFLOW-2886] Secure Flask SECRET_KEY

XD-DENG edited a comment on issue #3738: [AIRFLOW-2886] Secure Flask SECRET_KEY
URL: 
https://github.com/apache/incubator-airflow/pull/3738#issuecomment-412227570
 
 
   Hi @feng-tao , in cluster mode, given the `.cfg` files are initiated on 
different nodes, the SECRET_KEY will be different. That's also why Craig 
mentioned it fixes now on a single webserver instance (single worker or 
multiple workers), but doesn't work on a cluster of webservers 
([LINK](https://issues.apache.org/jira/browse/AIRFLOW-2866?focusedCommentId=16576717=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16576717)).
   
   I insist in that we should have random SECRET_KEY for the webserver. If we 
use the previous way, that is to have a default value in the template, it's 
gonna be risky.
   
   If users want to use a cluster of webservers, they need to either 
   - make sure that SECRET_KEYs are consistent across nodes (to manually 
specify)
   or
   - apply `ip-hash` strategy for the cluster load-balancing, rather than 
`round-robin`.
   
   Please let me know your thoughts?
   
   cc: @Fokko @kaxil 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] XD-DENG edited a comment on issue #3738: [AIRFLOW-2886] Secure Flask SECRET_KEY

XD-DENG edited a comment on issue #3738: [AIRFLOW-2886] Secure Flask SECRET_KEY
URL: 
https://github.com/apache/incubator-airflow/pull/3738#issuecomment-412227570
 
 
   Hi @feng-tao , in cluster mode, given the `.cfg` files are initiated on 
different nodes, the SECRET_KEY will be different. That's also why Craig 
mentioned it fixes now on a single webserver instance (multiple workers), but 
doesn't work on a cluster of webservers 
([LINK](https://issues.apache.org/jira/browse/AIRFLOW-2866?focusedCommentId=16576717=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16576717)).
   
   I insist in that we should have random SECRET_KEY for the webserver. If we 
use the previous way, that is to have a default value in the template, it's 
gonna be risky.
   
   If users want to use a cluster of webservers, they need to either 
   - make sure that SECRET_KEYs are consistent across nodes (to manually 
specify)
   or
   - apply `ip-hash` strategy for the cluster load-balancing, rather than 
`round-robin`.
   
   Please let me know your thoughts?
   
   cc: @Fokko @kaxil 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] XD-DENG commented on issue #3738: [AIRFLOW-2886] Secure Flask SECRET_KEY

XD-DENG commented on issue #3738: [AIRFLOW-2886] Secure Flask SECRET_KEY
URL: 
https://github.com/apache/incubator-airflow/pull/3738#issuecomment-412227570
 
 
   Hi @feng-tao , in cluster mode, given the `.cfg` files are initiated on 
different nodes, the SECRET_KEY will be different. That's also why Craig 
mentioned it fixes now on a single webserver instance (multiple workers), but 
doesn't work on a cluster of webservers 
([link](https://issues.apache.org/jira/browse/AIRFLOW-2866?focusedCommentId=16576717=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16576717)).
   
   I insist in that we should have random SECRET_KEY for the webserver. If we 
use the previous way, that is to have a default value in the template, it's 
gonna be risky.
   
   If users want to use a cluster of webservers, they need to either 
   - make sure that SECRET_KEYs are consistent across nodes (to manually 
specify)
   or
   - apply `ip-hash` strategy for the cluster load-balancing.
   
   Please let me know your thoughts?
   
   cc: @Fokko @kaxil 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] amir656 commented on issue #3739: [AIRFLOW-2868] Extend mesos_executor

amir656 commented on issue #3739: [AIRFLOW-2868] Extend mesos_executor
URL: 
https://github.com/apache/incubator-airflow/pull/3739#issuecomment-412219056
 
 
   I am not sure where to add details about this change to the documentation. 
There doesn't seem to be documentation about the executor_config, or even the 
kubernetes executor at all, which I would have hoped to extend. Please let me 
know if you have any ideas about where to add documentation


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-2868) Mesos Executor should use executor_config to specify CPU, Memory and Docker image on the task level

2018-08-10 Thread Amir Shahatit (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amir Shahatit updated AIRFLOW-2868:
---
External issue URL: https://github.com/apache/incubator-airflow/pull/3739

> Mesos Executor should use executor_config to specify CPU, Memory and Docker 
> image on the task level
> ---
>
> Key: AIRFLOW-2868
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2868
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: contrib
>Affects Versions: 1.10, 1.10.1
>Reporter: Amir Shahatit
>Assignee: Amir Shahatit
>Priority: Major
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> Executor_config was added as a part of 
> [AIRFLOW-1314|https://github.com/apache/incubator-airflow/commit/c0920efc012468681cff3d3c9cfe25c7381dc976].
>  This task extends the mesosExecutor to make use of specified executor 
> configs to pass on resource requirements (CPU/Memory) as well as docker 
> images on the task level. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2868) Mesos Executor should use executor_config to specify CPU, Memory and Docker image on the task level



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576881#comment-16576881
 ] 

ASF GitHub Bot commented on AIRFLOW-2868:
-

amir656 opened a new pull request #3739: [AIRFLOW-2868] Extend mesos_executor
URL: https://github.com/apache/incubator-airflow/pull/3739
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following 
[AIRFLOW-2868](https://issues.apache.org/jira/browse/AIRFLOW-2868) issue and 
references it in the PR title. 
   
   
   ### Description
   
   - [ ] I extended the mesos_executor to be able to execute different tasks 
with different CPU and Memory requirements, as well as different docker images. 
By providing an executor_config dictionary, users can specify these tasks in 
their operators. This builds off of the executor config created for the 
kubernetes_executor.
   
   ### Tests
   
   - [ ] My PR added to tests/contrib/executors/test_mesos_executor.py
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
   
   ### Code Quality
   
   - [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Mesos Executor should use executor_config to specify CPU, Memory and Docker 
> image on the task level
> ---
>
> Key: AIRFLOW-2868
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2868
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: contrib
>Affects Versions: 1.10, 1.10.1
>Reporter: Amir Shahatit
>Assignee: Amir Shahatit
>Priority: Major
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> Executor_config was added as a part of 
> [AIRFLOW-1314|https://github.com/apache/incubator-airflow/commit/c0920efc012468681cff3d3c9cfe25c7381dc976].
>  This task extends the mesosExecutor to make use of specified executor 
> configs to pass on resource requirements (CPU/Memory) as well as docker 
> images on the task level. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] amir656 opened a new pull request #3739: [AIRFLOW-2868] Extend mesos_executor

amir656 opened a new pull request #3739: [AIRFLOW-2868] Extend mesos_executor
URL: https://github.com/apache/incubator-airflow/pull/3739
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following 
[AIRFLOW-2868](https://issues.apache.org/jira/browse/AIRFLOW-2868) issue and 
references it in the PR title. 
   
   
   ### Description
   
   - [ ] I extended the mesos_executor to be able to execute different tasks 
with different CPU and Memory requirements, as well as different docker images. 
By providing an executor_config dictionary, users can specify these tasks in 
their operators. This builds off of the executor config created for the 
kubernetes_executor.
   
   ### Tests
   
   - [ ] My PR added to tests/contrib/executors/test_mesos_executor.py
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
   
   ### Code Quality
   
   - [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-2887) Add to BigQueryBaseCursor methods for creating and updating datasets

Iuliia Volkova created AIRFLOW-2887:
---

 Summary: Add to BigQueryBaseCursor methods for creating and 
updating datasets
 Key: AIRFLOW-2887
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2887
 Project: Apache Airflow
  Issue Type: New Feature
Reporter: Iuliia Volkova
Assignee: Iuliia Volkova


In BigQueryBaseCursor exist only:

def delete_dataset(self, project_id, dataset_id)

 

And there are no hooks to 
create([https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets/insert)]
  and update datasets 
([https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets/update])

[~kaxilnaik], or I'm not right?

If it's so, could I add methods and operators for those actions? 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2887) Add to BigQueryBaseCursor methods for creating and updating datasets



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Iuliia Volkova updated AIRFLOW-2887:

Description: 
In BigQueryBaseCursor exist only:

def delete_dataset(self, project_id, dataset_id)

 And there are no hooks to 
create([https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets/insert)]
  and update datasets 
([https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets/update])

[~kaxilnaik], or I'm not right?

If it's so, could I add methods and operators for those actions? 

  was:
In BigQueryBaseCursor exist only:

def delete_dataset(self, project_id, dataset_id)

 

And there are no hooks to 
create([https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets/insert)]
  and update datasets 
([https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets/update])

[~kaxilnaik], or I'm not right?

If it's so, could I add methods and operators for those actions? 


> Add to BigQueryBaseCursor methods for creating and updating datasets
> 
>
> Key: AIRFLOW-2887
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2887
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Iuliia Volkova
>Assignee: Iuliia Volkova
>Priority: Minor
>
> In BigQueryBaseCursor exist only:
> def delete_dataset(self, project_id, dataset_id)
>  And there are no hooks to 
> create([https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets/insert)]
>   and update datasets 
> ([https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets/update])
> [~kaxilnaik], or I'm not right?
> If it's so, could I add methods and operators for those actions? 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] feng-tao commented on issue #3738: [AIRFLOW-2886] Secure Flask SECRET_KEY

feng-tao commented on issue #3738: [AIRFLOW-2886] Secure Flask SECRET_KEY
URL: 
https://github.com/apache/incubator-airflow/pull/3738#issuecomment-412192087
 
 
   hey @XD-DENG , it seems that it will still be an issue if we have a cluster 
for webservers  based on the comment in 
https://issues.apache.org/jira/browse/AIRFLOW-2866. What do you think? cc 
@Fokko , @kaxil  as they comment / approve the original prs.  


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2866) Missing CSRF Token Error on Web UI Create/Update Operations

2018-08-10 Thread Tao Feng (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576793#comment-16576793
 ] 

Tao Feng commented on AIRFLOW-2866:
---

Thanks [~craigf] . Let me comment that back to ticket / pr.

> Missing CSRF Token Error on Web UI Create/Update Operations
> ---
>
> Key: AIRFLOW-2866
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2866
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: webapp
>Reporter: Jasper Kahn
>Priority: Major
>
> Attempting to modify or delete many resources (such as Connections or Users) 
> results in a 400 from the webserver:
> {quote}{{Bad Request}}
> {{The CSRF session token is missing.}}{quote}
> Logs report:
> {quote}{{[2018-08-07 18:45:15,771] \{csrf.py:251} INFO - The CSRF session 
> token is missing.}}
> {{192.168.9.1 - - [07/Aug/2018:18:45:15 +] "POST 
> /admin/connection/delete/ HTTP/1.1" 400 150 
> "http://localhost:8081/admin/connection/; "Mozilla/5.0 (X11; Linux x86_64) 
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.84 
> Safari/537.36"}}{quote}
> Chrome dev tools show the CSRF token is present in the request payload.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2866) Missing CSRF Token Error on Web UI Create/Update Operations



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576717#comment-16576717
 ] 

Craig Forster commented on AIRFLOW-2866:


[~TaoFeng] Running locally with that branch seems to fix things. I can toggle 
the on/off state for a DAG without the callbacks being rejected for CSRF 
errors.  This is for a single web server instance only.

 

I don't believe this will resolve the issue for a cluster of webservers serving 
requests in a round-robin fashion, though.  Each webserver instance will still 
have a different secret key and reject CSRF tokens from other cluster members.

> Missing CSRF Token Error on Web UI Create/Update Operations
> ---
>
> Key: AIRFLOW-2866
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2866
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: webapp
>Reporter: Jasper Kahn
>Priority: Major
>
> Attempting to modify or delete many resources (such as Connections or Users) 
> results in a 400 from the webserver:
> {quote}{{Bad Request}}
> {{The CSRF session token is missing.}}{quote}
> Logs report:
> {quote}{{[2018-08-07 18:45:15,771] \{csrf.py:251} INFO - The CSRF session 
> token is missing.}}
> {{192.168.9.1 - - [07/Aug/2018:18:45:15 +] "POST 
> /admin/connection/delete/ HTTP/1.1" 400 150 
> "http://localhost:8081/admin/connection/; "Mozilla/5.0 (X11; Linux x86_64) 
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.84 
> Safari/537.36"}}{quote}
> Chrome dev tools show the CSRF token is present in the request payload.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] troychen728 commented on a change in pull request #3658: [AIRFLOW-2524] Add Amazon SageMaker Training

troychen728 commented on a change in pull request #3658: [AIRFLOW-2524] Add 
Amazon SageMaker Training
URL: https://github.com/apache/incubator-airflow/pull/3658#discussion_r209352239
 
 

 ##
 File path: airflow/contrib/hooks/sagemaker_hook.py
 ##
 @@ -0,0 +1,239 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import copy
+import time
+from botocore.exceptions import ClientError
+
+from airflow.exceptions import AirflowException
+from airflow.contrib.hooks.aws_hook import AwsHook
+from airflow.hooks.S3_hook import S3Hook
+
+
+class SageMakerHook(AwsHook):
+"""
+Interact with Amazon SageMaker.
+sagemaker_conn_id is required for using
+the config stored in db for training/tuning
+"""
+
+def __init__(self,
+ sagemaker_conn_id=None,
+ use_db_config=False,
+ region_name=None,
+ check_interval=5,
+ max_ingestion_time=None,
+ *args, **kwargs):
+super(SageMakerHook, self).__init__(*args, **kwargs)
+self.sagemaker_conn_id = sagemaker_conn_id
+self.use_db_config = use_db_config
+self.region_name = region_name
+self.check_interval = check_interval
+self.max_ingestion_time = max_ingestion_time
+self.conn = self.get_conn()
+
+def check_for_url(self, s3url):
+"""
+check if the s3url exists
+:param s3url: S3 url
+:type s3url:str
+:return: bool
+"""
+bucket, key = S3Hook.parse_s3_url(s3url)
+s3hook = S3Hook(aws_conn_id=self.aws_conn_id)
+if not s3hook.check_for_bucket(bucket_name=bucket):
+raise AirflowException(
+"The input S3 Bucket {} does not exist ".format(bucket))
+if not s3hook.check_for_key(key=key, bucket_name=bucket):
+raise AirflowException("The input S3 Key {} does not exist in the 
Bucket"
+   .format(s3url, bucket))
+return True
+
+def check_valid_training_input(self, training_config):
+"""
+Run checks before a training starts
+:param config: training_config
+:type config: dict
+:return: None
+"""
+for channel in training_config['InputDataConfig']:
+self.check_for_url(channel['DataSource']
+   ['S3DataSource']['S3Uri'])
+
+def check_valid_tuning_input(self, tuning_config):
+"""
+Run checks before a tuning job starts
+:param config: tuning_config
+:type config: dict
+:return: None
+"""
+for channel in 
tuning_config['TrainingJobDefinition']['InputDataConfig']:
+self.check_for_url(channel['DataSource']
+   ['S3DataSource']['S3Uri'])
+
+def check_status(self, non_terminal_states,
+ failed_state, key,
+ describe_function, *args):
+"""
+:param non_terminal_states: the set of non_terminal states
+:type non_terminal_states: dict
+:param failed_state: the set of failed states
+:type failed_state: dict
+:param key: the key of the response dict
+that points to the state
+:type key: string
+:param describe_function: the function used to retrieve the status
+:type describe_function: python callable
+:param args: the arguments for the function
+:return: None
+"""
+sec = 0
+running = True
+
+while running:
+
+sec = sec + self.check_interval
+
+if self.max_ingestion_time and sec > self.max_ingestion_time:
+# ensure that the job gets killed if the max ingestion time is 
exceeded
+raise AirflowException("SageMaker job took more than "
+   "%s seconds", self.max_ingestion_time)
+
+time.sleep(self.check_interval)
+try:
+status = describe_function(*args)[key]
+self.log.info("Job still running for %s seconds... "
+  "current status is %s" % (sec, status))
+

[jira] [Commented] (AIRFLOW-2866) Missing CSRF Token Error on Web UI Create/Update Operations

2018-08-10 Thread Tao Feng (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576674#comment-16576674
 ] 

Tao Feng commented on AIRFLOW-2866:
---

[~craigf] , I don't have the setup in my mac. Could you apply this 
pr([https://github.com/apache/incubator-airflow/pull/3738)] and see if it 
solves the issue?

> Missing CSRF Token Error on Web UI Create/Update Operations
> ---
>
> Key: AIRFLOW-2866
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2866
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: webapp
>Reporter: Jasper Kahn
>Priority: Major
>
> Attempting to modify or delete many resources (such as Connections or Users) 
> results in a 400 from the webserver:
> {quote}{{Bad Request}}
> {{The CSRF session token is missing.}}{quote}
> Logs report:
> {quote}{{[2018-08-07 18:45:15,771] \{csrf.py:251} INFO - The CSRF session 
> token is missing.}}
> {{192.168.9.1 - - [07/Aug/2018:18:45:15 +] "POST 
> /admin/connection/delete/ HTTP/1.1" 400 150 
> "http://localhost:8081/admin/connection/; "Mozilla/5.0 (X11; Linux x86_64) 
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.84 
> Safari/537.36"}}{quote}
> Chrome dev tools show the CSRF token is present in the request payload.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2866) Missing CSRF Token Error on Web UI Create/Update Operations



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576657#comment-16576657
 ] 

Craig Forster commented on AIRFLOW-2866:


Yes, it looks related to that. I explicitly set a secret_key for my cluster of 
web servers and it resolved the issue.

> Missing CSRF Token Error on Web UI Create/Update Operations
> ---
>
> Key: AIRFLOW-2866
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2866
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: webapp
>Reporter: Jasper Kahn
>Priority: Major
>
> Attempting to modify or delete many resources (such as Connections or Users) 
> results in a 400 from the webserver:
> {quote}{{Bad Request}}
> {{The CSRF session token is missing.}}{quote}
> Logs report:
> {quote}{{[2018-08-07 18:45:15,771] \{csrf.py:251} INFO - The CSRF session 
> token is missing.}}
> {{192.168.9.1 - - [07/Aug/2018:18:45:15 +] "POST 
> /admin/connection/delete/ HTTP/1.1" 400 150 
> "http://localhost:8081/admin/connection/; "Mozilla/5.0 (X11; Linux x86_64) 
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.84 
> Safari/537.36"}}{quote}
> Chrome dev tools show the CSRF token is present in the request payload.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2844) Airflow Logs BrokenPipeException

2018-08-10 Thread Kyle Bridenstine (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576658#comment-16576658
 ] 

Kyle Bridenstine commented on AIRFLOW-2844:
---

*Root Cause Of Issue:*
 * [https://github.com/apache/incubator-airflow/pull/2484] 

*Stackoverflow Posts That Helped Solve The Issue:*
 * 
[https://stackoverflow.com/questions/51365911/airflow-logs-brokenpipeexception/51790409#51790409]
 * 
[https://stackoverflow.com/questions/51775370/airflowexception-celery-command-failed-the-recorded-hostname-does-not-match-t]
 

*Summary:*

This issue is a symptom of another issue I just resolved here 
[AirflowException: Celery command failed - The recorded hostname does not match 
this instance's hostname][1].

I didn't see the _AirflowException: Celery command failed_ for a while because 
it showed up on the _airflow worker_ logs. It wasn't until I watched the 
airflow worker logs in real time that I saw when the error is thrown I also got 
the BrokenPipeException in my task.

It gets somewhat weirder though. I would only see the BrokenPipeException 
thrown if I did `print("something to log")` *and* the `AirflowException: Celery 
command failed...` error happened on the Worker node. When I changed all of my 
print statements to use `import logging ... logging.info("something to log")` 
then I would not see the BrokenPipeException *but* the task would still fail 
because of the `AirflowException: Celery command failed...` error. But had I 
not seen the BrokenPipeException being thrown in my Airflow task logs I 
wouldn't have known why the task was failing because once I eliminated the 
print statements I never saw any error in the Airflow task logs (only on the 
_$airflow worker_ logs)

So long story short there are a few take aways.
 # Don't do `print("something to log")` use Airflow's built in logging by 
importing logging and then using the logging class like `import logging` then 
`logging.info("something to log")`
 # If you're using an AWS EC2-Instance as your server for Airflow then you may 
be experiencing this issue: 
https://github.com/apache/incubator-airflow/pull/2484 a fix to this issue has 
already been integrated into Airflow Version 1.10 (I'm currently using Airflow 
Version 1.9). So upgrade your [Airflow version to 1.10][2]. You can also use 
[the command here][3] but running `pip install 
git+git://github.com/apache/incubator-airflow.git` gives me version 
`v2.0.0.dev0+incubating` which I'm not sure if that's a development version or 
what, but it appears to be higher than 1.10 so perhaps it'll work. Also, if you 
don't want to upgrade your Airflow version then you could follow the steps on 
[the github issue][4] to either manually update the file with the fix or fork 
Airflow and cherry pick the commit that fixes it.

*Sources:*
 [1]: 
https://stackoverflow.com/questions/51775370/airflowexception-celery-command-failed-the-recorded-hostname-does-not-match-t
 [2]: https://github.com/apache/incubator-airflow/releases
 [3]: https://stackoverflow.com/a/47540377/3299397
 [4]: https://github.com/apache/incubator-airflow/pull/2484

> Airflow Logs BrokenPipeException
> 
>
> Key: AIRFLOW-2844
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2844
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: logging
>Affects Versions: 1.9.0
>Reporter: Kyle Bridenstine
>Priority: Critical
>
> I'm using a clustered Airflow environment where I have four AWS ec2-instances 
> for the servers.
> *ec2-instances*
>  - Server 1: Webserver, Scheduler, Redis Queue, PostgreSQL Database
>  - Server 2: Webserver
>  - Server 3: Worker
>  - Server 4: Worker
> My setup has been working perfectly fine for three months now but 
> sporadically about once a week I get a Broken Pipe Exception when Airflow is 
> attempting to log something.
> {code:java}
> Log file isn't local.
> Fetching here: 
> http://ip-1-2-3-4:8793/log/foobar/task_1/2018-07-13T00:00:00/1.log
> [2018-07-16 00:00:15,521] {cli.py:374} INFO - Running on host ip-1-2-3-4
> [2018-07-16 00:00:15,698] {models.py:1197} INFO - Dependencies all met for 
> 
> [2018-07-16 00:00:15,710] {models.py:1197} INFO - Dependencies all met for 
> 
> [2018-07-16 00:00:15,710] {models.py:1407} INFO - 
> 
> Starting attempt 1 of 1
> 
> [2018-07-16 00:00:15,719] {models.py:1428} INFO - Executing 
>  on 2018-07-13 00:00:00
> [2018-07-16 00:00:15,720] {base_task_runner.py:115} INFO - Running: ['bash', 
> '-c', 'airflow run foobar task_1 2018-07-13T00:00:00 --job_id 1320 --raw -sd 
> DAGS_FOLDER/datalake_digitalplatform_arl_workflow_schedule_test_2.py']
> [2018-07-16 00:00:16,532] {base_task_runner.py:98} INFO - Subtask:

[jira] [Commented] (AIRFLOW-2866) Missing CSRF Token Error on Web UI Create/Update Operations

2018-08-10 Thread Tao Feng (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576627#comment-16576627
 ] 

Tao Feng commented on AIRFLOW-2866:
---

which version did you run? Could it be related to AIRFLOW-2809?

> Missing CSRF Token Error on Web UI Create/Update Operations
> ---
>
> Key: AIRFLOW-2866
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2866
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: webapp
>Reporter: Jasper Kahn
>Priority: Major
>
> Attempting to modify or delete many resources (such as Connections or Users) 
> results in a 400 from the webserver:
> {quote}{{Bad Request}}
> {{The CSRF session token is missing.}}{quote}
> Logs report:
> {quote}{{[2018-08-07 18:45:15,771] \{csrf.py:251} INFO - The CSRF session 
> token is missing.}}
> {{192.168.9.1 - - [07/Aug/2018:18:45:15 +] "POST 
> /admin/connection/delete/ HTTP/1.1" 400 150 
> "http://localhost:8081/admin/connection/; "Mozilla/5.0 (X11; Linux x86_64) 
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.84 
> Safari/537.36"}}{quote}
> Chrome dev tools show the CSRF token is present in the request payload.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] feng-tao edited a comment on issue #3732: Fix typos detected by github.com/client9/misspell

feng-tao edited a comment on issue #3732: Fix typos detected by 
github.com/client9/misspell
URL: 
https://github.com/apache/incubator-airflow/pull/3732#issuecomment-412148787
 
 
   hey @seratch , the tool looks very useful :). Your change lgtm. But could 
you help to create a jira ticket and append that jira on your commit title(e.g 
https://github.com/apache/incubator-airflow/pull/3648)?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2866) Missing CSRF Token Error on Web UI Create/Update Operations



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576602#comment-16576602
 ] 

Craig Forster commented on AIRFLOW-2866:


Looks related to AIRFLOW-2866?

> Missing CSRF Token Error on Web UI Create/Update Operations
> ---
>
> Key: AIRFLOW-2866
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2866
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: webapp
>Reporter: Jasper Kahn
>Priority: Major
>
> Attempting to modify or delete many resources (such as Connections or Users) 
> results in a 400 from the webserver:
> {quote}{{Bad Request}}
> {{The CSRF session token is missing.}}{quote}
> Logs report:
> {quote}{{[2018-08-07 18:45:15,771] \{csrf.py:251} INFO - The CSRF session 
> token is missing.}}
> {{192.168.9.1 - - [07/Aug/2018:18:45:15 +] "POST 
> /admin/connection/delete/ HTTP/1.1" 400 150 
> "http://localhost:8081/admin/connection/; "Mozilla/5.0 (X11; Linux x86_64) 
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.84 
> Safari/537.36"}}{quote}
> Chrome dev tools show the CSRF token is present in the request payload.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] xnuinside commented on issue #3733: [WIP] [AIRFLOW-491] Add cache parameter in BigQuery query method

xnuinside commented on issue #3733: [WIP] [AIRFLOW-491] Add cache parameter in 
BigQuery query method
URL: 
https://github.com/apache/incubator-airflow/pull/3733#issuecomment-412147816
 
 
   @kaxil , how do you  think need I to remove other params relative to "query" 
like use_legacy? and just described what they can be set up using 
src_fmt_configs?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] XD-DENG commented on issue #3737: Revert "[AIRFLOW-2878] Fix www_rbac display issue"

XD-DENG commented on issue #3737: Revert "[AIRFLOW-2878] Fix www_rbac display 
issue"
URL: 
https://github.com/apache/incubator-airflow/pull/3737#issuecomment-412147753
 
 
   Thanks @feng-tao 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2866) Missing CSRF Token Error on Web UI Create/Update Operations

2018-08-10 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576600#comment-16576600
 ] 

Craig Forster commented on AIRFLOW-2866:


I'm seeing this on master as well.

> Missing CSRF Token Error on Web UI Create/Update Operations
> ---
>
> Key: AIRFLOW-2866
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2866
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: webapp
>Reporter: Jasper Kahn
>Priority: Major
>
> Attempting to modify or delete many resources (such as Connections or Users) 
> results in a 400 from the webserver:
> {quote}{{Bad Request}}
> {{The CSRF session token is missing.}}{quote}
> Logs report:
> {quote}{{[2018-08-07 18:45:15,771] \{csrf.py:251} INFO - The CSRF session 
> token is missing.}}
> {{192.168.9.1 - - [07/Aug/2018:18:45:15 +] "POST 
> /admin/connection/delete/ HTTP/1.1" 400 150 
> "http://localhost:8081/admin/connection/; "Mozilla/5.0 (X11; Linux x86_64) 
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.84 
> Safari/537.36"}}{quote}
> Chrome dev tools show the CSRF token is present in the request payload.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2878) "/www_rbac" Layout/UI Display Issue



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576596#comment-16576596
 ] 

ASF subversion and git services commented on AIRFLOW-2878:
--

Commit f999ce22c16b11e8b9205d1bbb6af0683a0c7a3d in incubator-airflow's branch 
refs/heads/master from [~TaoFeng]
[ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=f999ce2 ]

Revert "[AIRFLOW-2878] Fix www_rbac display issue" (#3737)

This reverts commit 1f57dafd70ac41da8295f20793f8913b7f5a5dff.

> "/www_rbac" Layout/UI Display Issue
> ---
>
> Key: AIRFLOW-2878
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2878
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ui
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Critical
>
> The new RBAC UI has some issues about layout/UI display.
> The header () is not shown ("hidden" by the Nav Bar), or tables are not 
> shown completely.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2878) "/www_rbac" Layout/UI Display Issue



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576595#comment-16576595
 ] 

ASF GitHub Bot commented on AIRFLOW-2878:
-

feng-tao closed pull request #3737: Revert "[AIRFLOW-2878] Fix www_rbac display 
issue"
URL: https://github.com/apache/incubator-airflow/pull/3737
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/airflow/www_rbac/templates/appbuilder/baselayout.html 
b/airflow/www_rbac/templates/appbuilder/baselayout.html
index b0728ca14b..1653a909d0 100644
--- a/airflow/www_rbac/templates/appbuilder/baselayout.html
+++ b/airflow/www_rbac/templates/appbuilder/baselayout.html
@@ -43,9 +43,6 @@
 
 
 
-  
-  
-  
   
   {% block messages %}
 {% include 'appbuilder/flash.html' %}


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> "/www_rbac" Layout/UI Display Issue
> ---
>
> Key: AIRFLOW-2878
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2878
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ui
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Critical
>
> The new RBAC UI has some issues about layout/UI display.
> The header () is not shown ("hidden" by the Nav Bar), or tables are not 
> shown completely.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] feng-tao closed pull request #3737: Revert "[AIRFLOW-2878] Fix www_rbac display issue"

feng-tao closed pull request #3737: Revert "[AIRFLOW-2878] Fix www_rbac display 
issue"
URL: https://github.com/apache/incubator-airflow/pull/3737
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/airflow/www_rbac/templates/appbuilder/baselayout.html 
b/airflow/www_rbac/templates/appbuilder/baselayout.html
index b0728ca14b..1653a909d0 100644
--- a/airflow/www_rbac/templates/appbuilder/baselayout.html
+++ b/airflow/www_rbac/templates/appbuilder/baselayout.html
@@ -43,9 +43,6 @@
 
 
 
-  
-  
-  
   
   {% block messages %}
 {% include 'appbuilder/flash.html' %}


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] XD-DENG commented on issue #3738: [AIRFLOW-2886] Secure Flask SECRET_KEY

XD-DENG commented on issue #3738: [AIRFLOW-2886] Secure Flask SECRET_KEY
URL: 
https://github.com/apache/incubator-airflow/pull/3738#issuecomment-412146841
 
 
   @feng-tao @Fokko PTAL. 
   
   We need to have random SECRET_KEY for Flask App, while it must be consistent 
among workers. So I propose to deal with it like how we dealt with `FERNET_KEY`.
   
   Thanks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2886) Secure Flask SECRET_KEY



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576591#comment-16576591
 ] 

ASF GitHub Bot commented on AIRFLOW-2886:
-

XD-DENG opened a new pull request #3738: [AIRFLOW-2886] Secure Flask SECRET_KEY
URL: https://github.com/apache/incubator-airflow/pull/3738
 
 
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-2886
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   In my earlier PRs, https://github.com/apache/incubator-airflow/pull/3651 and 
https://github.com/apache/incubator-airflow/pull/3729 , I proposed to generate 
random `SECRET_KEY` for the `webserver` (Flask App).
   
   However, I realise that we may encounter CSRF error `The CSRF session token 
is missing` when we have multiple workers for the Flask webserver, since the 
secret_key is not consistent among workers.
   
   On the other hand, it's still very important to have as random SECRET_KEY as 
possible for security reasons. We can deal with it like how we dealt with 
`FERNET_KEY` (i.e. generate a random value when the airflow.cfg file is 
initiated).
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Secure Flask SECRET_KEY
> ---
>
> Key: AIRFLOW-2886
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2886
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Critical
>
> In my earlier PRs, [https://github.com/apache/incubator-airflow/pull/3651] 
> and [https://github.com/apache/incubator-airflow/pull/3729] , I proposed to 
> generate random SECRET_KEY for Flask App.
> If we have multiple workers for the Flask webserver, we may encounter CSRF 
> error {{The CSRF session token is missing}} .
> On the other hand, it's still very important to have as random SECRET_KEY as 
> possible for security reasons. We can deal with it like how we dealt with 
> FERNET_KEY (i.e. generate a random value when the airflow.cfg file is 
> initiated).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] XD-DENG opened a new pull request #3738: [AIRFLOW-2886] Secure Flask SECRET_KEY

XD-DENG opened a new pull request #3738: [AIRFLOW-2886] Secure Flask SECRET_KEY
URL: https://github.com/apache/incubator-airflow/pull/3738
 
 
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-2886
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   In my earlier PRs, https://github.com/apache/incubator-airflow/pull/3651 and 
https://github.com/apache/incubator-airflow/pull/3729 , I proposed to generate 
random `SECRET_KEY` for the `webserver` (Flask App).
   
   However, I realise that we may encounter CSRF error `The CSRF session token 
is missing` when we have multiple workers for the Flask webserver, since the 
secret_key is not consistent among workers.
   
   On the other hand, it's still very important to have as random SECRET_KEY as 
possible for security reasons. We can deal with it like how we dealt with 
`FERNET_KEY` (i.e. generate a random value when the airflow.cfg file is 
initiated).
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-2886) Secure Flask SECRET_KEY

2018-08-10 Thread Xiaodong DENG (JIRA)

Xiaodong DENG created AIRFLOW-2886:
--

 Summary: Secure Flask SECRET_KEY
 Key: AIRFLOW-2886
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2886
 Project: Apache Airflow
  Issue Type: Bug
Reporter: Xiaodong DENG
Assignee: Xiaodong DENG


In my earlier PRs, [https://github.com/apache/incubator-airflow/pull/3651] and 
[https://github.com/apache/incubator-airflow/pull/3729] , I proposed to 
generate random SECRET_KEY for Flask App.

If we have multiple workers for the Flask webserver, we may encounter CSRF 
error {{The CSRF session token is missing}} .

On the other hand, it's still very important to have as random SECRET_KEY as 
possible for security reasons. We can deal with it like how we dealt with 
FERNET_KEY (i.e. generate a random value when the airflow.cfg file is 
initiated).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] feng-tao commented on issue #3734: Revert "[AIRFLOW-2878] Fix www_rbac display issue"

feng-tao commented on issue #3734: Revert "[AIRFLOW-2878] Fix www_rbac display 
issue"
URL: 
https://github.com/apache/incubator-airflow/pull/3734#issuecomment-412138578
 
 
   @XD-DENG , I think directly revertting your pr in github may revert other 
prs that get merged after yours. But never mind, I will take care of that. 
Thanks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao commented on issue #3734: Revert "[AIRFLOW-2878] Fix www_rbac display issue"

feng-tao commented on issue #3734: Revert "[AIRFLOW-2878] Fix www_rbac display 
issue"
URL: 
https://github.com/apache/incubator-airflow/pull/3734#issuecomment-412139622
 
 
   @XD-DENG , will do.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] XD-DENG commented on issue #3734: Revert "[AIRFLOW-2878] Fix www_rbac display issue"

XD-DENG commented on issue #3734: Revert "[AIRFLOW-2878] Fix www_rbac display 
issue"
URL: 
https://github.com/apache/incubator-airflow/pull/3734#issuecomment-412139420
 
 
   @feng-tao Or may I trouble you to do a separate commit to remove the three 
lines I added? i.e. it's actually reverting the changes made in my earlier PR 
but doesn't touch other PRs?
   
   Thanks for taking care of this, and sorry for the inconvenience caused.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao commented on issue #3737: Revert "[AIRFLOW-2878] Fix www_rbac display issue"

feng-tao commented on issue #3737: Revert "[AIRFLOW-2878] Fix www_rbac display 
issue"
URL: 
https://github.com/apache/incubator-airflow/pull/3737#issuecomment-412138003
 
 
   This is based on the discussion in 
https://github.com/apache/incubator-airflow/pull/3724. cc @r39132 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2878) "/www_rbac" Layout/UI Display Issue



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576546#comment-16576546
 ] 

ASF GitHub Bot commented on AIRFLOW-2878:
-

feng-tao opened a new pull request #3737: Revert "[AIRFLOW-2878] Fix www_rbac 
display issue"
URL: https://github.com/apache/incubator-airflow/pull/3737
 
 
   This reverts commit 1f57dafd70ac41da8295f20793f8913b7f5a5dff.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> "/www_rbac" Layout/UI Display Issue
> ---
>
> Key: AIRFLOW-2878
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2878
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ui
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Critical
>
> The new RBAC UI has some issues about layout/UI display.
> The header () is not shown ("hidden" by the Nav Bar), or tables are not 
> shown completely.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] feng-tao opened a new pull request #3737: Revert "[AIRFLOW-2878] Fix www_rbac display issue"

feng-tao opened a new pull request #3737: Revert "[AIRFLOW-2878] Fix www_rbac 
display issue"
URL: https://github.com/apache/incubator-airflow/pull/3737
 
 
   This reverts commit 1f57dafd70ac41da8295f20793f8913b7f5a5dff.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao closed pull request #3734: Revert "[AIRFLOW-2878] Fix www_rbac display issue"

feng-tao closed pull request #3734: Revert "[AIRFLOW-2878] Fix www_rbac display 
issue"
URL: https://github.com/apache/incubator-airflow/pull/3734
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/airflow/www_rbac/templates/appbuilder/baselayout.html 
b/airflow/www_rbac/templates/appbuilder/baselayout.html
index b0728ca14b..1653a909d0 100644
--- a/airflow/www_rbac/templates/appbuilder/baselayout.html
+++ b/airflow/www_rbac/templates/appbuilder/baselayout.html
@@ -43,9 +43,6 @@
 
 
 
-  
-  
-  
   
   {% block messages %}
 {% include 'appbuilder/flash.html' %}


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2878) "/www_rbac" Layout/UI Display Issue



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576536#comment-16576536
 ] 

ASF GitHub Bot commented on AIRFLOW-2878:
-

XD-DENG opened a new pull request #3734: Revert "[AIRFLOW-2878] Fix www_rbac 
display issue"
URL: https://github.com/apache/incubator-airflow/pull/3734
 
 
   Hi @r39132 , this is to revert my earlier PR 
https://github.com/apache/incubator-airflow/pull/3724.
   
   As pointed out by @verdan , I encountered the issues that I tried to fix in 
my earlier PR because I didn't build the frontend packages using `npm` and 
`webpack`. That commit may also introduce extra spaces in each page.
   
   Please refer to my conversation with Verdan in PR 
https://github.com/apache/incubator-airflow/pull/3724.
   
   Sorry for the inconvenience cause, and thanks @verdan for having pointed 
this out.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> "/www_rbac" Layout/UI Display Issue
> ---
>
> Key: AIRFLOW-2878
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2878
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ui
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Critical
>
> The new RBAC UI has some issues about layout/UI display.
> The header () is not shown ("hidden" by the Nav Bar), or tables are not 
> shown completely.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] feng-tao commented on issue #3728: [AIRFLOW-2883] Not search dag owner if owners are missing

feng-tao commented on issue #3728: [AIRFLOW-2883] Not search dag owner if 
owners are missing
URL: 
https://github.com/apache/incubator-airflow/pull/3728#issuecomment-412137090
 
 
   PTAL @r39132 , any other concern?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2878) "/www_rbac" Layout/UI Display Issue



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576532#comment-16576532
 ] 

ASF GitHub Bot commented on AIRFLOW-2878:
-

feng-tao closed pull request #3736: Revert "[AIRFLOW-2878] Fix www_rbac display 
issue"
URL: https://github.com/apache/incubator-airflow/pull/3736
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> "/www_rbac" Layout/UI Display Issue
> ---
>
> Key: AIRFLOW-2878
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2878
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ui
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Critical
>
> The new RBAC UI has some issues about layout/UI display.
> The header () is not shown ("hidden" by the Nav Bar), or tables are not 
> shown completely.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] feng-tao commented on issue #3648: [AIRFLOW-2786] Fix editing Variable with empty key crashing

feng-tao commented on issue #3648: [AIRFLOW-2786] Fix editing Variable with 
empty key crashing
URL: 
https://github.com/apache/incubator-airflow/pull/3648#issuecomment-412134502
 
 
   thanks @Noremac201 . Merge.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2786) Variables view fails to render if a variable has an empty key



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576521#comment-16576521
 ] 

ASF GitHub Bot commented on AIRFLOW-2786:
-

feng-tao closed pull request #3648: [AIRFLOW-2786] Fix editing Variable with 
empty key crashing
URL: https://github.com/apache/incubator-airflow/pull/3648
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/airflow/www/utils.py b/airflow/www/utils.py
index 1bbc0936b3..0c4f4b05d6 100644
--- a/airflow/www/utils.py
+++ b/airflow/www/utils.py
@@ -56,8 +56,13 @@
 
 
 def should_hide_value_for_key(key_name):
-return any(s in key_name.lower() for s in 
DEFAULT_SENSITIVE_VARIABLE_FIELDS) \
-and configuration.conf.getboolean('admin', 
'hide_sensitive_variable_fields')
+# It is possible via importing variables from file that a key is empty.
+if key_name:
+config_set = configuration.conf.getboolean('admin',
+   
'hide_sensitive_variable_fields')
+field_comp = any(s in key_name.lower() for s in 
DEFAULT_SENSITIVE_VARIABLE_FIELDS)
+return config_set and field_comp
+return False
 
 
 class LoginMixin(object):
diff --git a/airflow/www/views.py b/airflow/www/views.py
index 3e41d2d02a..0c0dcff801 100644
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -2012,9 +2012,20 @@ def varimport(self):
 except Exception as e:
 flash("Missing file or syntax error: {}.".format(e))
 else:
+suc_count = fail_count = 0
 for k, v in d.items():
-models.Variable.set(k, v, serialize_json=isinstance(v, dict))
-flash("{} variable(s) successfully updated.".format(len(d)))
+try:
+models.Variable.set(k, v, serialize_json=isinstance(v, 
dict))
+except Exception as e:
+logging.info('Variable import failed: {}'.format(repr(e)))
+fail_count += 1
+else:
+suc_count += 1
+flash("{} variable(s) successfully updated.".format(suc_count), 
'info')
+if fail_count:
+flash(
+"{} variables(s) failed to be 
updated.".format(fail_count), 'error')
+
 return redirect('/admin/variable')
 
 
diff --git a/airflow/www_rbac/utils.py b/airflow/www_rbac/utils.py
index 7bbdada555..a0e9258eae 100644
--- a/airflow/www_rbac/utils.py
+++ b/airflow/www_rbac/utils.py
@@ -54,8 +54,13 @@
 
 
 def should_hide_value_for_key(key_name):
-return any(s in key_name.lower() for s in 
DEFAULT_SENSITIVE_VARIABLE_FIELDS) \
-and configuration.getboolean('admin', 'hide_sensitive_variable_fields')
+# It is possible via importing variables from file that a key is empty.
+if key_name:
+config_set = configuration.conf.getboolean('admin',
+   
'hide_sensitive_variable_fields')
+field_comp = any(s in key_name.lower() for s in 
DEFAULT_SENSITIVE_VARIABLE_FIELDS)
+return config_set and field_comp
+return False
 
 
 def get_params(**kwargs):
diff --git a/airflow/www_rbac/views.py b/airflow/www_rbac/views.py
index a9947ae096..629f488fc7 100644
--- a/airflow/www_rbac/views.py
+++ b/airflow/www_rbac/views.py
@@ -2053,9 +2053,18 @@ def varimport(self):
 except Exception:
 flash("Missing file or syntax error.")
 else:
+suc_count = fail_count = 0
 for k, v in d.items():
-models.Variable.set(k, v, serialize_json=isinstance(v, dict))
-flash("{} variable(s) successfully updated.".format(len(d)))
+try:
+models.Variable.set(k, v, serialize_json=isinstance(v, 
dict))
+except Exception as e:
+logging.info('Variable import failed: {}'.format(repr(e)))
+fail_count += 1
+else:
+suc_count += 1
+flash("{} variable(s) successfully updated.".format(suc_count), 
'info')
+if fail_count:
+flash("{} variables(s) failed to be 
updated.".format(fail_count), 'error')
 self.update_redirect()
 return redirect(self.get_redirect())
 
diff --git a/tests/www/test_utils.py b/tests/www/test_utils.py
index 891298c0a9..9034b8b5fd 100644
--- a/tests/www/test_utils.py
+++ b/tests/www/test_utils.py
@@ -32,6 +32,10 @@ class UtilsTest(unittest.TestCase):
 def setUp(self):
 super(UtilsTest, self).setUp()
 
+def test_empty_variable_should_not_be_hidden(self):
+self.assertFalse(utils.should_hide_value_for_key(""))

[jira] [Commented] (AIRFLOW-2878) "/www_rbac" Layout/UI Display Issue



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576519#comment-16576519
 ] 

ASF GitHub Bot commented on AIRFLOW-2878:
-

XD-DENG closed pull request #3734: Revert "[AIRFLOW-2878] Fix www_rbac display 
issue"
URL: https://github.com/apache/incubator-airflow/pull/3734
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/airflow/www_rbac/templates/appbuilder/baselayout.html 
b/airflow/www_rbac/templates/appbuilder/baselayout.html
index b0728ca14b..1653a909d0 100644
--- a/airflow/www_rbac/templates/appbuilder/baselayout.html
+++ b/airflow/www_rbac/templates/appbuilder/baselayout.html
@@ -43,9 +43,6 @@
 
 
 
-  
-  
-  
   
   {% block messages %}
 {% include 'appbuilder/flash.html' %}


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> "/www_rbac" Layout/UI Display Issue
> ---
>
> Key: AIRFLOW-2878
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2878
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ui
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Critical
>
> The new RBAC UI has some issues about layout/UI display.
> The header () is not shown ("hidden" by the Nav Bar), or tables are not 
> shown completely.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] feng-tao commented on issue #3734: Revert "[AIRFLOW-2878] Fix www_rbac display issue"

feng-tao commented on issue #3734: Revert "[AIRFLOW-2878] Fix www_rbac display 
issue"
URL: 
https://github.com/apache/incubator-airflow/pull/3734#issuecomment-412134197
 
 
   I don't think we need another pull request. We directly revert the original 
request. I will do that.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2878) "/www_rbac" Layout/UI Display Issue



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576518#comment-16576518
 ] 

ASF GitHub Bot commented on AIRFLOW-2878:
-

feng-tao opened a new pull request #3736: Revert "[AIRFLOW-2878] Fix www_rbac 
display issue"
URL: https://github.com/apache/incubator-airflow/pull/3736
 
 
   Reverts apache/incubator-airflow#3724


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> "/www_rbac" Layout/UI Display Issue
> ---
>
> Key: AIRFLOW-2878
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2878
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ui
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Critical
>
> The new RBAC UI has some issues about layout/UI display.
> The header () is not shown ("hidden" by the Nav Bar), or tables are not 
> shown completely.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] feng-tao opened a new pull request #3736: Revert "[AIRFLOW-2878] Fix www_rbac display issue"

feng-tao opened a new pull request #3736: Revert "[AIRFLOW-2878] Fix www_rbac 
display issue"
URL: https://github.com/apache/incubator-airflow/pull/3736
 
 
   Reverts apache/incubator-airflow#3724


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2878) "/www_rbac" Layout/UI Display Issue



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576514#comment-16576514
 ] 

ASF GitHub Bot commented on AIRFLOW-2878:
-

feng-tao closed pull request #3735: Revert "[AIRFLOW-2878] Fix www_rbac display 
issue"
URL: https://github.com/apache/incubator-airflow/pull/3735
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> "/www_rbac" Layout/UI Display Issue
> ---
>
> Key: AIRFLOW-2878
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2878
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ui
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Critical
>
> The new RBAC UI has some issues about layout/UI display.
> The header () is not shown ("hidden" by the Nav Bar), or tables are not 
> shown completely.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2878) "/www_rbac" Layout/UI Display Issue



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576512#comment-16576512
 ] 

ASF GitHub Bot commented on AIRFLOW-2878:
-

feng-tao opened a new pull request #3735: Revert "[AIRFLOW-2878] Fix www_rbac 
display issue"
URL: https://github.com/apache/incubator-airflow/pull/3735
 
 
   Reverts apache/incubator-airflow#3724


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> "/www_rbac" Layout/UI Display Issue
> ---
>
> Key: AIRFLOW-2878
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2878
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ui
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Critical
>
> The new RBAC UI has some issues about layout/UI display.
> The header () is not shown ("hidden" by the Nav Bar), or tables are not 
> shown completely.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] feng-tao closed pull request #3735: Revert "[AIRFLOW-2878] Fix www_rbac display issue"

feng-tao closed pull request #3735: Revert "[AIRFLOW-2878] Fix www_rbac display 
issue"
URL: https://github.com/apache/incubator-airflow/pull/3735
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao opened a new pull request #3735: Revert "[AIRFLOW-2878] Fix www_rbac display issue"

feng-tao opened a new pull request #3735: Revert "[AIRFLOW-2878] Fix www_rbac 
display issue"
URL: https://github.com/apache/incubator-airflow/pull/3735
 
 
   Reverts apache/incubator-airflow#3724


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] XD-DENG commented on issue #3729: [AIRFLOW-2884] Fix Flask SECRET_KEY security issue in www_rbac

XD-DENG commented on issue #3729: [AIRFLOW-2884] Fix Flask SECRET_KEY security 
issue in www_rbac
URL: 
https://github.com/apache/incubator-airflow/pull/3729#issuecomment-412133255
 
 
   Thanks @kaxil 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] kaxil edited a comment on issue #3729: [AIRFLOW-2884] Fix Flask SECRET_KEY security issue in www_rbac

kaxil edited a comment on issue #3729: [AIRFLOW-2884] Fix Flask SECRET_KEY 
security issue in www_rbac
URL: 
https://github.com/apache/incubator-airflow/pull/3729#issuecomment-412132706
 
 
   @XD-DENG I would be on holidays and hence unreachable, please ping @feng-tao 
or @Fokko  when it is ready.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] kaxil commented on issue #3729: [AIRFLOW-2884] Fix Flask SECRET_KEY security issue in www_rbac

kaxil commented on issue #3729: [AIRFLOW-2884] Fix Flask SECRET_KEY security 
issue in www_rbac
URL: 
https://github.com/apache/incubator-airflow/pull/3729#issuecomment-412132706
 
 
   @XD-DENG I would be on holdiays and hence unreachable, please ping @feng-tao 
or @Fokko  when it is ready.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] kaxil commented on issue #3733: [WIP] [AIRFLOW-491] Add cache parameter in BigQuery query method

kaxil commented on issue #3733: [WIP] [AIRFLOW-491] Add cache parameter in 
BigQuery query method
URL: 
https://github.com/apache/incubator-airflow/pull/3733#issuecomment-412132227
 
 
   Hi @xnuinside Please add `src_fmt_configs` param to `run_query` method and 
the Operator.
   

https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/bigquery_operator.py#L463
   
   This would allow to add such parameters in future without having to add a 
new parameter each time.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] XD-DENG commented on issue #3729: [AIRFLOW-2884] Fix Flask SECRET_KEY security issue in www_rbac

XD-DENG commented on issue #3729: [AIRFLOW-2884] Fix Flask SECRET_KEY security 
issue in www_rbac
URL: 
https://github.com/apache/incubator-airflow/pull/3729#issuecomment-412129544
 
 
   Hi @kaxil , I have realised this method will cause CSRF error `The CSRF 
session token is missing` when we have multiple workers for `webserver` (we 
generate random secret_key for each worker, and then they're not consistent 
among workers).
   
   But I think it's still very necessary to have `as random secret_key as 
possible`. One feasible way is to generate it like how we generate `fernet_key`.
   
   I will raise a separate PR to address this and ping you then. Sorry for the 
inconvenience caused.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] Noremac201 commented on issue #3648: [AIRFLOW-2786] Fix editing Variable with empty key crashing

Noremac201 commented on issue #3648: [AIRFLOW-2786] Fix editing Variable with 
empty key crashing
URL: 
https://github.com/apache/incubator-airflow/pull/3648#issuecomment-412125247
 
 
   Looks like it was a python 3.5 incompatibility -- fixed now
   
   Thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-1059) Reset_state_for_orphaned_task should operate in batch for the scheduler

2018-08-10 Thread Feng Lu (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-1059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576372#comment-16576372
 ] 

Feng Lu commented on AIRFLOW-1059:
--

[~saguziel] Could you elaborate a bit more why orphaned tasks from backfilled 
dagruns can't be re-enqueued? 

> Reset_state_for_orphaned_task should operate in batch for the scheduler
> ---
>
> Key: AIRFLOW-1059
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1059
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Alex Guziel
>Assignee: Alex Guziel
>Priority: Major
> Fix For: 1.9.0
>
>
> Scheduler startup is very slow due to resetting state making a query for each 
> dag run. We should be able to do this in a constant number of queries which 
> will increase scheduler startup time significantly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2878) "/www_rbac" Layout/UI Display Issue



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576369#comment-16576369
 ] 

ASF GitHub Bot commented on AIRFLOW-2878:
-

XD-DENG opened a new pull request #3734: Revert "[AIRFLOW-2878] Fix www_rbac 
display issue"
URL: https://github.com/apache/incubator-airflow/pull/3734
 
 
   Hi @r39132 , this is to revert my earlier PR 
https://github.com/apache/incubator-airflow/pull/3724.
   
   As pointed out by @verdan , I encountered the issues that I tried to fix in 
my earlier PR because I didn't build the frontend packages using `npm` and 
`webpack`. That commit may also introduce extra spaces in each page.
   
   Please refer to my conversation with Verdan in PR 
https://github.com/apache/incubator-airflow/pull/3724.
   
   Sorry for the inconvenience cause, and thanks @verdan for having pointed 
this out.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> "/www_rbac" Layout/UI Display Issue
> ---
>
> Key: AIRFLOW-2878
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2878
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ui
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Critical
>
> The new RBAC UI has some issues about layout/UI display.
> The header () is not shown ("hidden" by the Nav Bar), or tables are not 
> shown completely.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] jrderuiter commented on issue #3560: [AIRFLOW-2697] Drop snakebite in favour of hdfs3

jrderuiter commented on issue #3560: [AIRFLOW-2697] Drop snakebite in favour of 
hdfs3
URL: 
https://github.com/apache/incubator-airflow/pull/3560#issuecomment-412103674
 
 
   @Fokko As we discussed, I removed the deprecation code so we can merge this 
in preparation for Airflow 2.0. I want to do some final testing, afterwards we 
can merge if everything passes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] XD-DENG opened a new pull request #3734: Revert "[AIRFLOW-2878] Fix www_rbac display issue"

XD-DENG opened a new pull request #3734: Revert "[AIRFLOW-2878] Fix www_rbac 
display issue"
URL: https://github.com/apache/incubator-airflow/pull/3734
 
 
   Hi @r39132 , this is to revert my earlier PR 
https://github.com/apache/incubator-airflow/pull/3724.
   
   As pointed out by @verdan , I encountered the issues that I tried to fix in 
my earlier PR because I didn't build the frontend packages using `npm` and 
`webpack`. That commit may also introduce extra spaces in each page.
   
   Please refer to my conversation with Verdan in PR 
https://github.com/apache/incubator-airflow/pull/3724.
   
   Sorry for the inconvenience cause, and thanks @verdan for having pointed 
this out.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] xnuinside commented on issue #3733: [AIRFLOW-491] Add cache parameter in BigQuery query method

xnuinside commented on issue #3733:  [AIRFLOW-491] Add cache parameter in 
BigQuery query method
URL: 
https://github.com/apache/incubator-airflow/pull/3733#issuecomment-412071257
 
 
   maybe make sense add use_query_cache=False for check operators


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] xnuinside opened a new pull request #3733: [AIRFLOW-491] Add cache parameter in BigQuery query method

xnuinside opened a new pull request #3733:  [AIRFLOW-491] Add cache parameter 
in BigQuery query method
URL: https://github.com/apache/incubator-airflow/pull/3733
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-491
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   Added "useQueryCache" from job BQ configuration 
https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query 
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-491) Add cache parameter in BigQuery query method



[ 
https://issues.apache.org/jira/browse/AIRFLOW-491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576223#comment-16576223
 ] 

ASF GitHub Bot commented on AIRFLOW-491:


xnuinside opened a new pull request #3733:  [AIRFLOW-491] Add cache parameter 
in BigQuery query method
URL: https://github.com/apache/incubator-airflow/pull/3733
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-491
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   Added "useQueryCache" from job BQ configuration 
https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query 
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add cache parameter in BigQuery query method
> 
>
> Key: AIRFLOW-491
> URL: https://issues.apache.org/jira/browse/AIRFLOW-491
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: contrib, gcp
>Affects Versions: Airflow 1.7.1
>Reporter: Chris Riccomini
>Assignee: Iuliia Volkova
>Priority: Major
> Fix For: Airflow 1.8
>
>
> The current BigQuery query() method does not have a user_query_cache 
> parameter. This param always defaults to true (see 
> [here|https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.query]).
>  I'd like to disable query caching for some data consistency checks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] XD-DENG commented on issue #3724: [AIRFLOW-2878] Fix www_rbac display issue

XD-DENG commented on issue #3724: [AIRFLOW-2878] Fix www_rbac display issue
URL: 
https://github.com/apache/incubator-airflow/pull/3724#issuecomment-412062778
 
 
   Sure. @r39132 may you please revert this commit as @verdan suggested?
   Thanks for pointing this out!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] verdan edited a comment on issue #3724: [AIRFLOW-2878] Fix www_rbac display issue

verdan edited a comment on issue #3724: [AIRFLOW-2878] Fix www_rbac display 
issue
URL: 
https://github.com/apache/incubator-airflow/pull/3724#issuecomment-412062054
 
 
   perfect, thanks!
   Also, I believe we need to revert this change, as this change would include 
extra spaces on each page. cc: @r39132 
   
   https://user-images.githubusercontent.com/25360476/43956876-100aba8a-9ca6-11e8-9582-e6104e8d6528.png;>
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] verdan commented on issue #3724: [AIRFLOW-2878] Fix www_rbac display issue

verdan commented on issue #3724: [AIRFLOW-2878] Fix www_rbac display issue
URL: 
https://github.com/apache/incubator-airflow/pull/3724#issuecomment-412062054
 
 
   perfect, thanks!
   Also, I believe we need to revert this change, as this change would include 
extra spaces on each page. cc: @r39132 
   
   https://user-images.githubusercontent.com/25360476/43956824-e462e952-9ca5-11e8-9d67-dc1c67ff8998.png;>
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] XD-DENG commented on issue #3724: [AIRFLOW-2878] Fix www_rbac display issue

XD-DENG commented on issue #3724: [AIRFLOW-2878] Fix www_rbac display issue
URL: 
https://github.com/apache/incubator-airflow/pull/3724#issuecomment-412061132
 
 
   Hi @verdan, I installed from the ‘tar.gz’ file directly.
   
   Will check the setting-up you introduced later. Thanks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] verdan commented on issue #3724: [AIRFLOW-2878] Fix www_rbac display issue

verdan commented on issue #3724: [AIRFLOW-2878] Fix www_rbac display issue
URL: 
https://github.com/apache/incubator-airflow/pull/3724#issuecomment-412059984
 
 
   @XD-DENG I believe you haven't built the frontend packages yet, causing all 
these issues. 
   We've implemented the `npm` and `webpack` to manage the JS libraries and 
dependencies [AIRFLOW-2691](https://issues.apache.org/jira/browse/AIRFLOW-2691) 
. Can you please follow the steps mentioned here to build the frontend packages 
and see if that fixes everything for you?
   
https://github.com/apache/incubator-airflow/blob/master/CONTRIBUTING.md#setting-up-the-node--npm-javascript-environment-only-for-www_rbac


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Closed] (AIRFLOW-559) Add support for BigQuery kwarg parameters

2018-08-10 Thread Kaxil Naik (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik closed AIRFLOW-559.
--
Resolution: Not A Problem

We don't need this as we plan to deprecate use of *args and **kwargs and we 
already have `src_fmt_configs` parameter 
(https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/bigquery_operator.py#L463)
 in the currently needed Bigquery operators where we might need to set extra 
parameters.

> Add support for BigQuery kwarg parameters
> -
>
> Key: AIRFLOW-559
> URL: https://issues.apache.org/jira/browse/AIRFLOW-559
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: contrib, gcp
>Reporter: Sam McVeety
>Assignee: Sam McVeety
>Priority: Minor
> Fix For: Airflow 1.8
>
>
> Many of the operators in 
> https://github.com/apache/incubator-airflow/tree/master/airflow/contrib/operators
>  add parameters over time, and plumbing these through multiple layers of 
> calls isn't always a high priority.
> The operators (and hooks) should support an end-to-end kwargs parameter that 
> allows for new fields (e.g. useLegacySql, defaultDataset) to be added by 
> users without needing to change the underlying code.   



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-559) Add support for BigQuery kwarg parameters



[ 
https://issues.apache.org/jira/browse/AIRFLOW-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576123#comment-16576123
 ] 

Iuliia Volkova commented on AIRFLOW-559:


[~kaxilnaik] , can you close this task to avoid confusion because as you 
mentioned in https://github.com/apache/incubator-airflow/pull/3717  "As far as 
I know, there is a plan to deprecate args and kwargs keywords in Airflow."?

 

Because of this task about the opposite thing. 

And some PRs was closed without merge based on this, for example:

[https://github.com/apache/incubator-airflow/pull/1784] 

> Add support for BigQuery kwarg parameters
> -
>
> Key: AIRFLOW-559
> URL: https://issues.apache.org/jira/browse/AIRFLOW-559
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: contrib, gcp
>Reporter: Sam McVeety
>Assignee: Sam McVeety
>Priority: Minor
> Fix For: Airflow 1.8
>
>
> Many of the operators in 
> https://github.com/apache/incubator-airflow/tree/master/airflow/contrib/operators
>  add parameters over time, and plumbing these through multiple layers of 
> calls isn't always a high priority.
> The operators (and hooks) should support an end-to-end kwargs parameter that 
> allows for new fields (e.g. useLegacySql, defaultDataset) to be added by 
> users without needing to change the underlying code.   



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (AIRFLOW-559) Add support for BigQuery kwarg parameters

2018-08-10 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576123#comment-16576123
 ] 

Iuliia Volkova edited comment on AIRFLOW-559 at 8/10/18 11:21 AM:
--

[~kaxilnaik] , can you close this task to avoid confusion because as you 
mentioned in [https://github.com/apache/incubator-airflow/pull/3717]  "As far 
as I know, there is a plan to deprecate args and kwargs keywords in Airflow."?

 

Because, this task about the opposite thing. 

And some PRs was closed without merge based on this, for example:

[https://github.com/apache/incubator-airflow/pull/1784] 


was (Author: xnuinside):
[~kaxilnaik] , can you close this task to avoid confusion because as you 
mentioned in https://github.com/apache/incubator-airflow/pull/3717  "As far as 
I know, there is a plan to deprecate args and kwargs keywords in Airflow."?

 

Because of this task about the opposite thing. 

And some PRs was closed without merge based on this, for example:

[https://github.com/apache/incubator-airflow/pull/1784] 

> Add support for BigQuery kwarg parameters
> -
>
> Key: AIRFLOW-559
> URL: https://issues.apache.org/jira/browse/AIRFLOW-559
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: contrib, gcp
>Reporter: Sam McVeety
>Assignee: Sam McVeety
>Priority: Minor
> Fix For: Airflow 1.8
>
>
> Many of the operators in 
> https://github.com/apache/incubator-airflow/tree/master/airflow/contrib/operators
>  add parameters over time, and plumbing these through multiple layers of 
> calls isn't always a high priority.
> The operators (and hooks) should support an end-to-end kwargs parameter that 
> allows for new fields (e.g. useLegacySql, defaultDataset) to be added by 
> users without needing to change the underlying code.   



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AIRFLOW-1874) Support standard SQL in Check, ValueCheck and IntervalCheck BigQuery operators

2018-08-10 Thread Kaxil Naik (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik resolved AIRFLOW-1874.
-
Resolution: Fixed

Resolved by https://github.com/apache/incubator-airflow/pull/3717

> Support standard SQL in Check, ValueCheck and IntervalCheck BigQuery operators
> --
>
> Key: AIRFLOW-1874
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1874
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: contrib, gcp, operators
>Reporter: Guillermo Rodríguez Cano
>Assignee: Iuliia Volkova
>Priority: Major
> Fix For: 2.0.0
>
>
> BigQueryCheckOperator, BigQueryValueCheckOperator and 
> BigQueryIntervalCheckOperator do not support disabling use of default legacy 
> SQL in BigQuery.
> This is a major blocker to support correct migration to standard SQL when 
> queries are complicated. For example, a query that can be queried in legacy 
> SQL may be blocked from any subsequent view done in standard SQL that this 
> view uses as the queries are bound to either standard or legacy SQL but not a 
> mix.
> These operators inherit from base ones of the same name (without the BigQuery 
> prefix) from Airflow which may make the process more complicated as the flag 
> to use standard SQL should be enabled because the underlying BigQueryHook has 
> the corresponding parameter, use_legacy_sql, set to True, when running a 
> query. But it is not possible to pass parameters all the way to it via the 
> aforementioned operators.
> The workaround of including #standardSQL and a new line before the query 
> doesn't work either as there is mismatch. BigQuery reports the following in 
> fact: "Query text specifies use_legacy_sql:false, while API options 
> specify:true"
> A workaround for queries on views using standard SQL is to persist the result 
> of the query in a temporary table, then run the check operation and 
> thereafter delete the temporary table. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-1874) Support standard SQL in Check, ValueCheck and IntervalCheck BigQuery operators



[ 
https://issues.apache.org/jira/browse/AIRFLOW-1874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576092#comment-16576092
 ] 

ASF subversion and git services commented on AIRFLOW-1874:
--

Commit 9952b23ba08fdc57c866e0f35f20a7a1a13ede74 in incubator-airflow's branch 
refs/heads/master from Yuliya Volkova
[ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=9952b23 ]

[AIRFLOW-1874] use_legacy_sql added to BigQueryCheck operators (#3717)



> Support standard SQL in Check, ValueCheck and IntervalCheck BigQuery operators
> --
>
> Key: AIRFLOW-1874
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1874
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: contrib, gcp, operators
>Reporter: Guillermo Rodríguez Cano
>Assignee: Iuliia Volkova
>Priority: Major
> Fix For: 2.0.0
>
>
> BigQueryCheckOperator, BigQueryValueCheckOperator and 
> BigQueryIntervalCheckOperator do not support disabling use of default legacy 
> SQL in BigQuery.
> This is a major blocker to support correct migration to standard SQL when 
> queries are complicated. For example, a query that can be queried in legacy 
> SQL may be blocked from any subsequent view done in standard SQL that this 
> view uses as the queries are bound to either standard or legacy SQL but not a 
> mix.
> These operators inherit from base ones of the same name (without the BigQuery 
> prefix) from Airflow which may make the process more complicated as the flag 
> to use standard SQL should be enabled because the underlying BigQueryHook has 
> the corresponding parameter, use_legacy_sql, set to True, when running a 
> query. But it is not possible to pass parameters all the way to it via the 
> aforementioned operators.
> The workaround of including #standardSQL and a new line before the query 
> doesn't work either as there is mismatch. BigQuery reports the following in 
> fact: "Query text specifies use_legacy_sql:false, while API options 
> specify:true"
> A workaround for queries on views using standard SQL is to persist the result 
> of the query in a temporary table, then run the check operation and 
> thereafter delete the temporary table. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-1874) Support standard SQL in Check, ValueCheck and IntervalCheck BigQuery operators



[ 
https://issues.apache.org/jira/browse/AIRFLOW-1874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576091#comment-16576091
 ] 

ASF GitHub Bot commented on AIRFLOW-1874:
-

kaxil closed pull request #3717: [AIRFLOW-1874] use_legacy_sql added to 
BigQueryCheck operators
URL: https://github.com/apache/incubator-airflow/pull/3717
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/airflow/contrib/operators/bigquery_check_operator.py 
b/airflow/contrib/operators/bigquery_check_operator.py
index 59ef5d377d..a9c493f4fd 100644
--- a/airflow/contrib/operators/bigquery_check_operator.py
+++ b/airflow/contrib/operators/bigquery_check_operator.py
@@ -55,21 +55,25 @@ class BigQueryCheckOperator(CheckOperator):
 :type sql: string
 :param bigquery_conn_id: reference to the BigQuery database
 :type bigquery_conn_id: string
+:param use_legacy_sql: Whether to use legacy SQL (true)
+or standard SQL (false).
+:type use_legacy_sql: boolean
 """
 
 @apply_defaults
-def __init__(
-self,
-sql,
-bigquery_conn_id='bigquery_default',
-*args,
-**kwargs):
+def __init__(self,
+ sql,
+ bigquery_conn_id='bigquery_default',
+ use_legacy_sql=True,
+ *args, **kwargs):
 super(BigQueryCheckOperator, self).__init__(sql=sql, *args, **kwargs)
 self.bigquery_conn_id = bigquery_conn_id
 self.sql = sql
+self.use_legacy_sql = use_legacy_sql
 
 def get_db_hook(self):
-return BigQueryHook(bigquery_conn_id=self.bigquery_conn_id)
+return BigQueryHook(bigquery_conn_id=self.bigquery_conn_id,
+use_legacy_sql=self.use_legacy_sql)
 
 
 class BigQueryValueCheckOperator(ValueCheckOperator):
@@ -78,20 +82,27 @@ class BigQueryValueCheckOperator(ValueCheckOperator):
 
 :param sql: the sql to be executed
 :type sql: string
+:param use_legacy_sql: Whether to use legacy SQL (true)
+or standard SQL (false).
+:type use_legacy_sql: boolean
 """
 
 @apply_defaults
-def __init__(
-self, sql, pass_value, tolerance=None,
-bigquery_conn_id='bigquery_default',
-*args, **kwargs):
+def __init__(self, sql,
+ pass_value,
+ tolerance=None,
+ bigquery_conn_id='bigquery_default',
+ use_legacy_sql=True,
+ *args, **kwargs):
 super(BigQueryValueCheckOperator, self).__init__(
 sql=sql, pass_value=pass_value, tolerance=tolerance,
 *args, **kwargs)
 self.bigquery_conn_id = bigquery_conn_id
+self.use_legacy_sql = use_legacy_sql
 
 def get_db_hook(self):
-return BigQueryHook(bigquery_conn_id=self.bigquery_conn_id)
+return BigQueryHook(bigquery_conn_id=self.bigquery_conn_id,
+use_legacy_sql=self.use_legacy_sql)
 
 
 class BigQueryIntervalCheckOperator(IntervalCheckOperator):
@@ -113,19 +124,22 @@ class 
BigQueryIntervalCheckOperator(IntervalCheckOperator):
 example 'COUNT(*)': 1.5 would require a 50 percent or less difference
 between the current day, and the prior days_back.
 :type metrics_threshold: dict
+:param use_legacy_sql: Whether to use legacy SQL (true)
+or standard SQL (false).
+:type use_legacy_sql: boolean
 """
 
 @apply_defaults
-def __init__(
-self, table, metrics_thresholds,
-date_filter_column='ds', days_back=-7,
-bigquery_conn_id='bigquery_default',
-*args, **kwargs):
+def __init__(self, table, metrics_thresholds, date_filter_column='ds',
+ days_back=-7, bigquery_conn_id='bigquery_default',
+ use_legacy_sql=True, *args, **kwargs):
 super(BigQueryIntervalCheckOperator, self).__init__(
 table=table, metrics_thresholds=metrics_thresholds,
 date_filter_column=date_filter_column, days_back=days_back,
 *args, **kwargs)
 self.bigquery_conn_id = bigquery_conn_id
+self.use_legacy_sql = use_legacy_sql
 
 def get_db_hook(self):
-return BigQueryHook(bigquery_conn_id=self.bigquery_conn_id)
+return BigQueryHook(bigquery_conn_id=self.bigquery_conn_id,
+use_legacy_sql=self.use_legacy_sql)


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact

[GitHub] kaxil closed pull request #3717: [AIRFLOW-1874] use_legacy_sql added to BigQueryCheck operators

kaxil closed pull request #3717: [AIRFLOW-1874] use_legacy_sql added to 
BigQueryCheck operators
URL: https://github.com/apache/incubator-airflow/pull/3717
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/airflow/contrib/operators/bigquery_check_operator.py 
b/airflow/contrib/operators/bigquery_check_operator.py
index 59ef5d377d..a9c493f4fd 100644
--- a/airflow/contrib/operators/bigquery_check_operator.py
+++ b/airflow/contrib/operators/bigquery_check_operator.py
@@ -55,21 +55,25 @@ class BigQueryCheckOperator(CheckOperator):
 :type sql: string
 :param bigquery_conn_id: reference to the BigQuery database
 :type bigquery_conn_id: string
+:param use_legacy_sql: Whether to use legacy SQL (true)
+or standard SQL (false).
+:type use_legacy_sql: boolean
 """
 
 @apply_defaults
-def __init__(
-self,
-sql,
-bigquery_conn_id='bigquery_default',
-*args,
-**kwargs):
+def __init__(self,
+ sql,
+ bigquery_conn_id='bigquery_default',
+ use_legacy_sql=True,
+ *args, **kwargs):
 super(BigQueryCheckOperator, self).__init__(sql=sql, *args, **kwargs)
 self.bigquery_conn_id = bigquery_conn_id
 self.sql = sql
+self.use_legacy_sql = use_legacy_sql
 
 def get_db_hook(self):
-return BigQueryHook(bigquery_conn_id=self.bigquery_conn_id)
+return BigQueryHook(bigquery_conn_id=self.bigquery_conn_id,
+use_legacy_sql=self.use_legacy_sql)
 
 
 class BigQueryValueCheckOperator(ValueCheckOperator):
@@ -78,20 +82,27 @@ class BigQueryValueCheckOperator(ValueCheckOperator):
 
 :param sql: the sql to be executed
 :type sql: string
+:param use_legacy_sql: Whether to use legacy SQL (true)
+or standard SQL (false).
+:type use_legacy_sql: boolean
 """
 
 @apply_defaults
-def __init__(
-self, sql, pass_value, tolerance=None,
-bigquery_conn_id='bigquery_default',
-*args, **kwargs):
+def __init__(self, sql,
+ pass_value,
+ tolerance=None,
+ bigquery_conn_id='bigquery_default',
+ use_legacy_sql=True,
+ *args, **kwargs):
 super(BigQueryValueCheckOperator, self).__init__(
 sql=sql, pass_value=pass_value, tolerance=tolerance,
 *args, **kwargs)
 self.bigquery_conn_id = bigquery_conn_id
+self.use_legacy_sql = use_legacy_sql
 
 def get_db_hook(self):
-return BigQueryHook(bigquery_conn_id=self.bigquery_conn_id)
+return BigQueryHook(bigquery_conn_id=self.bigquery_conn_id,
+use_legacy_sql=self.use_legacy_sql)
 
 
 class BigQueryIntervalCheckOperator(IntervalCheckOperator):
@@ -113,19 +124,22 @@ class 
BigQueryIntervalCheckOperator(IntervalCheckOperator):
 example 'COUNT(*)': 1.5 would require a 50 percent or less difference
 between the current day, and the prior days_back.
 :type metrics_threshold: dict
+:param use_legacy_sql: Whether to use legacy SQL (true)
+or standard SQL (false).
+:type use_legacy_sql: boolean
 """
 
 @apply_defaults
-def __init__(
-self, table, metrics_thresholds,
-date_filter_column='ds', days_back=-7,
-bigquery_conn_id='bigquery_default',
-*args, **kwargs):
+def __init__(self, table, metrics_thresholds, date_filter_column='ds',
+ days_back=-7, bigquery_conn_id='bigquery_default',
+ use_legacy_sql=True, *args, **kwargs):
 super(BigQueryIntervalCheckOperator, self).__init__(
 table=table, metrics_thresholds=metrics_thresholds,
 date_filter_column=date_filter_column, days_back=days_back,
 *args, **kwargs)
 self.bigquery_conn_id = bigquery_conn_id
+self.use_legacy_sql = use_legacy_sql
 
 def get_db_hook(self):
-return BigQueryHook(bigquery_conn_id=self.bigquery_conn_id)
+return BigQueryHook(bigquery_conn_id=self.bigquery_conn_id,
+use_legacy_sql=self.use_legacy_sql)


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (AIRFLOW-2884) Fix Flask SECRET_KEY security issue in www_rbac

2018-08-10 Thread Kaxil Naik (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik resolved AIRFLOW-2884.
-
   Resolution: Fixed
Fix Version/s: 2.0.0

Resolved by https://github.com/apache/incubator-airflow/pull/3729

> Fix Flask SECRET_KEY security issue in www_rbac 
> 
>
> Key: AIRFLOW-2884
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2884
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: webapp, webserver
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Critical
> Fix For: 2.0.0
>
>
> Flask secret key should be as random as possible, while it's not in Airflow 
> Flask App.
> This issue was fixed for *www* in ticket 
> https://issues.apache.org/jira/browse/AIRFLOW-2809 (merged in PR 
> [https://github.com/apache/incubator-airflow/pull/3651)] .
> But this issue was not fixed for *www_rbac* yet.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] kaxil commented on a change in pull request #3725: [AIRFLOW-2877] Make docs site URL consistent everywhere

kaxil commented on a change in pull request #3725: [AIRFLOW-2877] Make docs 
site URL consistent everywhere
URL: https://github.com/apache/incubator-airflow/pull/3725#discussion_r209215776
 
 

 ##
 File path: README.md
 ##
 @@ -24,9 +24,9 @@ monitor progress, and troubleshoot issues when needed.
 
 ## Getting started
 
-Please visit the Airflow Platform documentation (latest **stable** release) 
for help with [installing 
Airflow](https://airflow.incubator.apache.org/installation.html), getting a 
[quick start](https://airflow.incubator.apache.org/start.html), or a more 
complete [tutorial](https://airflow.incubator.apache.org/tutorial.html).
+Please visit the Airflow Platform documentation (latest **stable** release) 
for help with [installing 
Airflow](https://airflow.apache.org/installation.html), getting a [quick 
start](https://airflow.apache.org/start.html), or a more complete 
[tutorial](https://airflow.apache.org/tutorial.html).
 
 Review comment:
   I have raised the issue with Apache Infra over here: 
https://issues.apache.org/jira/browse/INFRA-16895


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] kaxil commented on a change in pull request #3725: [AIRFLOW-2877] Make docs site URL consistent everywhere

kaxil commented on a change in pull request #3725: [AIRFLOW-2877] Make docs 
site URL consistent everywhere
URL: https://github.com/apache/incubator-airflow/pull/3725#discussion_r209203140
 
 

 ##
 File path: README.md
 ##
 @@ -24,9 +24,9 @@ monitor progress, and troubleshoot issues when needed.
 
 ## Getting started
 
-Please visit the Airflow Platform documentation (latest **stable** release) 
for help with [installing 
Airflow](https://airflow.incubator.apache.org/installation.html), getting a 
[quick start](https://airflow.incubator.apache.org/start.html), or a more 
complete [tutorial](https://airflow.incubator.apache.org/tutorial.html).
+Please visit the Airflow Platform documentation (latest **stable** release) 
for help with [installing 
Airflow](https://airflow.apache.org/installation.html), getting a [quick 
start](https://airflow.apache.org/start.html), or a more complete 
[tutorial](https://airflow.apache.org/tutorial.html).
 
 Review comment:
   Cool. I will raise this today.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ashb commented on a change in pull request #3725: [AIRFLOW-2877] Make docs site URL consistent everywhere

ashb commented on a change in pull request #3725: [AIRFLOW-2877] Make docs site 
URL consistent everywhere
URL: https://github.com/apache/incubator-airflow/pull/3725#discussion_r209202215
 
 

 ##
 File path: README.md
 ##
 @@ -24,9 +24,9 @@ monitor progress, and troubleshoot issues when needed.
 
 ## Getting started
 
-Please visit the Airflow Platform documentation (latest **stable** release) 
for help with [installing 
Airflow](https://airflow.incubator.apache.org/installation.html), getting a 
[quick start](https://airflow.incubator.apache.org/start.html), or a more 
complete [tutorial](https://airflow.incubator.apache.org/tutorial.html).
+Please visit the Airflow Platform documentation (latest **stable** release) 
for help with [installing 
Airflow](https://airflow.apache.org/installation.html), getting a [quick 
start](https://airflow.apache.org/start.html), or a more complete 
[tutorial](https://airflow.apache.org/tutorial.html).
 
-Documentation of GitHub master (latest development branch): [ReadTheDocs 
Documentation](https://airflow.readthedocs.io/en/latest/)
+Documentation of GitHub master (latest development branch): [Read the Docs 
Documentation](https://airflow.readthedocs.io/en/latest/)
 
 Review comment:
   We should mention this distinction on the content on airflow.apache.org too, 
or use something like https://robpol86.github.io/sphinxcontrib-versioning/ to 
be able to host multiple versions on the a.a.o site too.
   
   Again, unrelated to this PR.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ashb commented on a change in pull request #3725: [AIRFLOW-2877] Make docs site URL consistent everywhere

ashb commented on a change in pull request #3725: [AIRFLOW-2877] Make docs site 
URL consistent everywhere
URL: https://github.com/apache/incubator-airflow/pull/3725#discussion_r209201782
 
 

 ##
 File path: README.md
 ##
 @@ -24,9 +24,9 @@ monitor progress, and troubleshoot issues when needed.
 
 ## Getting started
 
-Please visit the Airflow Platform documentation (latest **stable** release) 
for help with [installing 
Airflow](https://airflow.incubator.apache.org/installation.html), getting a 
[quick start](https://airflow.incubator.apache.org/start.html), or a more 
complete [tutorial](https://airflow.incubator.apache.org/tutorial.html).
+Please visit the Airflow Platform documentation (latest **stable** release) 
for help with [installing 
Airflow](https://airflow.apache.org/installation.html), getting a [quick 
start](https://airflow.apache.org/start.html), or a more complete 
[tutorial](https://airflow.apache.org/tutorial.html).
 
-Documentation of GitHub master (latest development branch): [ReadTheDocs 
Documentation](https://airflow.readthedocs.io/en/latest/)
+Documentation of GitHub master (latest development branch): [Read the Docs 
Documentation](https://airflow.readthedocs.io/en/latest/)
 
 Review comment:
   Unrelated to this PR (so sorry for hijacking) but I would be happier if the 
default version you get upon visiting https://airflow.readthedocs.io (i.e. from 
Google) is for the same version you get if you `pip install apache-airflow`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ashb commented on a change in pull request #3725: [AIRFLOW-2877] Make docs site URL consistent everywhere

ashb commented on a change in pull request #3725: [AIRFLOW-2877] Make docs site 
URL consistent everywhere
URL: https://github.com/apache/incubator-airflow/pull/3725#discussion_r209201237
 
 

 ##
 File path: README.md
 ##
 @@ -24,9 +24,9 @@ monitor progress, and troubleshoot issues when needed.
 
 ## Getting started
 
-Please visit the Airflow Platform documentation (latest **stable** release) 
for help with [installing 
Airflow](https://airflow.incubator.apache.org/installation.html), getting a 
[quick start](https://airflow.incubator.apache.org/start.html), or a more 
complete [tutorial](https://airflow.incubator.apache.org/tutorial.html).
+Please visit the Airflow Platform documentation (latest **stable** release) 
for help with [installing 
Airflow](https://airflow.apache.org/installation.html), getting a [quick 
start](https://airflow.apache.org/start.html), or a more complete 
[tutorial](https://airflow.apache.org/tutorial.html).
 
 Review comment:
   We can open an ticket with Apache INFRA to enable a webhook for this.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] seratch opened a new pull request #3732: Fix typos detected by github.com/client9/misspell

seratch opened a new pull request #3732: Fix typos detected by 
github.com/client9/misspell
URL: https://github.com/apache/incubator-airflow/pull/3732
 
 
   ### Description
   
   Fixing typos is sometimes very hard. It's not so easy to visually review 
them. Recently, I discovered a very useful tool for it, 
[misspell](https://github.com/client9/misspell). 
   
   This pull request fixes minor typos detected by 
[misspell](https://github.com/client9/misspell) except for the false positives. 
If you would like me to work on other files as well, let me know. 
   
    before
   
   ```
   $ misspell airflow/ | grep -v CHANGELOG.txt | grep -v www
   2018/08/10 18:14:00 Unable to stat "airflow/www/static/docs": stat 
airflow/www/static/docs: no such file or directory
   2018/08/10 18:14:00 Unable to stat "airflow/www_rbac/static/docs": stat 
airflow/www_rbac/static/docs: no such file or directory
   airflow/contrib/example_dags/example_databricks_operator.py:35:22: 
"succesful" is a misspelling of "successful"
   airflow/contrib/hooks/emr_hook.py:26:47: "neccessary" is a misspelling of 
"necessary"
   airflow/contrib/hooks/azure_fileshare_hook.py:103:21: "direcotry" is a 
misspelling of "directory"
   airflow/contrib/hooks/gcp_dataproc_hook.py:238:60: "compatability" is a 
misspelling of "compatibility"
   airflow/contrib/hooks/bigquery_hook.py:630:40: "paramaters" is a misspelling 
of "parameters"
   airflow/contrib/hooks/qubole_hook.py:128:31: "commmand" is a misspelling of 
"command"
   airflow/contrib/hooks/salesforce_hook.py:56:32: "conenction" is a 
misspelling of "connection"
   airflow/contrib/hooks/salesforce_hook.py:63:41: "SECRUITY" is a misspelling 
of "SECURITY"
   airflow/contrib/operators/gcs_to_bq.py:89:29: "Thsi" is a misspelling of 
"This"
   airflow/contrib/operators/mlengine_operator_utils.py:163:69: "prediciton" is 
a misspelling of "prediction"
   airflow/contrib/operators/qubole_check_operator.py:31:69: "Commmand" is a 
misspelling of "Command"
   
airflow/contrib/plugins/metastore_browser/templates/metastore_browser/table.html:31:106:
 "Atributes" is a misspelling of "Attributes"
   airflow/hooks/mysql_hook.py:124:27: "seperately" is a misspelling of 
"separately"
   airflow/operators/hive_to_druid.py:167:19: "compatibilty" is a misspelling 
of "compatibility"
   airflow/sensors/hdfs_sensor.py:91:20: "extentions" is a misspelling of 
"extensions"
   airflow/sensors/hdfs_sensor.py:94:24: "extentions" is a misspelling of 
"extensions"
   airflow/sensors/hdfs_sensor.py:96:55: "extentions" is a misspelling of 
"extensions"
   airflow/models.py:945:71: "acutally" is a misspelling of "actually"
   ```
   
    after
   
   ```
   $ misspell airflow/ | grep -v CHANGELOG.txt | grep -v www
   2018/08/10 18:13:50 Unable to stat "airflow/www/static/docs": stat 
airflow/www/static/docs: no such file or directory
   2018/08/10 18:13:50 Unable to stat "airflow/www_rbac/static/docs": stat 
airflow/www_rbac/static/docs: no such file or directory
   ```
   
   
   ### Code Quality
   
   - [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   
   hmm, I am not sure if I can fix this.
   
   ```
   $ git diff upstream/master -u -- "*.py" | flake8 --dif
   fatal: bad revision 'upstream/master'
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2885) A Bug in www_rbac.utils.get_params



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575938#comment-16575938
 ] 

ASF GitHub Bot commented on AIRFLOW-2885:
-

XD-DENG opened a new pull request #3731: [AIRFLOW-2885] Fix a bug in 
www_rbac.utils.get_params
URL: https://github.com/apache/incubator-airflow/pull/3731
 
 
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-2885
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   `get_params(page=0, search="abc",showPaused=False)` returns 
`search=abc=False`, while it's supposed to return 
`page=0=abc=False` (`page` is 0-indexed).
   
   This issue arose since `0` is considered as `False` by Python in conditional 
statement. `elif v` will not continue when `v` is `0` here, while it's supposed 
to continue.
   
   A test is added as well. 
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> A Bug in www_rbac.utils.get_params
> --
>
> Key: AIRFLOW-2885
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2885
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: webserver
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Critical
>
> *get_params(page=0, search="abc",showPaused=False)* returns 
> "_search=abc=False_", while it's supposed to return 
> "page=0=abc=False".
> This is because Python takes 0 as False when it's used in a conditional 
> statement.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] XD-DENG opened a new pull request #3731: [AIRFLOW-2885] Fix a bug in www_rbac.utils.get_params

XD-DENG opened a new pull request #3731: [AIRFLOW-2885] Fix a bug in 
www_rbac.utils.get_params
URL: https://github.com/apache/incubator-airflow/pull/3731
 
 
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-2885
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   `get_params(page=0, search="abc",showPaused=False)` returns 
`search=abc=False`, while it's supposed to return 
`page=0=abc=False` (`page` is 0-indexed).
   
   This issue arose since `0` is considered as `False` by Python in conditional 
statement. `elif v` will not continue when `v` is `0` here, while it's supposed 
to continue.
   
   A test is added as well. 
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-2885) A Bug in www_rbac.utils.get_params

2018-08-10 Thread Xiaodong DENG (JIRA)

Xiaodong DENG created AIRFLOW-2885:
--

 Summary: A Bug in www_rbac.utils.get_params
 Key: AIRFLOW-2885
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2885
 Project: Apache Airflow
  Issue Type: Bug
  Components: webserver
Reporter: Xiaodong DENG
Assignee: Xiaodong DENG


*get_params(page=0, search="abc",showPaused=False)* returns 
"_search=abc=False_", while it's supposed to return 
"page=0=abc=False".

This is because Python takes 0 as False when it's used in a conditional 
statement.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] Fokko commented on issue #3691: [AIRFLOW-2846] Add missing python test dependency to setup.py

Fokko commented on issue #3691: [AIRFLOW-2846] Add missing python test 
dependency to setup.py
URL: 
https://github.com/apache/incubator-airflow/pull/3691#issuecomment-412000219
 
 
   I'd say that we would need to remove this tox install as well: 
https://github.com/apache/incubator-airflow/blob/master/.travis.yml#L98


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] Fokko commented on a change in pull request #3730: [AIRFLOW-2882] Add import and export for pool cli using JSON

Fokko commented on a change in pull request #3730: [AIRFLOW-2882] Add import 
and export for pool cli using JSON
URL: https://github.com/apache/incubator-airflow/pull/3730#discussion_r209166353
 
 

 ##
 File path: tests/cli/test_cli.py
 ##
 @@ -165,3 +166,38 @@ def test_local_run(self):
 ti.refresh_from_db()
 state = ti.current_state()
 self.assertEqual(state, State.SUCCESS)
+
+def test_cli_pool_import_export(self):
+pool_config_input = {
+"s3_pool": {
+"description": "This is my test s3_pool",
+"slots": 5
+},
+"s3_pool2": {
+"description": "This is my test s3_pool",
+"slots": 8
+}
+}
+with open('pool_import.json', mode='w', encoding='utf-8') as f:
+json.dump(pool_config_input, f)
+process_import = psutil.Popen(["airflow", "pool", "-i", 
"pool_import.json"])
+sleep(3)  # wait for webserver to start
 
 Review comment:
   Please poll if the process is available. This makes the test a bit flaky. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] kaxil commented on a change in pull request #3725: [AIRFLOW-2877] Make docs site URL consistent everywhere

kaxil commented on a change in pull request #3725: [AIRFLOW-2877] Make docs 
site URL consistent everywhere
URL: https://github.com/apache/incubator-airflow/pull/3725#discussion_r209161488
 
 

 ##
 File path: README.md
 ##
 @@ -24,9 +24,9 @@ monitor progress, and troubleshoot issues when needed.
 
 ## Getting started
 
-Please visit the Airflow Platform documentation (latest **stable** release) 
for help with [installing 
Airflow](https://airflow.incubator.apache.org/installation.html), getting a 
[quick start](https://airflow.incubator.apache.org/start.html), or a more 
complete [tutorial](https://airflow.incubator.apache.org/tutorial.html).
+Please visit the Airflow Platform documentation (latest **stable** release) 
for help with [installing 
Airflow](https://airflow.apache.org/installation.html), getting a [quick 
start](https://airflow.apache.org/start.html), or a more complete 
[tutorial](https://airflow.apache.org/tutorial.html).
 
 Review comment:
   @r39132 Hi Sid, we had discussed this at the start of this year about using 
the webhooks but as we didn't have permissions to create hooks on Github, the 
current process is manual i.e. I try and trigger the build manually regularly.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao edited a comment on issue #3728: [AIRFLOW-2883] Not search dag owner if owners are missing