[GitHub] XD-DENG commented on issue #3724: [AIRFLOW-2878] Fix www_rbac display issue
XD-DENG commented on issue #3724: [AIRFLOW-2878] Fix www_rbac display issue URL: https://github.com/apache/incubator-airflow/pull/3724#issuecomment-412246789 Hi @verdan , FYI, this commit is already reverted via https://github.com/apache/incubator-airflow/pull/3737 . Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] XD-DENG edited a comment on issue #3738: [AIRFLOW-2886] Secure Flask SECRET_KEY
XD-DENG edited a comment on issue #3738: [AIRFLOW-2886] Secure Flask SECRET_KEY URL: https://github.com/apache/incubator-airflow/pull/3738#issuecomment-412227570 Hi @feng-tao , in cluster mode, given the `.cfg` files are initiated on different nodes, the SECRET_KEY will be different. That's also why Craig mentioned it fixes now on a single webserver instance (single worker or multiple workers), but still having `CSRF` error on a cluster of webservers ([LINK](https://issues.apache.org/jira/browse/AIRFLOW-2866?focusedCommentId=16576717=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16576717)). I insist in that we should have random SECRET_KEY for the webserver. If we use the previous way, that is to have a default value in the template, it's gonna be risky. If users want to use a cluster of webservers, they need to either - make sure that SECRET_KEYs are consistent across nodes (to manually specify). This is similar to what people need to do for `sql_alchemy_conn` if they're running multiple nodes for webserver. OR - apply `ip-hash` strategy for the cluster load-balancing, rather than `round-robin`. Please let me know your thoughts? cc: @Fokko @kaxil This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil commented on issue #3733: [WIP] [AIRFLOW-491] Add cache parameter in BigQuery query method
kaxil commented on issue #3733: [WIP] [AIRFLOW-491] Add cache parameter in BigQuery query method URL: https://github.com/apache/incubator-airflow/pull/3733#issuecomment-412239088 Don't remove any previous parameters, just add support to add new ones This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] XD-DENG edited a comment on issue #3738: [AIRFLOW-2886] Secure Flask SECRET_KEY
XD-DENG edited a comment on issue #3738: [AIRFLOW-2886] Secure Flask SECRET_KEY URL: https://github.com/apache/incubator-airflow/pull/3738#issuecomment-412227570 Hi @feng-tao , in cluster mode, given the `.cfg` files are initiated on different nodes, the SECRET_KEY will be different. That's also why Craig mentioned it fixes now on a single webserver instance (single worker or multiple workers), but still having `CSRF` error on a cluster of webservers ([LINK](https://issues.apache.org/jira/browse/AIRFLOW-2866?focusedCommentId=16576717=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16576717)). I insist in that we should have random SECRET_KEY for the webserver. If we use the previous way, that is to have a default value in the template, it's gonna be risky. If users want to use a cluster of webservers, they need to either - make sure that SECRET_KEYs are consistent across nodes (to manually specify) or - apply `ip-hash` strategy for the cluster load-balancing, rather than `round-robin`. Please let me know your thoughts? cc: @Fokko @kaxil This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] XD-DENG edited a comment on issue #3738: [AIRFLOW-2886] Secure Flask SECRET_KEY
XD-DENG edited a comment on issue #3738: [AIRFLOW-2886] Secure Flask SECRET_KEY URL: https://github.com/apache/incubator-airflow/pull/3738#issuecomment-412227570 Hi @feng-tao , in cluster mode, given the `.cfg` files are initiated on different nodes, the SECRET_KEY will be different. That's also why Craig mentioned it fixes now on a single webserver instance (single worker or multiple workers), but doesn't work on a cluster of webservers ([LINK](https://issues.apache.org/jira/browse/AIRFLOW-2866?focusedCommentId=16576717=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16576717)). I insist in that we should have random SECRET_KEY for the webserver. If we use the previous way, that is to have a default value in the template, it's gonna be risky. If users want to use a cluster of webservers, they need to either - make sure that SECRET_KEYs are consistent across nodes (to manually specify) or - apply `ip-hash` strategy for the cluster load-balancing, rather than `round-robin`. Please let me know your thoughts? cc: @Fokko @kaxil This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] XD-DENG edited a comment on issue #3738: [AIRFLOW-2886] Secure Flask SECRET_KEY
XD-DENG edited a comment on issue #3738: [AIRFLOW-2886] Secure Flask SECRET_KEY URL: https://github.com/apache/incubator-airflow/pull/3738#issuecomment-412227570 Hi @feng-tao , in cluster mode, given the `.cfg` files are initiated on different nodes, the SECRET_KEY will be different. That's also why Craig mentioned it fixes now on a single webserver instance (multiple workers), but doesn't work on a cluster of webservers ([LINK](https://issues.apache.org/jira/browse/AIRFLOW-2866?focusedCommentId=16576717=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16576717)). I insist in that we should have random SECRET_KEY for the webserver. If we use the previous way, that is to have a default value in the template, it's gonna be risky. If users want to use a cluster of webservers, they need to either - make sure that SECRET_KEYs are consistent across nodes (to manually specify) or - apply `ip-hash` strategy for the cluster load-balancing, rather than `round-robin`. Please let me know your thoughts? cc: @Fokko @kaxil This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] XD-DENG commented on issue #3738: [AIRFLOW-2886] Secure Flask SECRET_KEY
XD-DENG commented on issue #3738: [AIRFLOW-2886] Secure Flask SECRET_KEY URL: https://github.com/apache/incubator-airflow/pull/3738#issuecomment-412227570 Hi @feng-tao , in cluster mode, given the `.cfg` files are initiated on different nodes, the SECRET_KEY will be different. That's also why Craig mentioned it fixes now on a single webserver instance (multiple workers), but doesn't work on a cluster of webservers ([link](https://issues.apache.org/jira/browse/AIRFLOW-2866?focusedCommentId=16576717=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16576717)). I insist in that we should have random SECRET_KEY for the webserver. If we use the previous way, that is to have a default value in the template, it's gonna be risky. If users want to use a cluster of webservers, they need to either - make sure that SECRET_KEYs are consistent across nodes (to manually specify) or - apply `ip-hash` strategy for the cluster load-balancing. Please let me know your thoughts? cc: @Fokko @kaxil This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] amir656 commented on issue #3739: [AIRFLOW-2868] Extend mesos_executor
amir656 commented on issue #3739: [AIRFLOW-2868] Extend mesos_executor URL: https://github.com/apache/incubator-airflow/pull/3739#issuecomment-412219056 I am not sure where to add details about this change to the documentation. There doesn't seem to be documentation about the executor_config, or even the kubernetes executor at all, which I would have hoped to extend. Please let me know if you have any ideas about where to add documentation This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-2868) Mesos Executor should use executor_config to specify CPU, Memory and Docker image on the task level
[ https://issues.apache.org/jira/browse/AIRFLOW-2868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amir Shahatit updated AIRFLOW-2868: --- External issue URL: https://github.com/apache/incubator-airflow/pull/3739 > Mesos Executor should use executor_config to specify CPU, Memory and Docker > image on the task level > --- > > Key: AIRFLOW-2868 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2868 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib >Affects Versions: 1.10, 1.10.1 >Reporter: Amir Shahatit >Assignee: Amir Shahatit >Priority: Major > Original Estimate: 72h > Remaining Estimate: 72h > > Executor_config was added as a part of > [AIRFLOW-1314|https://github.com/apache/incubator-airflow/commit/c0920efc012468681cff3d3c9cfe25c7381dc976]. > This task extends the mesosExecutor to make use of specified executor > configs to pass on resource requirements (CPU/Memory) as well as docker > images on the task level. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2868) Mesos Executor should use executor_config to specify CPU, Memory and Docker image on the task level
[ https://issues.apache.org/jira/browse/AIRFLOW-2868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576881#comment-16576881 ] ASF GitHub Bot commented on AIRFLOW-2868: - amir656 opened a new pull request #3739: [AIRFLOW-2868] Extend mesos_executor URL: https://github.com/apache/incubator-airflow/pull/3739 Make sure you have checked _all_ steps below. ### Jira - [ ] My PR addresses the following [AIRFLOW-2868](https://issues.apache.org/jira/browse/AIRFLOW-2868) issue and references it in the PR title. ### Description - [ ] I extended the mesos_executor to be able to execute different tasks with different CPU and Memory requirements, as well as different docker images. By providing an executor_config dictionary, users can specify these tasks in their operators. This builds off of the executor config created for the kubernetes_executor. ### Tests - [ ] My PR added to tests/contrib/executors/test_mesos_executor.py ### Commits - [ ] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" ### Documentation - [ ] In case of new functionality, my PR adds documentation that describes how to use it. ### Code Quality - [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Mesos Executor should use executor_config to specify CPU, Memory and Docker > image on the task level > --- > > Key: AIRFLOW-2868 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2868 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib >Affects Versions: 1.10, 1.10.1 >Reporter: Amir Shahatit >Assignee: Amir Shahatit >Priority: Major > Original Estimate: 72h > Remaining Estimate: 72h > > Executor_config was added as a part of > [AIRFLOW-1314|https://github.com/apache/incubator-airflow/commit/c0920efc012468681cff3d3c9cfe25c7381dc976]. > This task extends the mesosExecutor to make use of specified executor > configs to pass on resource requirements (CPU/Memory) as well as docker > images on the task level. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] amir656 opened a new pull request #3739: [AIRFLOW-2868] Extend mesos_executor
amir656 opened a new pull request #3739: [AIRFLOW-2868] Extend mesos_executor URL: https://github.com/apache/incubator-airflow/pull/3739 Make sure you have checked _all_ steps below. ### Jira - [ ] My PR addresses the following [AIRFLOW-2868](https://issues.apache.org/jira/browse/AIRFLOW-2868) issue and references it in the PR title. ### Description - [ ] I extended the mesos_executor to be able to execute different tasks with different CPU and Memory requirements, as well as different docker images. By providing an executor_config dictionary, users can specify these tasks in their operators. This builds off of the executor config created for the kubernetes_executor. ### Tests - [ ] My PR added to tests/contrib/executors/test_mesos_executor.py ### Commits - [ ] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" ### Documentation - [ ] In case of new functionality, my PR adds documentation that describes how to use it. ### Code Quality - [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-2887) Add to BigQueryBaseCursor methods for creating and updating datasets
Iuliia Volkova created AIRFLOW-2887: --- Summary: Add to BigQueryBaseCursor methods for creating and updating datasets Key: AIRFLOW-2887 URL: https://issues.apache.org/jira/browse/AIRFLOW-2887 Project: Apache Airflow Issue Type: New Feature Reporter: Iuliia Volkova Assignee: Iuliia Volkova In BigQueryBaseCursor exist only: def delete_dataset(self, project_id, dataset_id) And there are no hooks to create([https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets/insert)] and update datasets ([https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets/update]) [~kaxilnaik], or I'm not right? If it's so, could I add methods and operators for those actions? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2887) Add to BigQueryBaseCursor methods for creating and updating datasets
[ https://issues.apache.org/jira/browse/AIRFLOW-2887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Iuliia Volkova updated AIRFLOW-2887: Description: In BigQueryBaseCursor exist only: def delete_dataset(self, project_id, dataset_id) And there are no hooks to create([https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets/insert)] and update datasets ([https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets/update]) [~kaxilnaik], or I'm not right? If it's so, could I add methods and operators for those actions? was: In BigQueryBaseCursor exist only: def delete_dataset(self, project_id, dataset_id) And there are no hooks to create([https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets/insert)] and update datasets ([https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets/update]) [~kaxilnaik], or I'm not right? If it's so, could I add methods and operators for those actions? > Add to BigQueryBaseCursor methods for creating and updating datasets > > > Key: AIRFLOW-2887 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2887 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Iuliia Volkova >Assignee: Iuliia Volkova >Priority: Minor > > In BigQueryBaseCursor exist only: > def delete_dataset(self, project_id, dataset_id) > And there are no hooks to > create([https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets/insert)] > and update datasets > ([https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets/update]) > [~kaxilnaik], or I'm not right? > If it's so, could I add methods and operators for those actions? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] feng-tao commented on issue #3738: [AIRFLOW-2886] Secure Flask SECRET_KEY
feng-tao commented on issue #3738: [AIRFLOW-2886] Secure Flask SECRET_KEY URL: https://github.com/apache/incubator-airflow/pull/3738#issuecomment-412192087 hey @XD-DENG , it seems that it will still be an issue if we have a cluster for webservers based on the comment in https://issues.apache.org/jira/browse/AIRFLOW-2866. What do you think? cc @Fokko , @kaxil as they comment / approve the original prs. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-2866) Missing CSRF Token Error on Web UI Create/Update Operations
[ https://issues.apache.org/jira/browse/AIRFLOW-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576793#comment-16576793 ] Tao Feng commented on AIRFLOW-2866: --- Thanks [~craigf] . Let me comment that back to ticket / pr. > Missing CSRF Token Error on Web UI Create/Update Operations > --- > > Key: AIRFLOW-2866 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2866 > Project: Apache Airflow > Issue Type: Bug > Components: webapp >Reporter: Jasper Kahn >Priority: Major > > Attempting to modify or delete many resources (such as Connections or Users) > results in a 400 from the webserver: > {quote}{{Bad Request}} > {{The CSRF session token is missing.}}{quote} > Logs report: > {quote}{{[2018-08-07 18:45:15,771] \{csrf.py:251} INFO - The CSRF session > token is missing.}} > {{192.168.9.1 - - [07/Aug/2018:18:45:15 +] "POST > /admin/connection/delete/ HTTP/1.1" 400 150 > "http://localhost:8081/admin/connection/; "Mozilla/5.0 (X11; Linux x86_64) > AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.84 > Safari/537.36"}}{quote} > Chrome dev tools show the CSRF token is present in the request payload. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2866) Missing CSRF Token Error on Web UI Create/Update Operations
[ https://issues.apache.org/jira/browse/AIRFLOW-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576717#comment-16576717 ] Craig Forster commented on AIRFLOW-2866: [~TaoFeng] Running locally with that branch seems to fix things. I can toggle the on/off state for a DAG without the callbacks being rejected for CSRF errors. This is for a single web server instance only. I don't believe this will resolve the issue for a cluster of webservers serving requests in a round-robin fashion, though. Each webserver instance will still have a different secret key and reject CSRF tokens from other cluster members. > Missing CSRF Token Error on Web UI Create/Update Operations > --- > > Key: AIRFLOW-2866 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2866 > Project: Apache Airflow > Issue Type: Bug > Components: webapp >Reporter: Jasper Kahn >Priority: Major > > Attempting to modify or delete many resources (such as Connections or Users) > results in a 400 from the webserver: > {quote}{{Bad Request}} > {{The CSRF session token is missing.}}{quote} > Logs report: > {quote}{{[2018-08-07 18:45:15,771] \{csrf.py:251} INFO - The CSRF session > token is missing.}} > {{192.168.9.1 - - [07/Aug/2018:18:45:15 +] "POST > /admin/connection/delete/ HTTP/1.1" 400 150 > "http://localhost:8081/admin/connection/; "Mozilla/5.0 (X11; Linux x86_64) > AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.84 > Safari/537.36"}}{quote} > Chrome dev tools show the CSRF token is present in the request payload. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] troychen728 commented on a change in pull request #3658: [AIRFLOW-2524] Add Amazon SageMaker Training
troychen728 commented on a change in pull request #3658: [AIRFLOW-2524] Add Amazon SageMaker Training URL: https://github.com/apache/incubator-airflow/pull/3658#discussion_r209352239 ## File path: airflow/contrib/hooks/sagemaker_hook.py ## @@ -0,0 +1,239 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +import copy +import time +from botocore.exceptions import ClientError + +from airflow.exceptions import AirflowException +from airflow.contrib.hooks.aws_hook import AwsHook +from airflow.hooks.S3_hook import S3Hook + + +class SageMakerHook(AwsHook): +""" +Interact with Amazon SageMaker. +sagemaker_conn_id is required for using +the config stored in db for training/tuning +""" + +def __init__(self, + sagemaker_conn_id=None, + use_db_config=False, + region_name=None, + check_interval=5, + max_ingestion_time=None, + *args, **kwargs): +super(SageMakerHook, self).__init__(*args, **kwargs) +self.sagemaker_conn_id = sagemaker_conn_id +self.use_db_config = use_db_config +self.region_name = region_name +self.check_interval = check_interval +self.max_ingestion_time = max_ingestion_time +self.conn = self.get_conn() + +def check_for_url(self, s3url): +""" +check if the s3url exists +:param s3url: S3 url +:type s3url:str +:return: bool +""" +bucket, key = S3Hook.parse_s3_url(s3url) +s3hook = S3Hook(aws_conn_id=self.aws_conn_id) +if not s3hook.check_for_bucket(bucket_name=bucket): +raise AirflowException( +"The input S3 Bucket {} does not exist ".format(bucket)) +if not s3hook.check_for_key(key=key, bucket_name=bucket): +raise AirflowException("The input S3 Key {} does not exist in the Bucket" + .format(s3url, bucket)) +return True + +def check_valid_training_input(self, training_config): +""" +Run checks before a training starts +:param config: training_config +:type config: dict +:return: None +""" +for channel in training_config['InputDataConfig']: +self.check_for_url(channel['DataSource'] + ['S3DataSource']['S3Uri']) + +def check_valid_tuning_input(self, tuning_config): +""" +Run checks before a tuning job starts +:param config: tuning_config +:type config: dict +:return: None +""" +for channel in tuning_config['TrainingJobDefinition']['InputDataConfig']: +self.check_for_url(channel['DataSource'] + ['S3DataSource']['S3Uri']) + +def check_status(self, non_terminal_states, + failed_state, key, + describe_function, *args): +""" +:param non_terminal_states: the set of non_terminal states +:type non_terminal_states: dict +:param failed_state: the set of failed states +:type failed_state: dict +:param key: the key of the response dict +that points to the state +:type key: string +:param describe_function: the function used to retrieve the status +:type describe_function: python callable +:param args: the arguments for the function +:return: None +""" +sec = 0 +running = True + +while running: + +sec = sec + self.check_interval + +if self.max_ingestion_time and sec > self.max_ingestion_time: +# ensure that the job gets killed if the max ingestion time is exceeded +raise AirflowException("SageMaker job took more than " + "%s seconds", self.max_ingestion_time) + +time.sleep(self.check_interval) +try: +status = describe_function(*args)[key] +self.log.info("Job still running for %s seconds... " + "current status is %s" % (sec, status)) +
[jira] [Commented] (AIRFLOW-2866) Missing CSRF Token Error on Web UI Create/Update Operations
[ https://issues.apache.org/jira/browse/AIRFLOW-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576674#comment-16576674 ] Tao Feng commented on AIRFLOW-2866: --- [~craigf] , I don't have the setup in my mac. Could you apply this pr([https://github.com/apache/incubator-airflow/pull/3738)] and see if it solves the issue? > Missing CSRF Token Error on Web UI Create/Update Operations > --- > > Key: AIRFLOW-2866 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2866 > Project: Apache Airflow > Issue Type: Bug > Components: webapp >Reporter: Jasper Kahn >Priority: Major > > Attempting to modify or delete many resources (such as Connections or Users) > results in a 400 from the webserver: > {quote}{{Bad Request}} > {{The CSRF session token is missing.}}{quote} > Logs report: > {quote}{{[2018-08-07 18:45:15,771] \{csrf.py:251} INFO - The CSRF session > token is missing.}} > {{192.168.9.1 - - [07/Aug/2018:18:45:15 +] "POST > /admin/connection/delete/ HTTP/1.1" 400 150 > "http://localhost:8081/admin/connection/; "Mozilla/5.0 (X11; Linux x86_64) > AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.84 > Safari/537.36"}}{quote} > Chrome dev tools show the CSRF token is present in the request payload. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2866) Missing CSRF Token Error on Web UI Create/Update Operations
[ https://issues.apache.org/jira/browse/AIRFLOW-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576657#comment-16576657 ] Craig Forster commented on AIRFLOW-2866: Yes, it looks related to that. I explicitly set a secret_key for my cluster of web servers and it resolved the issue. > Missing CSRF Token Error on Web UI Create/Update Operations > --- > > Key: AIRFLOW-2866 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2866 > Project: Apache Airflow > Issue Type: Bug > Components: webapp >Reporter: Jasper Kahn >Priority: Major > > Attempting to modify or delete many resources (such as Connections or Users) > results in a 400 from the webserver: > {quote}{{Bad Request}} > {{The CSRF session token is missing.}}{quote} > Logs report: > {quote}{{[2018-08-07 18:45:15,771] \{csrf.py:251} INFO - The CSRF session > token is missing.}} > {{192.168.9.1 - - [07/Aug/2018:18:45:15 +] "POST > /admin/connection/delete/ HTTP/1.1" 400 150 > "http://localhost:8081/admin/connection/; "Mozilla/5.0 (X11; Linux x86_64) > AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.84 > Safari/537.36"}}{quote} > Chrome dev tools show the CSRF token is present in the request payload. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2844) Airflow Logs BrokenPipeException
[ https://issues.apache.org/jira/browse/AIRFLOW-2844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576658#comment-16576658 ] Kyle Bridenstine commented on AIRFLOW-2844: --- *Root Cause Of Issue:* * [https://github.com/apache/incubator-airflow/pull/2484] *Stackoverflow Posts That Helped Solve The Issue:* * [https://stackoverflow.com/questions/51365911/airflow-logs-brokenpipeexception/51790409#51790409] * [https://stackoverflow.com/questions/51775370/airflowexception-celery-command-failed-the-recorded-hostname-does-not-match-t] *Summary:* This issue is a symptom of another issue I just resolved here [AirflowException: Celery command failed - The recorded hostname does not match this instance's hostname][1]. I didn't see the _AirflowException: Celery command failed_ for a while because it showed up on the _airflow worker_ logs. It wasn't until I watched the airflow worker logs in real time that I saw when the error is thrown I also got the BrokenPipeException in my task. It gets somewhat weirder though. I would only see the BrokenPipeException thrown if I did `print("something to log")` *and* the `AirflowException: Celery command failed...` error happened on the Worker node. When I changed all of my print statements to use `import logging ... logging.info("something to log")` then I would not see the BrokenPipeException *but* the task would still fail because of the `AirflowException: Celery command failed...` error. But had I not seen the BrokenPipeException being thrown in my Airflow task logs I wouldn't have known why the task was failing because once I eliminated the print statements I never saw any error in the Airflow task logs (only on the _$airflow worker_ logs) So long story short there are a few take aways. # Don't do `print("something to log")` use Airflow's built in logging by importing logging and then using the logging class like `import logging` then `logging.info("something to log")` # If you're using an AWS EC2-Instance as your server for Airflow then you may be experiencing this issue: https://github.com/apache/incubator-airflow/pull/2484 a fix to this issue has already been integrated into Airflow Version 1.10 (I'm currently using Airflow Version 1.9). So upgrade your [Airflow version to 1.10][2]. You can also use [the command here][3] but running `pip install git+git://github.com/apache/incubator-airflow.git` gives me version `v2.0.0.dev0+incubating` which I'm not sure if that's a development version or what, but it appears to be higher than 1.10 so perhaps it'll work. Also, if you don't want to upgrade your Airflow version then you could follow the steps on [the github issue][4] to either manually update the file with the fix or fork Airflow and cherry pick the commit that fixes it. *Sources:* [1]: https://stackoverflow.com/questions/51775370/airflowexception-celery-command-failed-the-recorded-hostname-does-not-match-t [2]: https://github.com/apache/incubator-airflow/releases [3]: https://stackoverflow.com/a/47540377/3299397 [4]: https://github.com/apache/incubator-airflow/pull/2484 > Airflow Logs BrokenPipeException > > > Key: AIRFLOW-2844 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2844 > Project: Apache Airflow > Issue Type: Bug > Components: logging >Affects Versions: 1.9.0 >Reporter: Kyle Bridenstine >Priority: Critical > > I'm using a clustered Airflow environment where I have four AWS ec2-instances > for the servers. > *ec2-instances* > - Server 1: Webserver, Scheduler, Redis Queue, PostgreSQL Database > - Server 2: Webserver > - Server 3: Worker > - Server 4: Worker > My setup has been working perfectly fine for three months now but > sporadically about once a week I get a Broken Pipe Exception when Airflow is > attempting to log something. > {code:java} > Log file isn't local. > Fetching here: > http://ip-1-2-3-4:8793/log/foobar/task_1/2018-07-13T00:00:00/1.log > [2018-07-16 00:00:15,521] {cli.py:374} INFO - Running on host ip-1-2-3-4 > [2018-07-16 00:00:15,698] {models.py:1197} INFO - Dependencies all met for > > [2018-07-16 00:00:15,710] {models.py:1197} INFO - Dependencies all met for > > [2018-07-16 00:00:15,710] {models.py:1407} INFO - > > Starting attempt 1 of 1 > > [2018-07-16 00:00:15,719] {models.py:1428} INFO - Executing > on 2018-07-13 00:00:00 > [2018-07-16 00:00:15,720] {base_task_runner.py:115} INFO - Running: ['bash', > '-c', 'airflow run foobar task_1 2018-07-13T00:00:00 --job_id 1320 --raw -sd > DAGS_FOLDER/datalake_digitalplatform_arl_workflow_schedule_test_2.py'] > [2018-07-16 00:00:16,532] {base_task_runner.py:98} INFO - Subtask:
[jira] [Commented] (AIRFLOW-2866) Missing CSRF Token Error on Web UI Create/Update Operations
[ https://issues.apache.org/jira/browse/AIRFLOW-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576627#comment-16576627 ] Tao Feng commented on AIRFLOW-2866: --- which version did you run? Could it be related to AIRFLOW-2809? > Missing CSRF Token Error on Web UI Create/Update Operations > --- > > Key: AIRFLOW-2866 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2866 > Project: Apache Airflow > Issue Type: Bug > Components: webapp >Reporter: Jasper Kahn >Priority: Major > > Attempting to modify or delete many resources (such as Connections or Users) > results in a 400 from the webserver: > {quote}{{Bad Request}} > {{The CSRF session token is missing.}}{quote} > Logs report: > {quote}{{[2018-08-07 18:45:15,771] \{csrf.py:251} INFO - The CSRF session > token is missing.}} > {{192.168.9.1 - - [07/Aug/2018:18:45:15 +] "POST > /admin/connection/delete/ HTTP/1.1" 400 150 > "http://localhost:8081/admin/connection/; "Mozilla/5.0 (X11; Linux x86_64) > AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.84 > Safari/537.36"}}{quote} > Chrome dev tools show the CSRF token is present in the request payload. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] feng-tao edited a comment on issue #3732: Fix typos detected by github.com/client9/misspell
feng-tao edited a comment on issue #3732: Fix typos detected by github.com/client9/misspell URL: https://github.com/apache/incubator-airflow/pull/3732#issuecomment-412148787 hey @seratch , the tool looks very useful :). Your change lgtm. But could you help to create a jira ticket and append that jira on your commit title(e.g https://github.com/apache/incubator-airflow/pull/3648)? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-2866) Missing CSRF Token Error on Web UI Create/Update Operations
[ https://issues.apache.org/jira/browse/AIRFLOW-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576602#comment-16576602 ] Craig Forster commented on AIRFLOW-2866: Looks related to AIRFLOW-2866? > Missing CSRF Token Error on Web UI Create/Update Operations > --- > > Key: AIRFLOW-2866 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2866 > Project: Apache Airflow > Issue Type: Bug > Components: webapp >Reporter: Jasper Kahn >Priority: Major > > Attempting to modify or delete many resources (such as Connections or Users) > results in a 400 from the webserver: > {quote}{{Bad Request}} > {{The CSRF session token is missing.}}{quote} > Logs report: > {quote}{{[2018-08-07 18:45:15,771] \{csrf.py:251} INFO - The CSRF session > token is missing.}} > {{192.168.9.1 - - [07/Aug/2018:18:45:15 +] "POST > /admin/connection/delete/ HTTP/1.1" 400 150 > "http://localhost:8081/admin/connection/; "Mozilla/5.0 (X11; Linux x86_64) > AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.84 > Safari/537.36"}}{quote} > Chrome dev tools show the CSRF token is present in the request payload. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] xnuinside commented on issue #3733: [WIP] [AIRFLOW-491] Add cache parameter in BigQuery query method
xnuinside commented on issue #3733: [WIP] [AIRFLOW-491] Add cache parameter in BigQuery query method URL: https://github.com/apache/incubator-airflow/pull/3733#issuecomment-412147816 @kaxil , how do you think need I to remove other params relative to "query" like use_legacy? and just described what they can be set up using src_fmt_configs? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] XD-DENG commented on issue #3737: Revert "[AIRFLOW-2878] Fix www_rbac display issue"
XD-DENG commented on issue #3737: Revert "[AIRFLOW-2878] Fix www_rbac display issue" URL: https://github.com/apache/incubator-airflow/pull/3737#issuecomment-412147753 Thanks @feng-tao This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-2866) Missing CSRF Token Error on Web UI Create/Update Operations
[ https://issues.apache.org/jira/browse/AIRFLOW-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576600#comment-16576600 ] Craig Forster commented on AIRFLOW-2866: I'm seeing this on master as well. > Missing CSRF Token Error on Web UI Create/Update Operations > --- > > Key: AIRFLOW-2866 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2866 > Project: Apache Airflow > Issue Type: Bug > Components: webapp >Reporter: Jasper Kahn >Priority: Major > > Attempting to modify or delete many resources (such as Connections or Users) > results in a 400 from the webserver: > {quote}{{Bad Request}} > {{The CSRF session token is missing.}}{quote} > Logs report: > {quote}{{[2018-08-07 18:45:15,771] \{csrf.py:251} INFO - The CSRF session > token is missing.}} > {{192.168.9.1 - - [07/Aug/2018:18:45:15 +] "POST > /admin/connection/delete/ HTTP/1.1" 400 150 > "http://localhost:8081/admin/connection/; "Mozilla/5.0 (X11; Linux x86_64) > AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.84 > Safari/537.36"}}{quote} > Chrome dev tools show the CSRF token is present in the request payload. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2878) "/www_rbac" Layout/UI Display Issue
[ https://issues.apache.org/jira/browse/AIRFLOW-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576596#comment-16576596 ] ASF subversion and git services commented on AIRFLOW-2878: -- Commit f999ce22c16b11e8b9205d1bbb6af0683a0c7a3d in incubator-airflow's branch refs/heads/master from [~TaoFeng] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=f999ce2 ] Revert "[AIRFLOW-2878] Fix www_rbac display issue" (#3737) This reverts commit 1f57dafd70ac41da8295f20793f8913b7f5a5dff. > "/www_rbac" Layout/UI Display Issue > --- > > Key: AIRFLOW-2878 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2878 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > > The new RBAC UI has some issues about layout/UI display. > The header () is not shown ("hidden" by the Nav Bar), or tables are not > shown completely. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2878) "/www_rbac" Layout/UI Display Issue
[ https://issues.apache.org/jira/browse/AIRFLOW-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576595#comment-16576595 ] ASF GitHub Bot commented on AIRFLOW-2878: - feng-tao closed pull request #3737: Revert "[AIRFLOW-2878] Fix www_rbac display issue" URL: https://github.com/apache/incubator-airflow/pull/3737 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/airflow/www_rbac/templates/appbuilder/baselayout.html b/airflow/www_rbac/templates/appbuilder/baselayout.html index b0728ca14b..1653a909d0 100644 --- a/airflow/www_rbac/templates/appbuilder/baselayout.html +++ b/airflow/www_rbac/templates/appbuilder/baselayout.html @@ -43,9 +43,6 @@ - - - {% block messages %} {% include 'appbuilder/flash.html' %} This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > "/www_rbac" Layout/UI Display Issue > --- > > Key: AIRFLOW-2878 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2878 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > > The new RBAC UI has some issues about layout/UI display. > The header () is not shown ("hidden" by the Nav Bar), or tables are not > shown completely. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] feng-tao closed pull request #3737: Revert "[AIRFLOW-2878] Fix www_rbac display issue"
feng-tao closed pull request #3737: Revert "[AIRFLOW-2878] Fix www_rbac display issue" URL: https://github.com/apache/incubator-airflow/pull/3737 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/airflow/www_rbac/templates/appbuilder/baselayout.html b/airflow/www_rbac/templates/appbuilder/baselayout.html index b0728ca14b..1653a909d0 100644 --- a/airflow/www_rbac/templates/appbuilder/baselayout.html +++ b/airflow/www_rbac/templates/appbuilder/baselayout.html @@ -43,9 +43,6 @@ - - - {% block messages %} {% include 'appbuilder/flash.html' %} This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] XD-DENG commented on issue #3738: [AIRFLOW-2886] Secure Flask SECRET_KEY
XD-DENG commented on issue #3738: [AIRFLOW-2886] Secure Flask SECRET_KEY URL: https://github.com/apache/incubator-airflow/pull/3738#issuecomment-412146841 @feng-tao @Fokko PTAL. We need to have random SECRET_KEY for Flask App, while it must be consistent among workers. So I propose to deal with it like how we dealt with `FERNET_KEY`. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-2886) Secure Flask SECRET_KEY
[ https://issues.apache.org/jira/browse/AIRFLOW-2886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576591#comment-16576591 ] ASF GitHub Bot commented on AIRFLOW-2886: - XD-DENG opened a new pull request #3738: [AIRFLOW-2886] Secure Flask SECRET_KEY URL: https://github.com/apache/incubator-airflow/pull/3738 ### Jira - [x] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-2886 - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. ### Description - [x] Here are some details about my PR, including screenshots of any UI changes: In my earlier PRs, https://github.com/apache/incubator-airflow/pull/3651 and https://github.com/apache/incubator-airflow/pull/3729 , I proposed to generate random `SECRET_KEY` for the `webserver` (Flask App). However, I realise that we may encounter CSRF error `The CSRF session token is missing` when we have multiple workers for the Flask webserver, since the secret_key is not consistent among workers. On the other hand, it's still very important to have as random SECRET_KEY as possible for security reasons. We can deal with it like how we dealt with `FERNET_KEY` (i.e. generate a random value when the airflow.cfg file is initiated). ### Tests - [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: ### Commits - [x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [ ] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. ### Code Quality - [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Secure Flask SECRET_KEY > --- > > Key: AIRFLOW-2886 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2886 > Project: Apache Airflow > Issue Type: Bug >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > > In my earlier PRs, [https://github.com/apache/incubator-airflow/pull/3651] > and [https://github.com/apache/incubator-airflow/pull/3729] , I proposed to > generate random SECRET_KEY for Flask App. > If we have multiple workers for the Flask webserver, we may encounter CSRF > error {{The CSRF session token is missing}} . > On the other hand, it's still very important to have as random SECRET_KEY as > possible for security reasons. We can deal with it like how we dealt with > FERNET_KEY (i.e. generate a random value when the airflow.cfg file is > initiated). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] XD-DENG opened a new pull request #3738: [AIRFLOW-2886] Secure Flask SECRET_KEY
XD-DENG opened a new pull request #3738: [AIRFLOW-2886] Secure Flask SECRET_KEY URL: https://github.com/apache/incubator-airflow/pull/3738 ### Jira - [x] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-2886 - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. ### Description - [x] Here are some details about my PR, including screenshots of any UI changes: In my earlier PRs, https://github.com/apache/incubator-airflow/pull/3651 and https://github.com/apache/incubator-airflow/pull/3729 , I proposed to generate random `SECRET_KEY` for the `webserver` (Flask App). However, I realise that we may encounter CSRF error `The CSRF session token is missing` when we have multiple workers for the Flask webserver, since the secret_key is not consistent among workers. On the other hand, it's still very important to have as random SECRET_KEY as possible for security reasons. We can deal with it like how we dealt with `FERNET_KEY` (i.e. generate a random value when the airflow.cfg file is initiated). ### Tests - [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: ### Commits - [x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [ ] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. ### Code Quality - [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-2886) Secure Flask SECRET_KEY
Xiaodong DENG created AIRFLOW-2886: -- Summary: Secure Flask SECRET_KEY Key: AIRFLOW-2886 URL: https://issues.apache.org/jira/browse/AIRFLOW-2886 Project: Apache Airflow Issue Type: Bug Reporter: Xiaodong DENG Assignee: Xiaodong DENG In my earlier PRs, [https://github.com/apache/incubator-airflow/pull/3651] and [https://github.com/apache/incubator-airflow/pull/3729] , I proposed to generate random SECRET_KEY for Flask App. If we have multiple workers for the Flask webserver, we may encounter CSRF error {{The CSRF session token is missing}} . On the other hand, it's still very important to have as random SECRET_KEY as possible for security reasons. We can deal with it like how we dealt with FERNET_KEY (i.e. generate a random value when the airflow.cfg file is initiated). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] feng-tao commented on issue #3734: Revert "[AIRFLOW-2878] Fix www_rbac display issue"
feng-tao commented on issue #3734: Revert "[AIRFLOW-2878] Fix www_rbac display issue" URL: https://github.com/apache/incubator-airflow/pull/3734#issuecomment-412138578 @XD-DENG , I think directly revertting your pr in github may revert other prs that get merged after yours. But never mind, I will take care of that. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feng-tao commented on issue #3734: Revert "[AIRFLOW-2878] Fix www_rbac display issue"
feng-tao commented on issue #3734: Revert "[AIRFLOW-2878] Fix www_rbac display issue" URL: https://github.com/apache/incubator-airflow/pull/3734#issuecomment-412139622 @XD-DENG , will do. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] XD-DENG commented on issue #3734: Revert "[AIRFLOW-2878] Fix www_rbac display issue"
XD-DENG commented on issue #3734: Revert "[AIRFLOW-2878] Fix www_rbac display issue" URL: https://github.com/apache/incubator-airflow/pull/3734#issuecomment-412139420 @feng-tao Or may I trouble you to do a separate commit to remove the three lines I added? i.e. it's actually reverting the changes made in my earlier PR but doesn't touch other PRs? Thanks for taking care of this, and sorry for the inconvenience caused. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feng-tao commented on issue #3737: Revert "[AIRFLOW-2878] Fix www_rbac display issue"
feng-tao commented on issue #3737: Revert "[AIRFLOW-2878] Fix www_rbac display issue" URL: https://github.com/apache/incubator-airflow/pull/3737#issuecomment-412138003 This is based on the discussion in https://github.com/apache/incubator-airflow/pull/3724. cc @r39132 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-2878) "/www_rbac" Layout/UI Display Issue
[ https://issues.apache.org/jira/browse/AIRFLOW-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576546#comment-16576546 ] ASF GitHub Bot commented on AIRFLOW-2878: - feng-tao opened a new pull request #3737: Revert "[AIRFLOW-2878] Fix www_rbac display issue" URL: https://github.com/apache/incubator-airflow/pull/3737 This reverts commit 1f57dafd70ac41da8295f20793f8913b7f5a5dff. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > "/www_rbac" Layout/UI Display Issue > --- > > Key: AIRFLOW-2878 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2878 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > > The new RBAC UI has some issues about layout/UI display. > The header () is not shown ("hidden" by the Nav Bar), or tables are not > shown completely. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] feng-tao opened a new pull request #3737: Revert "[AIRFLOW-2878] Fix www_rbac display issue"
feng-tao opened a new pull request #3737: Revert "[AIRFLOW-2878] Fix www_rbac display issue" URL: https://github.com/apache/incubator-airflow/pull/3737 This reverts commit 1f57dafd70ac41da8295f20793f8913b7f5a5dff. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feng-tao closed pull request #3734: Revert "[AIRFLOW-2878] Fix www_rbac display issue"
feng-tao closed pull request #3734: Revert "[AIRFLOW-2878] Fix www_rbac display issue" URL: https://github.com/apache/incubator-airflow/pull/3734 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/airflow/www_rbac/templates/appbuilder/baselayout.html b/airflow/www_rbac/templates/appbuilder/baselayout.html index b0728ca14b..1653a909d0 100644 --- a/airflow/www_rbac/templates/appbuilder/baselayout.html +++ b/airflow/www_rbac/templates/appbuilder/baselayout.html @@ -43,9 +43,6 @@ - - - {% block messages %} {% include 'appbuilder/flash.html' %} This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-2878) "/www_rbac" Layout/UI Display Issue
[ https://issues.apache.org/jira/browse/AIRFLOW-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576536#comment-16576536 ] ASF GitHub Bot commented on AIRFLOW-2878: - XD-DENG opened a new pull request #3734: Revert "[AIRFLOW-2878] Fix www_rbac display issue" URL: https://github.com/apache/incubator-airflow/pull/3734 Hi @r39132 , this is to revert my earlier PR https://github.com/apache/incubator-airflow/pull/3724. As pointed out by @verdan , I encountered the issues that I tried to fix in my earlier PR because I didn't build the frontend packages using `npm` and `webpack`. That commit may also introduce extra spaces in each page. Please refer to my conversation with Verdan in PR https://github.com/apache/incubator-airflow/pull/3724. Sorry for the inconvenience cause, and thanks @verdan for having pointed this out. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > "/www_rbac" Layout/UI Display Issue > --- > > Key: AIRFLOW-2878 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2878 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > > The new RBAC UI has some issues about layout/UI display. > The header () is not shown ("hidden" by the Nav Bar), or tables are not > shown completely. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] feng-tao commented on issue #3728: [AIRFLOW-2883] Not search dag owner if owners are missing
feng-tao commented on issue #3728: [AIRFLOW-2883] Not search dag owner if owners are missing URL: https://github.com/apache/incubator-airflow/pull/3728#issuecomment-412137090 PTAL @r39132 , any other concern? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-2878) "/www_rbac" Layout/UI Display Issue
[ https://issues.apache.org/jira/browse/AIRFLOW-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576532#comment-16576532 ] ASF GitHub Bot commented on AIRFLOW-2878: - feng-tao closed pull request #3736: Revert "[AIRFLOW-2878] Fix www_rbac display issue" URL: https://github.com/apache/incubator-airflow/pull/3736 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > "/www_rbac" Layout/UI Display Issue > --- > > Key: AIRFLOW-2878 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2878 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > > The new RBAC UI has some issues about layout/UI display. > The header () is not shown ("hidden" by the Nav Bar), or tables are not > shown completely. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] feng-tao commented on issue #3648: [AIRFLOW-2786] Fix editing Variable with empty key crashing
feng-tao commented on issue #3648: [AIRFLOW-2786] Fix editing Variable with empty key crashing URL: https://github.com/apache/incubator-airflow/pull/3648#issuecomment-412134502 thanks @Noremac201 . Merge. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-2786) Variables view fails to render if a variable has an empty key
[ https://issues.apache.org/jira/browse/AIRFLOW-2786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576521#comment-16576521 ] ASF GitHub Bot commented on AIRFLOW-2786: - feng-tao closed pull request #3648: [AIRFLOW-2786] Fix editing Variable with empty key crashing URL: https://github.com/apache/incubator-airflow/pull/3648 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/airflow/www/utils.py b/airflow/www/utils.py index 1bbc0936b3..0c4f4b05d6 100644 --- a/airflow/www/utils.py +++ b/airflow/www/utils.py @@ -56,8 +56,13 @@ def should_hide_value_for_key(key_name): -return any(s in key_name.lower() for s in DEFAULT_SENSITIVE_VARIABLE_FIELDS) \ -and configuration.conf.getboolean('admin', 'hide_sensitive_variable_fields') +# It is possible via importing variables from file that a key is empty. +if key_name: +config_set = configuration.conf.getboolean('admin', + 'hide_sensitive_variable_fields') +field_comp = any(s in key_name.lower() for s in DEFAULT_SENSITIVE_VARIABLE_FIELDS) +return config_set and field_comp +return False class LoginMixin(object): diff --git a/airflow/www/views.py b/airflow/www/views.py index 3e41d2d02a..0c0dcff801 100644 --- a/airflow/www/views.py +++ b/airflow/www/views.py @@ -2012,9 +2012,20 @@ def varimport(self): except Exception as e: flash("Missing file or syntax error: {}.".format(e)) else: +suc_count = fail_count = 0 for k, v in d.items(): -models.Variable.set(k, v, serialize_json=isinstance(v, dict)) -flash("{} variable(s) successfully updated.".format(len(d))) +try: +models.Variable.set(k, v, serialize_json=isinstance(v, dict)) +except Exception as e: +logging.info('Variable import failed: {}'.format(repr(e))) +fail_count += 1 +else: +suc_count += 1 +flash("{} variable(s) successfully updated.".format(suc_count), 'info') +if fail_count: +flash( +"{} variables(s) failed to be updated.".format(fail_count), 'error') + return redirect('/admin/variable') diff --git a/airflow/www_rbac/utils.py b/airflow/www_rbac/utils.py index 7bbdada555..a0e9258eae 100644 --- a/airflow/www_rbac/utils.py +++ b/airflow/www_rbac/utils.py @@ -54,8 +54,13 @@ def should_hide_value_for_key(key_name): -return any(s in key_name.lower() for s in DEFAULT_SENSITIVE_VARIABLE_FIELDS) \ -and configuration.getboolean('admin', 'hide_sensitive_variable_fields') +# It is possible via importing variables from file that a key is empty. +if key_name: +config_set = configuration.conf.getboolean('admin', + 'hide_sensitive_variable_fields') +field_comp = any(s in key_name.lower() for s in DEFAULT_SENSITIVE_VARIABLE_FIELDS) +return config_set and field_comp +return False def get_params(**kwargs): diff --git a/airflow/www_rbac/views.py b/airflow/www_rbac/views.py index a9947ae096..629f488fc7 100644 --- a/airflow/www_rbac/views.py +++ b/airflow/www_rbac/views.py @@ -2053,9 +2053,18 @@ def varimport(self): except Exception: flash("Missing file or syntax error.") else: +suc_count = fail_count = 0 for k, v in d.items(): -models.Variable.set(k, v, serialize_json=isinstance(v, dict)) -flash("{} variable(s) successfully updated.".format(len(d))) +try: +models.Variable.set(k, v, serialize_json=isinstance(v, dict)) +except Exception as e: +logging.info('Variable import failed: {}'.format(repr(e))) +fail_count += 1 +else: +suc_count += 1 +flash("{} variable(s) successfully updated.".format(suc_count), 'info') +if fail_count: +flash("{} variables(s) failed to be updated.".format(fail_count), 'error') self.update_redirect() return redirect(self.get_redirect()) diff --git a/tests/www/test_utils.py b/tests/www/test_utils.py index 891298c0a9..9034b8b5fd 100644 --- a/tests/www/test_utils.py +++ b/tests/www/test_utils.py @@ -32,6 +32,10 @@ class UtilsTest(unittest.TestCase): def setUp(self): super(UtilsTest, self).setUp() +def test_empty_variable_should_not_be_hidden(self): +self.assertFalse(utils.should_hide_value_for_key(""))
[jira] [Commented] (AIRFLOW-2878) "/www_rbac" Layout/UI Display Issue
[ https://issues.apache.org/jira/browse/AIRFLOW-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576519#comment-16576519 ] ASF GitHub Bot commented on AIRFLOW-2878: - XD-DENG closed pull request #3734: Revert "[AIRFLOW-2878] Fix www_rbac display issue" URL: https://github.com/apache/incubator-airflow/pull/3734 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/airflow/www_rbac/templates/appbuilder/baselayout.html b/airflow/www_rbac/templates/appbuilder/baselayout.html index b0728ca14b..1653a909d0 100644 --- a/airflow/www_rbac/templates/appbuilder/baselayout.html +++ b/airflow/www_rbac/templates/appbuilder/baselayout.html @@ -43,9 +43,6 @@ - - - {% block messages %} {% include 'appbuilder/flash.html' %} This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > "/www_rbac" Layout/UI Display Issue > --- > > Key: AIRFLOW-2878 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2878 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > > The new RBAC UI has some issues about layout/UI display. > The header () is not shown ("hidden" by the Nav Bar), or tables are not > shown completely. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] feng-tao commented on issue #3734: Revert "[AIRFLOW-2878] Fix www_rbac display issue"
feng-tao commented on issue #3734: Revert "[AIRFLOW-2878] Fix www_rbac display issue" URL: https://github.com/apache/incubator-airflow/pull/3734#issuecomment-412134197 I don't think we need another pull request. We directly revert the original request. I will do that. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-2878) "/www_rbac" Layout/UI Display Issue
[ https://issues.apache.org/jira/browse/AIRFLOW-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576518#comment-16576518 ] ASF GitHub Bot commented on AIRFLOW-2878: - feng-tao opened a new pull request #3736: Revert "[AIRFLOW-2878] Fix www_rbac display issue" URL: https://github.com/apache/incubator-airflow/pull/3736 Reverts apache/incubator-airflow#3724 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > "/www_rbac" Layout/UI Display Issue > --- > > Key: AIRFLOW-2878 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2878 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > > The new RBAC UI has some issues about layout/UI display. > The header () is not shown ("hidden" by the Nav Bar), or tables are not > shown completely. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] feng-tao opened a new pull request #3736: Revert "[AIRFLOW-2878] Fix www_rbac display issue"
feng-tao opened a new pull request #3736: Revert "[AIRFLOW-2878] Fix www_rbac display issue" URL: https://github.com/apache/incubator-airflow/pull/3736 Reverts apache/incubator-airflow#3724 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-2878) "/www_rbac" Layout/UI Display Issue
[ https://issues.apache.org/jira/browse/AIRFLOW-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576514#comment-16576514 ] ASF GitHub Bot commented on AIRFLOW-2878: - feng-tao closed pull request #3735: Revert "[AIRFLOW-2878] Fix www_rbac display issue" URL: https://github.com/apache/incubator-airflow/pull/3735 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > "/www_rbac" Layout/UI Display Issue > --- > > Key: AIRFLOW-2878 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2878 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > > The new RBAC UI has some issues about layout/UI display. > The header () is not shown ("hidden" by the Nav Bar), or tables are not > shown completely. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2878) "/www_rbac" Layout/UI Display Issue
[ https://issues.apache.org/jira/browse/AIRFLOW-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576512#comment-16576512 ] ASF GitHub Bot commented on AIRFLOW-2878: - feng-tao opened a new pull request #3735: Revert "[AIRFLOW-2878] Fix www_rbac display issue" URL: https://github.com/apache/incubator-airflow/pull/3735 Reverts apache/incubator-airflow#3724 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > "/www_rbac" Layout/UI Display Issue > --- > > Key: AIRFLOW-2878 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2878 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > > The new RBAC UI has some issues about layout/UI display. > The header () is not shown ("hidden" by the Nav Bar), or tables are not > shown completely. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] feng-tao closed pull request #3735: Revert "[AIRFLOW-2878] Fix www_rbac display issue"
feng-tao closed pull request #3735: Revert "[AIRFLOW-2878] Fix www_rbac display issue" URL: https://github.com/apache/incubator-airflow/pull/3735 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feng-tao opened a new pull request #3735: Revert "[AIRFLOW-2878] Fix www_rbac display issue"
feng-tao opened a new pull request #3735: Revert "[AIRFLOW-2878] Fix www_rbac display issue" URL: https://github.com/apache/incubator-airflow/pull/3735 Reverts apache/incubator-airflow#3724 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] XD-DENG commented on issue #3729: [AIRFLOW-2884] Fix Flask SECRET_KEY security issue in www_rbac
XD-DENG commented on issue #3729: [AIRFLOW-2884] Fix Flask SECRET_KEY security issue in www_rbac URL: https://github.com/apache/incubator-airflow/pull/3729#issuecomment-412133255 Thanks @kaxil This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil edited a comment on issue #3729: [AIRFLOW-2884] Fix Flask SECRET_KEY security issue in www_rbac
kaxil edited a comment on issue #3729: [AIRFLOW-2884] Fix Flask SECRET_KEY security issue in www_rbac URL: https://github.com/apache/incubator-airflow/pull/3729#issuecomment-412132706 @XD-DENG I would be on holidays and hence unreachable, please ping @feng-tao or @Fokko when it is ready. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil commented on issue #3729: [AIRFLOW-2884] Fix Flask SECRET_KEY security issue in www_rbac
kaxil commented on issue #3729: [AIRFLOW-2884] Fix Flask SECRET_KEY security issue in www_rbac URL: https://github.com/apache/incubator-airflow/pull/3729#issuecomment-412132706 @XD-DENG I would be on holdiays and hence unreachable, please ping @feng-tao or @Fokko when it is ready. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil commented on issue #3733: [WIP] [AIRFLOW-491] Add cache parameter in BigQuery query method
kaxil commented on issue #3733: [WIP] [AIRFLOW-491] Add cache parameter in BigQuery query method URL: https://github.com/apache/incubator-airflow/pull/3733#issuecomment-412132227 Hi @xnuinside Please add `src_fmt_configs` param to `run_query` method and the Operator. https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/bigquery_operator.py#L463 This would allow to add such parameters in future without having to add a new parameter each time. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] XD-DENG commented on issue #3729: [AIRFLOW-2884] Fix Flask SECRET_KEY security issue in www_rbac
XD-DENG commented on issue #3729: [AIRFLOW-2884] Fix Flask SECRET_KEY security issue in www_rbac URL: https://github.com/apache/incubator-airflow/pull/3729#issuecomment-412129544 Hi @kaxil , I have realised this method will cause CSRF error `The CSRF session token is missing` when we have multiple workers for `webserver` (we generate random secret_key for each worker, and then they're not consistent among workers). But I think it's still very necessary to have `as random secret_key as possible`. One feasible way is to generate it like how we generate `fernet_key`. I will raise a separate PR to address this and ping you then. Sorry for the inconvenience caused. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Noremac201 commented on issue #3648: [AIRFLOW-2786] Fix editing Variable with empty key crashing
Noremac201 commented on issue #3648: [AIRFLOW-2786] Fix editing Variable with empty key crashing URL: https://github.com/apache/incubator-airflow/pull/3648#issuecomment-412125247 Looks like it was a python 3.5 incompatibility -- fixed now Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-1059) Reset_state_for_orphaned_task should operate in batch for the scheduler
[ https://issues.apache.org/jira/browse/AIRFLOW-1059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576372#comment-16576372 ] Feng Lu commented on AIRFLOW-1059: -- [~saguziel] Could you elaborate a bit more why orphaned tasks from backfilled dagruns can't be re-enqueued? > Reset_state_for_orphaned_task should operate in batch for the scheduler > --- > > Key: AIRFLOW-1059 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1059 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Alex Guziel >Assignee: Alex Guziel >Priority: Major > Fix For: 1.9.0 > > > Scheduler startup is very slow due to resetting state making a query for each > dag run. We should be able to do this in a constant number of queries which > will increase scheduler startup time significantly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2878) "/www_rbac" Layout/UI Display Issue
[ https://issues.apache.org/jira/browse/AIRFLOW-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576369#comment-16576369 ] ASF GitHub Bot commented on AIRFLOW-2878: - XD-DENG opened a new pull request #3734: Revert "[AIRFLOW-2878] Fix www_rbac display issue" URL: https://github.com/apache/incubator-airflow/pull/3734 Hi @r39132 , this is to revert my earlier PR https://github.com/apache/incubator-airflow/pull/3724. As pointed out by @verdan , I encountered the issues that I tried to fix in my earlier PR because I didn't build the frontend packages using `npm` and `webpack`. That commit may also introduce extra spaces in each page. Please refer to my conversation with Verdan in PR https://github.com/apache/incubator-airflow/pull/3724. Sorry for the inconvenience cause, and thanks @verdan for having pointed this out. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > "/www_rbac" Layout/UI Display Issue > --- > > Key: AIRFLOW-2878 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2878 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > > The new RBAC UI has some issues about layout/UI display. > The header () is not shown ("hidden" by the Nav Bar), or tables are not > shown completely. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] jrderuiter commented on issue #3560: [AIRFLOW-2697] Drop snakebite in favour of hdfs3
jrderuiter commented on issue #3560: [AIRFLOW-2697] Drop snakebite in favour of hdfs3 URL: https://github.com/apache/incubator-airflow/pull/3560#issuecomment-412103674 @Fokko As we discussed, I removed the deprecation code so we can merge this in preparation for Airflow 2.0. I want to do some final testing, afterwards we can merge if everything passes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] XD-DENG opened a new pull request #3734: Revert "[AIRFLOW-2878] Fix www_rbac display issue"
XD-DENG opened a new pull request #3734: Revert "[AIRFLOW-2878] Fix www_rbac display issue" URL: https://github.com/apache/incubator-airflow/pull/3734 Hi @r39132 , this is to revert my earlier PR https://github.com/apache/incubator-airflow/pull/3724. As pointed out by @verdan , I encountered the issues that I tried to fix in my earlier PR because I didn't build the frontend packages using `npm` and `webpack`. That commit may also introduce extra spaces in each page. Please refer to my conversation with Verdan in PR https://github.com/apache/incubator-airflow/pull/3724. Sorry for the inconvenience cause, and thanks @verdan for having pointed this out. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] xnuinside commented on issue #3733: [AIRFLOW-491] Add cache parameter in BigQuery query method
xnuinside commented on issue #3733: [AIRFLOW-491] Add cache parameter in BigQuery query method URL: https://github.com/apache/incubator-airflow/pull/3733#issuecomment-412071257 maybe make sense add use_query_cache=False for check operators This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] xnuinside opened a new pull request #3733: [AIRFLOW-491] Add cache parameter in BigQuery query method
xnuinside opened a new pull request #3733: [AIRFLOW-491] Add cache parameter in BigQuery query method URL: https://github.com/apache/incubator-airflow/pull/3733 Make sure you have checked _all_ steps below. ### Jira - [ ] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-491 - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. ### Description - [ ] Here are some details about my PR, including screenshots of any UI changes: Added "useQueryCache" from job BQ configuration https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query ### Tests - [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: ### Commits - [ ] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [ ] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. ### Code Quality - [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-491) Add cache parameter in BigQuery query method
[ https://issues.apache.org/jira/browse/AIRFLOW-491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576223#comment-16576223 ] ASF GitHub Bot commented on AIRFLOW-491: xnuinside opened a new pull request #3733: [AIRFLOW-491] Add cache parameter in BigQuery query method URL: https://github.com/apache/incubator-airflow/pull/3733 Make sure you have checked _all_ steps below. ### Jira - [ ] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-491 - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. ### Description - [ ] Here are some details about my PR, including screenshots of any UI changes: Added "useQueryCache" from job BQ configuration https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query ### Tests - [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: ### Commits - [ ] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [ ] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. ### Code Quality - [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add cache parameter in BigQuery query method > > > Key: AIRFLOW-491 > URL: https://issues.apache.org/jira/browse/AIRFLOW-491 > Project: Apache Airflow > Issue Type: Bug > Components: contrib, gcp >Affects Versions: Airflow 1.7.1 >Reporter: Chris Riccomini >Assignee: Iuliia Volkova >Priority: Major > Fix For: Airflow 1.8 > > > The current BigQuery query() method does not have a user_query_cache > parameter. This param always defaults to true (see > [here|https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.query]). > I'd like to disable query caching for some data consistency checks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] XD-DENG commented on issue #3724: [AIRFLOW-2878] Fix www_rbac display issue
XD-DENG commented on issue #3724: [AIRFLOW-2878] Fix www_rbac display issue URL: https://github.com/apache/incubator-airflow/pull/3724#issuecomment-412062778 Sure. @r39132 may you please revert this commit as @verdan suggested? Thanks for pointing this out! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] verdan edited a comment on issue #3724: [AIRFLOW-2878] Fix www_rbac display issue
verdan edited a comment on issue #3724: [AIRFLOW-2878] Fix www_rbac display issue URL: https://github.com/apache/incubator-airflow/pull/3724#issuecomment-412062054 perfect, thanks! Also, I believe we need to revert this change, as this change would include extra spaces on each page. cc: @r39132 https://user-images.githubusercontent.com/25360476/43956876-100aba8a-9ca6-11e8-9582-e6104e8d6528.png;> This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] verdan commented on issue #3724: [AIRFLOW-2878] Fix www_rbac display issue
verdan commented on issue #3724: [AIRFLOW-2878] Fix www_rbac display issue URL: https://github.com/apache/incubator-airflow/pull/3724#issuecomment-412062054 perfect, thanks! Also, I believe we need to revert this change, as this change would include extra spaces on each page. cc: @r39132 https://user-images.githubusercontent.com/25360476/43956824-e462e952-9ca5-11e8-9d67-dc1c67ff8998.png;> This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] XD-DENG commented on issue #3724: [AIRFLOW-2878] Fix www_rbac display issue
XD-DENG commented on issue #3724: [AIRFLOW-2878] Fix www_rbac display issue URL: https://github.com/apache/incubator-airflow/pull/3724#issuecomment-412061132 Hi @verdan, I installed from the ‘tar.gz’ file directly. Will check the setting-up you introduced later. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] verdan commented on issue #3724: [AIRFLOW-2878] Fix www_rbac display issue
verdan commented on issue #3724: [AIRFLOW-2878] Fix www_rbac display issue URL: https://github.com/apache/incubator-airflow/pull/3724#issuecomment-412059984 @XD-DENG I believe you haven't built the frontend packages yet, causing all these issues. We've implemented the `npm` and `webpack` to manage the JS libraries and dependencies [AIRFLOW-2691](https://issues.apache.org/jira/browse/AIRFLOW-2691) . Can you please follow the steps mentioned here to build the frontend packages and see if that fixes everything for you? https://github.com/apache/incubator-airflow/blob/master/CONTRIBUTING.md#setting-up-the-node--npm-javascript-environment-only-for-www_rbac This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (AIRFLOW-559) Add support for BigQuery kwarg parameters
[ https://issues.apache.org/jira/browse/AIRFLOW-559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik closed AIRFLOW-559. -- Resolution: Not A Problem We don't need this as we plan to deprecate use of *args and **kwargs and we already have `src_fmt_configs` parameter (https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/bigquery_operator.py#L463) in the currently needed Bigquery operators where we might need to set extra parameters. > Add support for BigQuery kwarg parameters > - > > Key: AIRFLOW-559 > URL: https://issues.apache.org/jira/browse/AIRFLOW-559 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib, gcp >Reporter: Sam McVeety >Assignee: Sam McVeety >Priority: Minor > Fix For: Airflow 1.8 > > > Many of the operators in > https://github.com/apache/incubator-airflow/tree/master/airflow/contrib/operators > add parameters over time, and plumbing these through multiple layers of > calls isn't always a high priority. > The operators (and hooks) should support an end-to-end kwargs parameter that > allows for new fields (e.g. useLegacySql, defaultDataset) to be added by > users without needing to change the underlying code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-559) Add support for BigQuery kwarg parameters
[ https://issues.apache.org/jira/browse/AIRFLOW-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576123#comment-16576123 ] Iuliia Volkova commented on AIRFLOW-559: [~kaxilnaik] , can you close this task to avoid confusion because as you mentioned in https://github.com/apache/incubator-airflow/pull/3717 "As far as I know, there is a plan to deprecate args and kwargs keywords in Airflow."? Because of this task about the opposite thing. And some PRs was closed without merge based on this, for example: [https://github.com/apache/incubator-airflow/pull/1784] > Add support for BigQuery kwarg parameters > - > > Key: AIRFLOW-559 > URL: https://issues.apache.org/jira/browse/AIRFLOW-559 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib, gcp >Reporter: Sam McVeety >Assignee: Sam McVeety >Priority: Minor > Fix For: Airflow 1.8 > > > Many of the operators in > https://github.com/apache/incubator-airflow/tree/master/airflow/contrib/operators > add parameters over time, and plumbing these through multiple layers of > calls isn't always a high priority. > The operators (and hooks) should support an end-to-end kwargs parameter that > allows for new fields (e.g. useLegacySql, defaultDataset) to be added by > users without needing to change the underlying code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-559) Add support for BigQuery kwarg parameters
[ https://issues.apache.org/jira/browse/AIRFLOW-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576123#comment-16576123 ] Iuliia Volkova edited comment on AIRFLOW-559 at 8/10/18 11:21 AM: -- [~kaxilnaik] , can you close this task to avoid confusion because as you mentioned in [https://github.com/apache/incubator-airflow/pull/3717] "As far as I know, there is a plan to deprecate args and kwargs keywords in Airflow."? Because, this task about the opposite thing. And some PRs was closed without merge based on this, for example: [https://github.com/apache/incubator-airflow/pull/1784] was (Author: xnuinside): [~kaxilnaik] , can you close this task to avoid confusion because as you mentioned in https://github.com/apache/incubator-airflow/pull/3717 "As far as I know, there is a plan to deprecate args and kwargs keywords in Airflow."? Because of this task about the opposite thing. And some PRs was closed without merge based on this, for example: [https://github.com/apache/incubator-airflow/pull/1784] > Add support for BigQuery kwarg parameters > - > > Key: AIRFLOW-559 > URL: https://issues.apache.org/jira/browse/AIRFLOW-559 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib, gcp >Reporter: Sam McVeety >Assignee: Sam McVeety >Priority: Minor > Fix For: Airflow 1.8 > > > Many of the operators in > https://github.com/apache/incubator-airflow/tree/master/airflow/contrib/operators > add parameters over time, and plumbing these through multiple layers of > calls isn't always a high priority. > The operators (and hooks) should support an end-to-end kwargs parameter that > allows for new fields (e.g. useLegacySql, defaultDataset) to be added by > users without needing to change the underlying code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-1874) Support standard SQL in Check, ValueCheck and IntervalCheck BigQuery operators
[ https://issues.apache.org/jira/browse/AIRFLOW-1874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-1874. - Resolution: Fixed Resolved by https://github.com/apache/incubator-airflow/pull/3717 > Support standard SQL in Check, ValueCheck and IntervalCheck BigQuery operators > -- > > Key: AIRFLOW-1874 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1874 > Project: Apache Airflow > Issue Type: Bug > Components: contrib, gcp, operators >Reporter: Guillermo Rodríguez Cano >Assignee: Iuliia Volkova >Priority: Major > Fix For: 2.0.0 > > > BigQueryCheckOperator, BigQueryValueCheckOperator and > BigQueryIntervalCheckOperator do not support disabling use of default legacy > SQL in BigQuery. > This is a major blocker to support correct migration to standard SQL when > queries are complicated. For example, a query that can be queried in legacy > SQL may be blocked from any subsequent view done in standard SQL that this > view uses as the queries are bound to either standard or legacy SQL but not a > mix. > These operators inherit from base ones of the same name (without the BigQuery > prefix) from Airflow which may make the process more complicated as the flag > to use standard SQL should be enabled because the underlying BigQueryHook has > the corresponding parameter, use_legacy_sql, set to True, when running a > query. But it is not possible to pass parameters all the way to it via the > aforementioned operators. > The workaround of including #standardSQL and a new line before the query > doesn't work either as there is mismatch. BigQuery reports the following in > fact: "Query text specifies use_legacy_sql:false, while API options > specify:true" > A workaround for queries on views using standard SQL is to persist the result > of the query in a temporary table, then run the check operation and > thereafter delete the temporary table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1874) Support standard SQL in Check, ValueCheck and IntervalCheck BigQuery operators
[ https://issues.apache.org/jira/browse/AIRFLOW-1874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576092#comment-16576092 ] ASF subversion and git services commented on AIRFLOW-1874: -- Commit 9952b23ba08fdc57c866e0f35f20a7a1a13ede74 in incubator-airflow's branch refs/heads/master from Yuliya Volkova [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=9952b23 ] [AIRFLOW-1874] use_legacy_sql added to BigQueryCheck operators (#3717) > Support standard SQL in Check, ValueCheck and IntervalCheck BigQuery operators > -- > > Key: AIRFLOW-1874 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1874 > Project: Apache Airflow > Issue Type: Bug > Components: contrib, gcp, operators >Reporter: Guillermo Rodríguez Cano >Assignee: Iuliia Volkova >Priority: Major > Fix For: 2.0.0 > > > BigQueryCheckOperator, BigQueryValueCheckOperator and > BigQueryIntervalCheckOperator do not support disabling use of default legacy > SQL in BigQuery. > This is a major blocker to support correct migration to standard SQL when > queries are complicated. For example, a query that can be queried in legacy > SQL may be blocked from any subsequent view done in standard SQL that this > view uses as the queries are bound to either standard or legacy SQL but not a > mix. > These operators inherit from base ones of the same name (without the BigQuery > prefix) from Airflow which may make the process more complicated as the flag > to use standard SQL should be enabled because the underlying BigQueryHook has > the corresponding parameter, use_legacy_sql, set to True, when running a > query. But it is not possible to pass parameters all the way to it via the > aforementioned operators. > The workaround of including #standardSQL and a new line before the query > doesn't work either as there is mismatch. BigQuery reports the following in > fact: "Query text specifies use_legacy_sql:false, while API options > specify:true" > A workaround for queries on views using standard SQL is to persist the result > of the query in a temporary table, then run the check operation and > thereafter delete the temporary table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1874) Support standard SQL in Check, ValueCheck and IntervalCheck BigQuery operators
[ https://issues.apache.org/jira/browse/AIRFLOW-1874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576091#comment-16576091 ] ASF GitHub Bot commented on AIRFLOW-1874: - kaxil closed pull request #3717: [AIRFLOW-1874] use_legacy_sql added to BigQueryCheck operators URL: https://github.com/apache/incubator-airflow/pull/3717 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/airflow/contrib/operators/bigquery_check_operator.py b/airflow/contrib/operators/bigquery_check_operator.py index 59ef5d377d..a9c493f4fd 100644 --- a/airflow/contrib/operators/bigquery_check_operator.py +++ b/airflow/contrib/operators/bigquery_check_operator.py @@ -55,21 +55,25 @@ class BigQueryCheckOperator(CheckOperator): :type sql: string :param bigquery_conn_id: reference to the BigQuery database :type bigquery_conn_id: string +:param use_legacy_sql: Whether to use legacy SQL (true) +or standard SQL (false). +:type use_legacy_sql: boolean """ @apply_defaults -def __init__( -self, -sql, -bigquery_conn_id='bigquery_default', -*args, -**kwargs): +def __init__(self, + sql, + bigquery_conn_id='bigquery_default', + use_legacy_sql=True, + *args, **kwargs): super(BigQueryCheckOperator, self).__init__(sql=sql, *args, **kwargs) self.bigquery_conn_id = bigquery_conn_id self.sql = sql +self.use_legacy_sql = use_legacy_sql def get_db_hook(self): -return BigQueryHook(bigquery_conn_id=self.bigquery_conn_id) +return BigQueryHook(bigquery_conn_id=self.bigquery_conn_id, +use_legacy_sql=self.use_legacy_sql) class BigQueryValueCheckOperator(ValueCheckOperator): @@ -78,20 +82,27 @@ class BigQueryValueCheckOperator(ValueCheckOperator): :param sql: the sql to be executed :type sql: string +:param use_legacy_sql: Whether to use legacy SQL (true) +or standard SQL (false). +:type use_legacy_sql: boolean """ @apply_defaults -def __init__( -self, sql, pass_value, tolerance=None, -bigquery_conn_id='bigquery_default', -*args, **kwargs): +def __init__(self, sql, + pass_value, + tolerance=None, + bigquery_conn_id='bigquery_default', + use_legacy_sql=True, + *args, **kwargs): super(BigQueryValueCheckOperator, self).__init__( sql=sql, pass_value=pass_value, tolerance=tolerance, *args, **kwargs) self.bigquery_conn_id = bigquery_conn_id +self.use_legacy_sql = use_legacy_sql def get_db_hook(self): -return BigQueryHook(bigquery_conn_id=self.bigquery_conn_id) +return BigQueryHook(bigquery_conn_id=self.bigquery_conn_id, +use_legacy_sql=self.use_legacy_sql) class BigQueryIntervalCheckOperator(IntervalCheckOperator): @@ -113,19 +124,22 @@ class BigQueryIntervalCheckOperator(IntervalCheckOperator): example 'COUNT(*)': 1.5 would require a 50 percent or less difference between the current day, and the prior days_back. :type metrics_threshold: dict +:param use_legacy_sql: Whether to use legacy SQL (true) +or standard SQL (false). +:type use_legacy_sql: boolean """ @apply_defaults -def __init__( -self, table, metrics_thresholds, -date_filter_column='ds', days_back=-7, -bigquery_conn_id='bigquery_default', -*args, **kwargs): +def __init__(self, table, metrics_thresholds, date_filter_column='ds', + days_back=-7, bigquery_conn_id='bigquery_default', + use_legacy_sql=True, *args, **kwargs): super(BigQueryIntervalCheckOperator, self).__init__( table=table, metrics_thresholds=metrics_thresholds, date_filter_column=date_filter_column, days_back=days_back, *args, **kwargs) self.bigquery_conn_id = bigquery_conn_id +self.use_legacy_sql = use_legacy_sql def get_db_hook(self): -return BigQueryHook(bigquery_conn_id=self.bigquery_conn_id) +return BigQueryHook(bigquery_conn_id=self.bigquery_conn_id, +use_legacy_sql=self.use_legacy_sql) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact
[GitHub] kaxil closed pull request #3717: [AIRFLOW-1874] use_legacy_sql added to BigQueryCheck operators
kaxil closed pull request #3717: [AIRFLOW-1874] use_legacy_sql added to BigQueryCheck operators URL: https://github.com/apache/incubator-airflow/pull/3717 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/airflow/contrib/operators/bigquery_check_operator.py b/airflow/contrib/operators/bigquery_check_operator.py index 59ef5d377d..a9c493f4fd 100644 --- a/airflow/contrib/operators/bigquery_check_operator.py +++ b/airflow/contrib/operators/bigquery_check_operator.py @@ -55,21 +55,25 @@ class BigQueryCheckOperator(CheckOperator): :type sql: string :param bigquery_conn_id: reference to the BigQuery database :type bigquery_conn_id: string +:param use_legacy_sql: Whether to use legacy SQL (true) +or standard SQL (false). +:type use_legacy_sql: boolean """ @apply_defaults -def __init__( -self, -sql, -bigquery_conn_id='bigquery_default', -*args, -**kwargs): +def __init__(self, + sql, + bigquery_conn_id='bigquery_default', + use_legacy_sql=True, + *args, **kwargs): super(BigQueryCheckOperator, self).__init__(sql=sql, *args, **kwargs) self.bigquery_conn_id = bigquery_conn_id self.sql = sql +self.use_legacy_sql = use_legacy_sql def get_db_hook(self): -return BigQueryHook(bigquery_conn_id=self.bigquery_conn_id) +return BigQueryHook(bigquery_conn_id=self.bigquery_conn_id, +use_legacy_sql=self.use_legacy_sql) class BigQueryValueCheckOperator(ValueCheckOperator): @@ -78,20 +82,27 @@ class BigQueryValueCheckOperator(ValueCheckOperator): :param sql: the sql to be executed :type sql: string +:param use_legacy_sql: Whether to use legacy SQL (true) +or standard SQL (false). +:type use_legacy_sql: boolean """ @apply_defaults -def __init__( -self, sql, pass_value, tolerance=None, -bigquery_conn_id='bigquery_default', -*args, **kwargs): +def __init__(self, sql, + pass_value, + tolerance=None, + bigquery_conn_id='bigquery_default', + use_legacy_sql=True, + *args, **kwargs): super(BigQueryValueCheckOperator, self).__init__( sql=sql, pass_value=pass_value, tolerance=tolerance, *args, **kwargs) self.bigquery_conn_id = bigquery_conn_id +self.use_legacy_sql = use_legacy_sql def get_db_hook(self): -return BigQueryHook(bigquery_conn_id=self.bigquery_conn_id) +return BigQueryHook(bigquery_conn_id=self.bigquery_conn_id, +use_legacy_sql=self.use_legacy_sql) class BigQueryIntervalCheckOperator(IntervalCheckOperator): @@ -113,19 +124,22 @@ class BigQueryIntervalCheckOperator(IntervalCheckOperator): example 'COUNT(*)': 1.5 would require a 50 percent or less difference between the current day, and the prior days_back. :type metrics_threshold: dict +:param use_legacy_sql: Whether to use legacy SQL (true) +or standard SQL (false). +:type use_legacy_sql: boolean """ @apply_defaults -def __init__( -self, table, metrics_thresholds, -date_filter_column='ds', days_back=-7, -bigquery_conn_id='bigquery_default', -*args, **kwargs): +def __init__(self, table, metrics_thresholds, date_filter_column='ds', + days_back=-7, bigquery_conn_id='bigquery_default', + use_legacy_sql=True, *args, **kwargs): super(BigQueryIntervalCheckOperator, self).__init__( table=table, metrics_thresholds=metrics_thresholds, date_filter_column=date_filter_column, days_back=days_back, *args, **kwargs) self.bigquery_conn_id = bigquery_conn_id +self.use_legacy_sql = use_legacy_sql def get_db_hook(self): -return BigQueryHook(bigquery_conn_id=self.bigquery_conn_id) +return BigQueryHook(bigquery_conn_id=self.bigquery_conn_id, +use_legacy_sql=self.use_legacy_sql) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-2884) Fix Flask SECRET_KEY security issue in www_rbac
[ https://issues.apache.org/jira/browse/AIRFLOW-2884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-2884. - Resolution: Fixed Fix Version/s: 2.0.0 Resolved by https://github.com/apache/incubator-airflow/pull/3729 > Fix Flask SECRET_KEY security issue in www_rbac > > > Key: AIRFLOW-2884 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2884 > Project: Apache Airflow > Issue Type: Bug > Components: webapp, webserver >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > Fix For: 2.0.0 > > > Flask secret key should be as random as possible, while it's not in Airflow > Flask App. > This issue was fixed for *www* in ticket > https://issues.apache.org/jira/browse/AIRFLOW-2809 (merged in PR > [https://github.com/apache/incubator-airflow/pull/3651)] . > But this issue was not fixed for *www_rbac* yet. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] kaxil commented on a change in pull request #3725: [AIRFLOW-2877] Make docs site URL consistent everywhere
kaxil commented on a change in pull request #3725: [AIRFLOW-2877] Make docs site URL consistent everywhere URL: https://github.com/apache/incubator-airflow/pull/3725#discussion_r209215776 ## File path: README.md ## @@ -24,9 +24,9 @@ monitor progress, and troubleshoot issues when needed. ## Getting started -Please visit the Airflow Platform documentation (latest **stable** release) for help with [installing Airflow](https://airflow.incubator.apache.org/installation.html), getting a [quick start](https://airflow.incubator.apache.org/start.html), or a more complete [tutorial](https://airflow.incubator.apache.org/tutorial.html). +Please visit the Airflow Platform documentation (latest **stable** release) for help with [installing Airflow](https://airflow.apache.org/installation.html), getting a [quick start](https://airflow.apache.org/start.html), or a more complete [tutorial](https://airflow.apache.org/tutorial.html). Review comment: I have raised the issue with Apache Infra over here: https://issues.apache.org/jira/browse/INFRA-16895 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil commented on a change in pull request #3725: [AIRFLOW-2877] Make docs site URL consistent everywhere
kaxil commented on a change in pull request #3725: [AIRFLOW-2877] Make docs site URL consistent everywhere URL: https://github.com/apache/incubator-airflow/pull/3725#discussion_r209203140 ## File path: README.md ## @@ -24,9 +24,9 @@ monitor progress, and troubleshoot issues when needed. ## Getting started -Please visit the Airflow Platform documentation (latest **stable** release) for help with [installing Airflow](https://airflow.incubator.apache.org/installation.html), getting a [quick start](https://airflow.incubator.apache.org/start.html), or a more complete [tutorial](https://airflow.incubator.apache.org/tutorial.html). +Please visit the Airflow Platform documentation (latest **stable** release) for help with [installing Airflow](https://airflow.apache.org/installation.html), getting a [quick start](https://airflow.apache.org/start.html), or a more complete [tutorial](https://airflow.apache.org/tutorial.html). Review comment: Cool. I will raise this today. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb commented on a change in pull request #3725: [AIRFLOW-2877] Make docs site URL consistent everywhere
ashb commented on a change in pull request #3725: [AIRFLOW-2877] Make docs site URL consistent everywhere URL: https://github.com/apache/incubator-airflow/pull/3725#discussion_r209202215 ## File path: README.md ## @@ -24,9 +24,9 @@ monitor progress, and troubleshoot issues when needed. ## Getting started -Please visit the Airflow Platform documentation (latest **stable** release) for help with [installing Airflow](https://airflow.incubator.apache.org/installation.html), getting a [quick start](https://airflow.incubator.apache.org/start.html), or a more complete [tutorial](https://airflow.incubator.apache.org/tutorial.html). +Please visit the Airflow Platform documentation (latest **stable** release) for help with [installing Airflow](https://airflow.apache.org/installation.html), getting a [quick start](https://airflow.apache.org/start.html), or a more complete [tutorial](https://airflow.apache.org/tutorial.html). -Documentation of GitHub master (latest development branch): [ReadTheDocs Documentation](https://airflow.readthedocs.io/en/latest/) +Documentation of GitHub master (latest development branch): [Read the Docs Documentation](https://airflow.readthedocs.io/en/latest/) Review comment: We should mention this distinction on the content on airflow.apache.org too, or use something like https://robpol86.github.io/sphinxcontrib-versioning/ to be able to host multiple versions on the a.a.o site too. Again, unrelated to this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb commented on a change in pull request #3725: [AIRFLOW-2877] Make docs site URL consistent everywhere
ashb commented on a change in pull request #3725: [AIRFLOW-2877] Make docs site URL consistent everywhere URL: https://github.com/apache/incubator-airflow/pull/3725#discussion_r209201782 ## File path: README.md ## @@ -24,9 +24,9 @@ monitor progress, and troubleshoot issues when needed. ## Getting started -Please visit the Airflow Platform documentation (latest **stable** release) for help with [installing Airflow](https://airflow.incubator.apache.org/installation.html), getting a [quick start](https://airflow.incubator.apache.org/start.html), or a more complete [tutorial](https://airflow.incubator.apache.org/tutorial.html). +Please visit the Airflow Platform documentation (latest **stable** release) for help with [installing Airflow](https://airflow.apache.org/installation.html), getting a [quick start](https://airflow.apache.org/start.html), or a more complete [tutorial](https://airflow.apache.org/tutorial.html). -Documentation of GitHub master (latest development branch): [ReadTheDocs Documentation](https://airflow.readthedocs.io/en/latest/) +Documentation of GitHub master (latest development branch): [Read the Docs Documentation](https://airflow.readthedocs.io/en/latest/) Review comment: Unrelated to this PR (so sorry for hijacking) but I would be happier if the default version you get upon visiting https://airflow.readthedocs.io (i.e. from Google) is for the same version you get if you `pip install apache-airflow`. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb commented on a change in pull request #3725: [AIRFLOW-2877] Make docs site URL consistent everywhere
ashb commented on a change in pull request #3725: [AIRFLOW-2877] Make docs site URL consistent everywhere URL: https://github.com/apache/incubator-airflow/pull/3725#discussion_r209201237 ## File path: README.md ## @@ -24,9 +24,9 @@ monitor progress, and troubleshoot issues when needed. ## Getting started -Please visit the Airflow Platform documentation (latest **stable** release) for help with [installing Airflow](https://airflow.incubator.apache.org/installation.html), getting a [quick start](https://airflow.incubator.apache.org/start.html), or a more complete [tutorial](https://airflow.incubator.apache.org/tutorial.html). +Please visit the Airflow Platform documentation (latest **stable** release) for help with [installing Airflow](https://airflow.apache.org/installation.html), getting a [quick start](https://airflow.apache.org/start.html), or a more complete [tutorial](https://airflow.apache.org/tutorial.html). Review comment: We can open an ticket with Apache INFRA to enable a webhook for this. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] seratch opened a new pull request #3732: Fix typos detected by github.com/client9/misspell
seratch opened a new pull request #3732: Fix typos detected by github.com/client9/misspell URL: https://github.com/apache/incubator-airflow/pull/3732 ### Description Fixing typos is sometimes very hard. It's not so easy to visually review them. Recently, I discovered a very useful tool for it, [misspell](https://github.com/client9/misspell). This pull request fixes minor typos detected by [misspell](https://github.com/client9/misspell) except for the false positives. If you would like me to work on other files as well, let me know. before ``` $ misspell airflow/ | grep -v CHANGELOG.txt | grep -v www 2018/08/10 18:14:00 Unable to stat "airflow/www/static/docs": stat airflow/www/static/docs: no such file or directory 2018/08/10 18:14:00 Unable to stat "airflow/www_rbac/static/docs": stat airflow/www_rbac/static/docs: no such file or directory airflow/contrib/example_dags/example_databricks_operator.py:35:22: "succesful" is a misspelling of "successful" airflow/contrib/hooks/emr_hook.py:26:47: "neccessary" is a misspelling of "necessary" airflow/contrib/hooks/azure_fileshare_hook.py:103:21: "direcotry" is a misspelling of "directory" airflow/contrib/hooks/gcp_dataproc_hook.py:238:60: "compatability" is a misspelling of "compatibility" airflow/contrib/hooks/bigquery_hook.py:630:40: "paramaters" is a misspelling of "parameters" airflow/contrib/hooks/qubole_hook.py:128:31: "commmand" is a misspelling of "command" airflow/contrib/hooks/salesforce_hook.py:56:32: "conenction" is a misspelling of "connection" airflow/contrib/hooks/salesforce_hook.py:63:41: "SECRUITY" is a misspelling of "SECURITY" airflow/contrib/operators/gcs_to_bq.py:89:29: "Thsi" is a misspelling of "This" airflow/contrib/operators/mlengine_operator_utils.py:163:69: "prediciton" is a misspelling of "prediction" airflow/contrib/operators/qubole_check_operator.py:31:69: "Commmand" is a misspelling of "Command" airflow/contrib/plugins/metastore_browser/templates/metastore_browser/table.html:31:106: "Atributes" is a misspelling of "Attributes" airflow/hooks/mysql_hook.py:124:27: "seperately" is a misspelling of "separately" airflow/operators/hive_to_druid.py:167:19: "compatibilty" is a misspelling of "compatibility" airflow/sensors/hdfs_sensor.py:91:20: "extentions" is a misspelling of "extensions" airflow/sensors/hdfs_sensor.py:94:24: "extentions" is a misspelling of "extensions" airflow/sensors/hdfs_sensor.py:96:55: "extentions" is a misspelling of "extensions" airflow/models.py:945:71: "acutally" is a misspelling of "actually" ``` after ``` $ misspell airflow/ | grep -v CHANGELOG.txt | grep -v www 2018/08/10 18:13:50 Unable to stat "airflow/www/static/docs": stat airflow/www/static/docs: no such file or directory 2018/08/10 18:13:50 Unable to stat "airflow/www_rbac/static/docs": stat airflow/www_rbac/static/docs: no such file or directory ``` ### Code Quality - [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff` hmm, I am not sure if I can fix this. ``` $ git diff upstream/master -u -- "*.py" | flake8 --dif fatal: bad revision 'upstream/master' ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-2885) A Bug in www_rbac.utils.get_params
[ https://issues.apache.org/jira/browse/AIRFLOW-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575938#comment-16575938 ] ASF GitHub Bot commented on AIRFLOW-2885: - XD-DENG opened a new pull request #3731: [AIRFLOW-2885] Fix a bug in www_rbac.utils.get_params URL: https://github.com/apache/incubator-airflow/pull/3731 ### Jira - [x] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-2885 - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. ### Description - [x] Here are some details about my PR, including screenshots of any UI changes: `get_params(page=0, search="abc",showPaused=False)` returns `search=abc=False`, while it's supposed to return `page=0=abc=False` (`page` is 0-indexed). This issue arose since `0` is considered as `False` by Python in conditional statement. `elif v` will not continue when `v` is `0` here, while it's supposed to continue. A test is added as well. ### Tests - [x] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: ### Commits - [ ] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. ### Code Quality - [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > A Bug in www_rbac.utils.get_params > -- > > Key: AIRFLOW-2885 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2885 > Project: Apache Airflow > Issue Type: Bug > Components: webserver >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > > *get_params(page=0, search="abc",showPaused=False)* returns > "_search=abc=False_", while it's supposed to return > "page=0=abc=False". > This is because Python takes 0 as False when it's used in a conditional > statement. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] XD-DENG opened a new pull request #3731: [AIRFLOW-2885] Fix a bug in www_rbac.utils.get_params
XD-DENG opened a new pull request #3731: [AIRFLOW-2885] Fix a bug in www_rbac.utils.get_params URL: https://github.com/apache/incubator-airflow/pull/3731 ### Jira - [x] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-2885 - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. ### Description - [x] Here are some details about my PR, including screenshots of any UI changes: `get_params(page=0, search="abc",showPaused=False)` returns `search=abc=False`, while it's supposed to return `page=0=abc=False` (`page` is 0-indexed). This issue arose since `0` is considered as `False` by Python in conditional statement. `elif v` will not continue when `v` is `0` here, while it's supposed to continue. A test is added as well. ### Tests - [x] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: ### Commits - [ ] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. ### Code Quality - [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-2885) A Bug in www_rbac.utils.get_params
Xiaodong DENG created AIRFLOW-2885: -- Summary: A Bug in www_rbac.utils.get_params Key: AIRFLOW-2885 URL: https://issues.apache.org/jira/browse/AIRFLOW-2885 Project: Apache Airflow Issue Type: Bug Components: webserver Reporter: Xiaodong DENG Assignee: Xiaodong DENG *get_params(page=0, search="abc",showPaused=False)* returns "_search=abc=False_", while it's supposed to return "page=0=abc=False". This is because Python takes 0 as False when it's used in a conditional statement. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] Fokko commented on issue #3691: [AIRFLOW-2846] Add missing python test dependency to setup.py
Fokko commented on issue #3691: [AIRFLOW-2846] Add missing python test dependency to setup.py URL: https://github.com/apache/incubator-airflow/pull/3691#issuecomment-412000219 I'd say that we would need to remove this tox install as well: https://github.com/apache/incubator-airflow/blob/master/.travis.yml#L98 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on a change in pull request #3730: [AIRFLOW-2882] Add import and export for pool cli using JSON
Fokko commented on a change in pull request #3730: [AIRFLOW-2882] Add import and export for pool cli using JSON URL: https://github.com/apache/incubator-airflow/pull/3730#discussion_r209166353 ## File path: tests/cli/test_cli.py ## @@ -165,3 +166,38 @@ def test_local_run(self): ti.refresh_from_db() state = ti.current_state() self.assertEqual(state, State.SUCCESS) + +def test_cli_pool_import_export(self): +pool_config_input = { +"s3_pool": { +"description": "This is my test s3_pool", +"slots": 5 +}, +"s3_pool2": { +"description": "This is my test s3_pool", +"slots": 8 +} +} +with open('pool_import.json', mode='w', encoding='utf-8') as f: +json.dump(pool_config_input, f) +process_import = psutil.Popen(["airflow", "pool", "-i", "pool_import.json"]) +sleep(3) # wait for webserver to start Review comment: Please poll if the process is available. This makes the test a bit flaky. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil commented on a change in pull request #3725: [AIRFLOW-2877] Make docs site URL consistent everywhere
kaxil commented on a change in pull request #3725: [AIRFLOW-2877] Make docs site URL consistent everywhere URL: https://github.com/apache/incubator-airflow/pull/3725#discussion_r209161488 ## File path: README.md ## @@ -24,9 +24,9 @@ monitor progress, and troubleshoot issues when needed. ## Getting started -Please visit the Airflow Platform documentation (latest **stable** release) for help with [installing Airflow](https://airflow.incubator.apache.org/installation.html), getting a [quick start](https://airflow.incubator.apache.org/start.html), or a more complete [tutorial](https://airflow.incubator.apache.org/tutorial.html). +Please visit the Airflow Platform documentation (latest **stable** release) for help with [installing Airflow](https://airflow.apache.org/installation.html), getting a [quick start](https://airflow.apache.org/start.html), or a more complete [tutorial](https://airflow.apache.org/tutorial.html). Review comment: @r39132 Hi Sid, we had discussed this at the start of this year about using the webhooks but as we didn't have permissions to create hooks on Github, the current process is manual i.e. I try and trigger the build manually regularly. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feng-tao edited a comment on issue #3728: [AIRFLOW-2883] Not search dag owner if owners are missing
feng-tao edited a comment on issue #3728: [AIRFLOW-2883] Not search dag owner if owners are missing URL: https://github.com/apache/incubator-airflow/pull/3728#issuecomment-411962160 hey @r39132 , the issue is because the owners field for the dag is missing(https://github.com/apache/incubator-airflow/blob/master/airflow/models.py#L3087) . And when we search certain regex in the search bar, it will hit this line(https://github.com/apache/incubator-airflow/blob/master/airflow/www/views.py#L2131) which will check for all the dags in dag table whether the dag_id or dag.owners match the search term. If the owners is None, it will throw an exception as following. https://user-images.githubusercontent.com/3223098/43941666-d6bcec66-9c29-11e8-98a1-6debcca66148.png;> This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services