[jira] [Commented] (AIRFLOW-2889) Fix typos detected by github.com/client9/misspell
[ https://issues.apache.org/jira/browse/AIRFLOW-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577407#comment-16577407 ] ASF subversion and git services commented on AIRFLOW-2889: -- Commit b78c7fb8512f7a40f58b46530e9b3d5562fe84ea in incubator-airflow's branch refs/heads/master from [~seratch] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=b78c7fb ] [AIRFLOW-2889] Fix typos detected by github.com/client9/misspell (#3732) > Fix typos detected by github.com/client9/misspell > - > > Key: AIRFLOW-2889 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2889 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Kazuhiro Sera >Priority: Minor > > Fixing typos is sometimes very hard. It's not so easy to visually review > them. Recently, I discovered a very useful tool for it, > [misspell](https://github.com/client9/misspell). > This pull request fixes minor typos detected by > [misspell](https://github.com/client9/misspell) except for the false > positives. If you would like me to work on other files as well, let me know. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2889) Fix typos detected by github.com/client9/misspell
[ https://issues.apache.org/jira/browse/AIRFLOW-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577406#comment-16577406 ] ASF GitHub Bot commented on AIRFLOW-2889: - feng-tao closed pull request #3732: [AIRFLOW-2889] Fix typos detected by github.com/client9/misspell URL: https://github.com/apache/incubator-airflow/pull/3732 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/airflow/contrib/example_dags/example_databricks_operator.py b/airflow/contrib/example_dags/example_databricks_operator.py index bc827d465b..79f947ba1c 100644 --- a/airflow/contrib/example_dags/example_databricks_operator.py +++ b/airflow/contrib/example_dags/example_databricks_operator.py @@ -32,7 +32,7 @@ # the spark jar task will NOT run until the notebook task completes # successfully. # -# The definition of a succesful run is if the run has a result_state of "SUCCESS". +# The definition of a successful run is if the run has a result_state of "SUCCESS". # For more information about the state of a run refer to # https://docs.databricks.com/api/latest/jobs.html#runstate diff --git a/airflow/contrib/hooks/azure_fileshare_hook.py b/airflow/contrib/hooks/azure_fileshare_hook.py index edabc17293..d4066ee549 100644 --- a/airflow/contrib/hooks/azure_fileshare_hook.py +++ b/airflow/contrib/hooks/azure_fileshare_hook.py @@ -100,7 +100,7 @@ def list_directories_and_files(self, share_name, directory_name=None, **kwargs): def create_directory(self, share_name, directory_name, **kwargs): """ -Create a new direcotry on a Azure File Share. +Create a new directory on a Azure File Share. :param share_name: Name of the share. :type share_name: str diff --git a/airflow/contrib/hooks/bigquery_hook.py b/airflow/contrib/hooks/bigquery_hook.py index 2a94580f50..e4c0653bfe 100644 --- a/airflow/contrib/hooks/bigquery_hook.py +++ b/airflow/contrib/hooks/bigquery_hook.py @@ -627,7 +627,7 @@ def run_query(self, if query_params: if self.use_legacy_sql: -raise ValueError("Query paramaters are not allowed when using " +raise ValueError("Query parameters are not allowed when using " "legacy SQL") else: configuration['query']['queryParameters'] = query_params diff --git a/airflow/contrib/hooks/emr_hook.py b/airflow/contrib/hooks/emr_hook.py index 6cd92c6d85..d116f2275f 100644 --- a/airflow/contrib/hooks/emr_hook.py +++ b/airflow/contrib/hooks/emr_hook.py @@ -23,7 +23,7 @@ class EmrHook(AwsHook): """ -Interact with AWS EMR. emr_conn_id is only neccessary for using the +Interact with AWS EMR. emr_conn_id is only necessary for using the create_job_flow method. """ diff --git a/airflow/contrib/hooks/gcp_dataproc_hook.py b/airflow/contrib/hooks/gcp_dataproc_hook.py index 8e4f32b137..57c48bde59 100644 --- a/airflow/contrib/hooks/gcp_dataproc_hook.py +++ b/airflow/contrib/hooks/gcp_dataproc_hook.py @@ -235,6 +235,6 @@ def wait(self, operation): DataProcHook, "await", deprecation.deprecated( -DataProcHook.wait, "renamed to 'wait' for Python3.7 compatability" +DataProcHook.wait, "renamed to 'wait' for Python3.7 compatibility" ), ) diff --git a/airflow/contrib/hooks/qubole_hook.py b/airflow/contrib/hooks/qubole_hook.py index 5be5923e7b..3df77d3a1f 100755 --- a/airflow/contrib/hooks/qubole_hook.py +++ b/airflow/contrib/hooks/qubole_hook.py @@ -125,7 +125,7 @@ def execute(self, context): def kill(self, ti): """ -Kill (cancel) a Qubole commmand +Kill (cancel) a Qubole command :param ti: Task Instance of the dag, used to determine the Quboles command id :return: response from Qubole """ diff --git a/airflow/contrib/hooks/salesforce_hook.py b/airflow/contrib/hooks/salesforce_hook.py index 24b67f49fc..efc819e543 100644 --- a/airflow/contrib/hooks/salesforce_hook.py +++ b/airflow/contrib/hooks/salesforce_hook.py @@ -53,14 +53,14 @@ def __init__( :param conn_id: the name of the connection that has the parameters we need to connect to Salesforce. -The conenction shoud be type `http` and include a +The connection shoud be type `http` and include a user's security token in the `Extras` field. .. note:: For the HTTP connection type, you can include a JSON structure in the `Extras` field. We need a user's security token to connect to Salesforce. So we define it in the `Extras`
[GitHub] feng-tao commented on issue #3732: [AIRFLOW-2889] Fix typos detected by github.com/client9/misspell
feng-tao commented on issue #3732: [AIRFLOW-2889] Fix typos detected by github.com/client9/misspell URL: https://github.com/apache/incubator-airflow/pull/3732#issuecomment-412317414 Thanks @seratch . lgtm. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feng-tao closed pull request #3732: [AIRFLOW-2889] Fix typos detected by github.com/client9/misspell
feng-tao closed pull request #3732: [AIRFLOW-2889] Fix typos detected by github.com/client9/misspell URL: https://github.com/apache/incubator-airflow/pull/3732 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/airflow/contrib/example_dags/example_databricks_operator.py b/airflow/contrib/example_dags/example_databricks_operator.py index bc827d465b..79f947ba1c 100644 --- a/airflow/contrib/example_dags/example_databricks_operator.py +++ b/airflow/contrib/example_dags/example_databricks_operator.py @@ -32,7 +32,7 @@ # the spark jar task will NOT run until the notebook task completes # successfully. # -# The definition of a succesful run is if the run has a result_state of "SUCCESS". +# The definition of a successful run is if the run has a result_state of "SUCCESS". # For more information about the state of a run refer to # https://docs.databricks.com/api/latest/jobs.html#runstate diff --git a/airflow/contrib/hooks/azure_fileshare_hook.py b/airflow/contrib/hooks/azure_fileshare_hook.py index edabc17293..d4066ee549 100644 --- a/airflow/contrib/hooks/azure_fileshare_hook.py +++ b/airflow/contrib/hooks/azure_fileshare_hook.py @@ -100,7 +100,7 @@ def list_directories_and_files(self, share_name, directory_name=None, **kwargs): def create_directory(self, share_name, directory_name, **kwargs): """ -Create a new direcotry on a Azure File Share. +Create a new directory on a Azure File Share. :param share_name: Name of the share. :type share_name: str diff --git a/airflow/contrib/hooks/bigquery_hook.py b/airflow/contrib/hooks/bigquery_hook.py index 2a94580f50..e4c0653bfe 100644 --- a/airflow/contrib/hooks/bigquery_hook.py +++ b/airflow/contrib/hooks/bigquery_hook.py @@ -627,7 +627,7 @@ def run_query(self, if query_params: if self.use_legacy_sql: -raise ValueError("Query paramaters are not allowed when using " +raise ValueError("Query parameters are not allowed when using " "legacy SQL") else: configuration['query']['queryParameters'] = query_params diff --git a/airflow/contrib/hooks/emr_hook.py b/airflow/contrib/hooks/emr_hook.py index 6cd92c6d85..d116f2275f 100644 --- a/airflow/contrib/hooks/emr_hook.py +++ b/airflow/contrib/hooks/emr_hook.py @@ -23,7 +23,7 @@ class EmrHook(AwsHook): """ -Interact with AWS EMR. emr_conn_id is only neccessary for using the +Interact with AWS EMR. emr_conn_id is only necessary for using the create_job_flow method. """ diff --git a/airflow/contrib/hooks/gcp_dataproc_hook.py b/airflow/contrib/hooks/gcp_dataproc_hook.py index 8e4f32b137..57c48bde59 100644 --- a/airflow/contrib/hooks/gcp_dataproc_hook.py +++ b/airflow/contrib/hooks/gcp_dataproc_hook.py @@ -235,6 +235,6 @@ def wait(self, operation): DataProcHook, "await", deprecation.deprecated( -DataProcHook.wait, "renamed to 'wait' for Python3.7 compatability" +DataProcHook.wait, "renamed to 'wait' for Python3.7 compatibility" ), ) diff --git a/airflow/contrib/hooks/qubole_hook.py b/airflow/contrib/hooks/qubole_hook.py index 5be5923e7b..3df77d3a1f 100755 --- a/airflow/contrib/hooks/qubole_hook.py +++ b/airflow/contrib/hooks/qubole_hook.py @@ -125,7 +125,7 @@ def execute(self, context): def kill(self, ti): """ -Kill (cancel) a Qubole commmand +Kill (cancel) a Qubole command :param ti: Task Instance of the dag, used to determine the Quboles command id :return: response from Qubole """ diff --git a/airflow/contrib/hooks/salesforce_hook.py b/airflow/contrib/hooks/salesforce_hook.py index 24b67f49fc..efc819e543 100644 --- a/airflow/contrib/hooks/salesforce_hook.py +++ b/airflow/contrib/hooks/salesforce_hook.py @@ -53,14 +53,14 @@ def __init__( :param conn_id: the name of the connection that has the parameters we need to connect to Salesforce. -The conenction shoud be type `http` and include a +The connection shoud be type `http` and include a user's security token in the `Extras` field. .. note:: For the HTTP connection type, you can include a JSON structure in the `Extras` field. We need a user's security token to connect to Salesforce. So we define it in the `Extras` field as: -`{"security_token":"YOUR_SECRUITY_TOKEN"}` +`{"security_token":"YOUR_SECURITY_TOKEN"}` """ self.conn_id = conn_id self._args = args diff --git
[GitHub] seratch commented on issue #3732: Fix typos detected by github.com/client9/misspell
seratch commented on issue #3732: Fix typos detected by github.com/client9/misspell URL: https://github.com/apache/incubator-airflow/pull/3732#issuecomment-412304728 @feng-tao I created an issue on JIRA. Also, the description of this PR has been updated. https://issues.apache.org/jira/browse/AIRFLOW-2889 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-2889) Fix typos detected by github.com/client9/misspell
Kazuhiro Sera created AIRFLOW-2889: -- Summary: Fix typos detected by github.com/client9/misspell Key: AIRFLOW-2889 URL: https://issues.apache.org/jira/browse/AIRFLOW-2889 Project: Apache Airflow Issue Type: Improvement Reporter: Kazuhiro Sera Fixing typos is sometimes very hard. It's not so easy to visually review them. Recently, I discovered a very useful tool for it, [misspell](https://github.com/client9/misspell). This pull request fixes minor typos detected by [misspell](https://github.com/client9/misspell) except for the false positives. If you would like me to work on other files as well, let me know. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] dimberman commented on issue #3740: [AIRFLOW-2888] Remove shell=True and bash from task launch
dimberman commented on issue #3740: [AIRFLOW-2888] Remove shell=True and bash from task launch URL: https://github.com/apache/incubator-airflow/pull/3740#issuecomment-412300993 @bolkedebruin Let me know when you're ready and I'll spin up a k8sexecutor instance on this branch and let you know if anything goes wrong. There _shouldn't_ be any issues and it seems like the k8s tests are passing so I'd imagine this should go smoothly. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] bolkedebruin commented on issue #3702: [AIRFLOW-81] Add ScheduleBlackoutSensor
bolkedebruin commented on issue #3702: [AIRFLOW-81] Add ScheduleBlackoutSensor URL: https://github.com/apache/incubator-airflow/pull/3702#issuecomment-412285492 I think this is better solved by allowing the interval of a dag to have gaps. Pendulum (the underlying lib) allows for this afaik This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] shariat commented on a change in pull request #3739: [AIRFLOW-2868] Extend mesos_executor
shariat commented on a change in pull request #3739: [AIRFLOW-2868] Extend mesos_executor URL: https://github.com/apache/incubator-airflow/pull/3739#discussion_r209429301 ## File path: airflow/contrib/executors/mesos_executor.py ## @@ -238,15 +290,15 @@ def start(self): framework.name = get_framework_name() -if not configuration.conf.get('mesos', 'TASK_CPU'): -task_cpu = 1 -else: -task_cpu = configuration.conf.getint('mesos', 'TASK_CPU') +task_cpu = configuration.conf.getint('mesos', 'TASK_CPU') +task_mem = configuration.conf.getint('mesos', 'TASK_MEMORY') -if not configuration.conf.get('mesos', 'TASK_MEMORY'): -task_memory = 256 -else: -task_memory = configuration.conf.getint('mesos', 'TASK_MEMORY') +docker_image = None +# Configuration errors when DOCKER_IMAGE_SLAVE is not present in config +if 'DOCKER_IMAGE_SLAVE' in configuration.as_dict()['mesos']: Review comment: `try: catch:` ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-2888) Do not use Shell=True and bash to launch tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-2888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577213#comment-16577213 ] ASF GitHub Bot commented on AIRFLOW-2888: - bolkedebruin opened a new pull request #3740: [AIRFLOW-2888] Remove shell=True and bash from task launch URL: https://github.com/apache/incubator-airflow/pull/3740 shell=True is a security risk. Bash is not required to launch tasks and will consume extra resources. Make sure you have checked _all_ steps below. ### Jira - [x] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-2888 - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. ### Description - [X] Here are some details about my PR, including screenshots of any UI changes: ### Tests - [X] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: Tests updated (but not functionally) ### Commits - [X] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [X] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. ### Code Quality - [X] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff` cc: @aoen @mistercrunch @ashb This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Do not use Shell=True and bash to launch tasks > -- > > Key: AIRFLOW-2888 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2888 > Project: Apache Airflow > Issue Type: Bug >Reporter: Bolke de Bruin >Priority: Major > > Using shell=True is a security risk and there is no need to use bash -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] bolkedebruin opened a new pull request #3740: [AIRFLOW-2888] Remove shell=True and bash from task launch
bolkedebruin opened a new pull request #3740: [AIRFLOW-2888] Remove shell=True and bash from task launch URL: https://github.com/apache/incubator-airflow/pull/3740 shell=True is a security risk. Bash is not required to launch tasks and will consume extra resources. Make sure you have checked _all_ steps below. ### Jira - [x] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-2888 - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. ### Description - [X] Here are some details about my PR, including screenshots of any UI changes: ### Tests - [X] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: Tests updated (but not functionally) ### Commits - [X] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [X] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. ### Code Quality - [X] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff` cc: @aoen @mistercrunch @ashb This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] shariat commented on issue #3739: [AIRFLOW-2868] Extend mesos_executor
shariat commented on issue #3739: [AIRFLOW-2868] Extend mesos_executor URL: https://github.com/apache/incubator-airflow/pull/3739#issuecomment-412281622 fix the commit and force push, it is not pointing to ur github account. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-2888) Do not use Shell=True and bash to launch tasks
Bolke de Bruin created AIRFLOW-2888: --- Summary: Do not use Shell=True and bash to launch tasks Key: AIRFLOW-2888 URL: https://issues.apache.org/jira/browse/AIRFLOW-2888 Project: Apache Airflow Issue Type: Bug Reporter: Bolke de Bruin Using shell=True is a security risk and there is no need to use bash -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] boxysean commented on a change in pull request #3702: [AIRFLOW-81] Add ScheduleBlackoutSensor
boxysean commented on a change in pull request #3702: [AIRFLOW-81] Add ScheduleBlackoutSensor URL: https://github.com/apache/incubator-airflow/pull/3702#discussion_r209429062 ## File path: docs/code.rst ## @@ -222,6 +222,7 @@ Sensors .. autoclass:: airflow.contrib.sensors.pubsub_sensor.PubSubPullSensor .. autoclass:: airflow.contrib.sensors.qubole_sensor.QuboleSensor .. autoclass:: airflow.contrib.sensors.redis_key_sensor.RedisKeySensor +.. autoclass:: airflow.contrib.sensors.sftp_sensor.ScheduleBlackoutSensor Review comment: I believe this should be `airflow.contrib.sensors.ScheduleBlackoutSensor`. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services