[jira] [Updated] (AIRFLOW-2821) Doc "Plugins" can be improved
[ https://issues.apache.org/jira/browse/AIRFLOW-2821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-2821: --- External issue URL: https://github.com/apache/incubator-airflow/pull/3664 > Doc "Plugins" can be improved > - > > Key: AIRFLOW-2821 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2821 > Project: Apache Airflow > Issue Type: Improvement > Components: Documentation >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > > The doc "Plugins" is not very clear in some sense. > Something that users may need to take note is not mentioned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2821) Doc "Plugins" can be improved
Xiaodong DENG created AIRFLOW-2821: -- Summary: Doc "Plugins" can be improved Key: AIRFLOW-2821 URL: https://issues.apache.org/jira/browse/AIRFLOW-2821 Project: Apache Airflow Issue Type: Improvement Components: Documentation Reporter: Xiaodong DENG Assignee: Xiaodong DENG The doc "Plugins" is not very clear in some sense. Something that users may need to take note is not mentioned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2820) In Doc "Scheduling & Triggers", missed to mention manual Dag Trigger feature in web UI
[ https://issues.apache.org/jira/browse/AIRFLOW-2820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-2820: --- External issue URL: https://github.com/apache/incubator-airflow/pull/3663 > In Doc "Scheduling & Triggers", missed to mention manual Dag Trigger feature > in web UI > --- > > Key: AIRFLOW-2820 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2820 > Project: Apache Airflow > Issue Type: Improvement > Components: Documentation >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > > In documentation page *Scheduling & Triggers*, it only mentioned the CLI > method to manually trigger a DAG run. > However, the manual trigger feature in Web UI should be mentioned as well (it > may be even more frequently used by users). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2820) In Doc "Scheduling & Triggers", missed to mention manual Dag Trigger feature in web UI
Xiaodong DENG created AIRFLOW-2820: -- Summary: In Doc "Scheduling & Triggers", missed to mention manual Dag Trigger feature in web UI Key: AIRFLOW-2820 URL: https://issues.apache.org/jira/browse/AIRFLOW-2820 Project: Apache Airflow Issue Type: Improvement Components: Documentation Reporter: Xiaodong DENG Assignee: Xiaodong DENG In documentation page *Scheduling & Triggers*, it only mentioned the CLI method to manually trigger a DAG run. However, the manual trigger feature in Web UI should be mentioned as well (it may be even more frequently used by users). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2819) Clarify the Timezone support for Cron schedules
Le Van Giap created AIRFLOW-2819: Summary: Clarify the Timezone support for Cron schedules Key: AIRFLOW-2819 URL: https://issues.apache.org/jira/browse/AIRFLOW-2819 Project: Apache Airflow Issue Type: Improvement Components: scheduler Affects Versions: 1.10.0 Reporter: Le Van Giap Assignee: Le Van Giap Fix For: 1.10.0 h3. [https://airflow.readthedocs.io/en/v1-10-test/timezone.html#cron-schedules] The explanation here is quite hard to understand. May I know what is the behavior (running time) of the Scheduler when I have a *America/New_York* Timezone aware DAG? For example: "5 9 * * *" with timezone *America/New_York* -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-695) Retries do not execute because dagrun is in FAILED state
[
https://issues.apache.org/jira/browse/AIRFLOW-695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561356#comment-16561356
]
Brian Kleszyk commented on AIRFLOW-695:
---
Also appear still have this issue in 1.8.1. This was supposedly fixed by 1.8.1.
> Retries do not execute because dagrun is in FAILED state
>
>
> Key: AIRFLOW-695
> URL: https://issues.apache.org/jira/browse/AIRFLOW-695
> Project: Apache Airflow
> Issue Type: Bug
> Components: DagRun
>Reporter: Harvey Xia
>Priority: Blocker
> Labels: executor, scheduler
>
> Currently on the latest master commit
> (15ff540ecd5e60e7ce080177ea3ea227582a4672), running on the LocalExecutor,
> retries on tasks do not execute because the state of the corresponding dagrun
> changes to FAILED. The task instance then gets blocked because "Task
> instance's dagrun was not in the 'running' state but in the state 'failed',"
> the error message produced by the following lines:
> https://github.com/apache/incubator-airflow/blob/master/airflow/ti_deps/deps/dagrun_exists_dep.py#L48-L50
> This error can be reproduced with the following simple DAG:
> {code:title=DAG.py|borderStyle=solid}
> dag = models.DAG(dag_id='test_retry_handling')
> task = BashOperator(
> task_id='test_retry_handling_op',
> bash_command='exit 1',
> retries=1,
> retry_delay=datetime.timedelta(minutes=1),
> dag=dag,
> owner='airflow',
> start_date=datetime.datetime(2016, 2, 1, 0, 0, 0))
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-2813) `pip install apache-airflow` fails
[
https://issues.apache.org/jira/browse/AIRFLOW-2813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561273#comment-16561273
]
Jeff Schwab edited comment on AIRFLOW-2813 at 7/29/18 9:24 PM:
---
Thank you for the response. FYI, the pip install still fails by default on
Alpine (even with Python 3.6) because it is missing prerequisites. Here is a
Dockerfile showing commands that must be run before {{pip install
apache-airflow}}: https://gist.github.com/jeffs/195e4710800a1d7354836ef02db9a1ba
was (Author: unbuggy):
Thank you for the response. FYI, the pip install still fails by default on
Alpine (even with Python 3.6) because it is missing prerequisites. Here is a
[Dockerfile|https://gist.github.com/jeffs/195e4710800a1d7354836ef02db9a1ba]
showing commands that must be run before {{pip install apache-airflow}}.
> `pip install apache-airflow` fails
> --
>
> Key: AIRFLOW-2813
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2813
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.9.0
> Environment: Mac OS, Linux, Windows
>Reporter: Jeff Schwab
>Priority: Major
>
> `pip install apache-airflow` fails with a SyntaxError on Mac OS, and with a
> different (extremely verbose) error on Linux. This happens both on my
> MacBook and on a fresh Alpine Linux Docker image, and with both pip2 and
> pip3; a friend just tried `pip install apache-airflow` for me on his Windows
> box, and it died with yet another error. Googling quickly found someone else
> seeing the same issue over a week ago:
> https://gitter.im/apache/incubator-airflow?at=5b5130bac86c4f0b47201af0
> Please let me know what further information you would like, and/or what I am
> doing wrong.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2813) `pip install apache-airflow` fails
[ https://issues.apache.org/jira/browse/AIRFLOW-2813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Schwab updated AIRFLOW-2813: - Description: `pip install apache-airflow` fails with a SyntaxError on Mac OS, and with a different (extremely verbose) error on Linux. This happens both on my MacBook and on a fresh Alpine Linux Docker image, and with both pip2 and pip3; a friend just tried `pip install apache-airflow` for me on his Windows box, and it died with yet another error. Googling quickly found someone else seeing the same issue over a week ago: https://gitter.im/apache/incubator-airflow?at=5b5130bac86c4f0b47201af0 Please let me know what further information you would like, and/or what I am doing wrong. was: `pip install apache-airflow` fails with a SyntaxError on Mac OS, and with a different (extremely verbose) error on Linux. This happens both on my MacBook and on a fresh Alpine Linux Docker image, and with both pip2 and pip3; a friend just tried `pip install apache-airflow` for me on his Windows box, and it died with yet another error. Googling quickly found someone else seeing the same issue over a week ago: https://gitter.im/apache/incubator-airflow?at=5b5130bac86c4f0b47201af0 Here's a Dockerfile that consistently fails at `pip install apache-airflow`: https://gist.github.com/jeffs/cff7cb01d4ea3969fa5a65a8bd434e5d Please let me know what further information you would like, and/or what I am doing wrong. > `pip install apache-airflow` fails > -- > > Key: AIRFLOW-2813 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2813 > Project: Apache Airflow > Issue Type: Bug >Affects Versions: 1.9.0 > Environment: Mac OS, Linux, Windows >Reporter: Jeff Schwab >Priority: Major > > `pip install apache-airflow` fails with a SyntaxError on Mac OS, and with a > different (extremely verbose) error on Linux. This happens both on my > MacBook and on a fresh Alpine Linux Docker image, and with both pip2 and > pip3; a friend just tried `pip install apache-airflow` for me on his Windows > box, and it died with yet another error. Googling quickly found someone else > seeing the same issue over a week ago: > https://gitter.im/apache/incubator-airflow?at=5b5130bac86c4f0b47201af0 > Please let me know what further information you would like, and/or what I am > doing wrong. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2813) `pip install apache-airflow` fails
[
https://issues.apache.org/jira/browse/AIRFLOW-2813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561273#comment-16561273
]
Jeff Schwab commented on AIRFLOW-2813:
--
Thank you for the response. FYI, the pip install still fails by default on
Alpine (even with Python 3.6) because it is missing prerequisites. Here is a
[Dockerfile|https://gist.github.com/jeffs/195e4710800a1d7354836ef02db9a1ba]
showing commands that must be run before {{pip install apache-airflow}}.
> `pip install apache-airflow` fails
> --
>
> Key: AIRFLOW-2813
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2813
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.9.0
> Environment: Mac OS, Linux, Windows
>Reporter: Jeff Schwab
>Priority: Major
>
> `pip install apache-airflow` fails with a SyntaxError on Mac OS, and with a
> different (extremely verbose) error on Linux. This happens both on my
> MacBook and on a fresh Alpine Linux Docker image, and with both pip2 and
> pip3; a friend just tried `pip install apache-airflow` for me on his Windows
> box, and it died with yet another error. Googling quickly found someone else
> seeing the same issue over a week ago:
> https://gitter.im/apache/incubator-airflow?at=5b5130bac86c4f0b47201af0
> Here's a Dockerfile that consistently fails at `pip install apache-airflow`:
> https://gist.github.com/jeffs/cff7cb01d4ea3969fa5a65a8bd434e5d
> Please let me know what further information you would like, and/or what I am
> doing wrong.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2813) `pip install apache-airflow` fails
[ https://issues.apache.org/jira/browse/AIRFLOW-2813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor resolved AIRFLOW-2813. Resolution: Duplicate Airflow doesn'tt currently support python 3.7 (it made `async` a keyword when it wasn't before. And python 3.7 was released since our last release) This may-or-may-not be fixed in 1.10. If it's not fixed there it will be fixed in the version after that (which may be a 1.10.1, a 1.11 or 2.0. Unclear at the moment) > `pip install apache-airflow` fails > -- > > Key: AIRFLOW-2813 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2813 > Project: Apache Airflow > Issue Type: Bug >Affects Versions: 1.9.0 > Environment: Mac OS, Linux, Windows >Reporter: Jeff Schwab >Priority: Major > > `pip install apache-airflow` fails with a SyntaxError on Mac OS, and with a > different (extremely verbose) error on Linux. This happens both on my > MacBook and on a fresh Alpine Linux Docker image, and with both pip2 and > pip3; a friend just tried `pip install apache-airflow` for me on his Windows > box, and it died with yet another error. Googling quickly found someone else > seeing the same issue over a week ago: > https://gitter.im/apache/incubator-airflow?at=5b5130bac86c4f0b47201af0 > Here's a Dockerfile that consistently fails at `pip install apache-airflow`: > https://gist.github.com/jeffs/cff7cb01d4ea3969fa5a65a8bd434e5d > Please let me know what further information you would like, and/or what I am > doing wrong. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2809) Fix security issue regarding Flask SECRET_KEY
[ https://issues.apache.org/jira/browse/AIRFLOW-2809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-2809. --- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request #3651 [https://github.com/apache/incubator-airflow/pull/3651] > Fix security issue regarding Flask SECRET_KEY > - > > Key: AIRFLOW-2809 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2809 > Project: Apache Airflow > Issue Type: Bug > Components: webserver >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Major > Fix For: 2.0.0 > > > h2. Background > Currently there is a configuration item *secret_key* in the configuration > .cfg file, with a default value "temporary_key". > h2. Issue > Most admins would ignore it and just use the default value "temporary_key". > However, this may be very dangerous. User may modify the cookie if they try > the default SECRET_KEY while the admin didn't change it. > In Flask documentation, it's suggested to have a SECRET_KEY which is as > random as possible ([http://flask.pocoo.org/docs/1.0/quickstart/] ). > h2. My Proposal > If Admin explicitly specified the SECRET_KEY in *.cfg* file, we use this > SECRET_KEY given by Admin. > If the default SECRET_KEY is not changed in *.cfg* file, randomly generate > SECRET_KEY. Meanwhile, print INFO to remind that a randomly generated > SECRET_KEY is used. > This solution will not affect user experience at all. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
incubator-airflow git commit: [AIRFLOW-2809] Fix security issue regarding Flask SECRET_KEY
Repository: incubator-airflow
Updated Branches:
refs/heads/master fcd51f362 -> dfa7b26dd
[AIRFLOW-2809] Fix security issue regarding Flask SECRET_KEY
It's recommended by Falsk community to use random
SECRET_KEY for security reason.
However, in Airflow there is a default value for
secret_key and most users will ignore to change
it.
This may cause security concern.
Closes #3651 from XD-DENG/patch-2
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/dfa7b26d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/dfa7b26d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/dfa7b26d
Branch: refs/heads/master
Commit: dfa7b26ddaca80ee8fd9915ee9f6eac50fac77f6
Parents: fcd51f3
Author: XD-DENG
Authored: Sun Jul 29 11:57:46 2018 +0200
Committer: Fokko Driesprong
Committed: Sun Jul 29 11:57:46 2018 +0200
--
airflow/www/app.py | 13 +++--
1 file changed, 11 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/dfa7b26d/airflow/www/app.py
--
diff --git a/airflow/www/app.py b/airflow/www/app.py
index 6eea5d2..319fe11 100644
--- a/airflow/www/app.py
+++ b/airflow/www/app.py
@@ -18,6 +18,7 @@
# under the License.
#
import six
+import os
from flask import Flask
from flask_admin import Admin, base
@@ -43,9 +44,18 @@ csrf = CSRFProtect()
def create_app(config=None, testing=False):
+
+log = LoggingMixin().log
+
app = Flask(__name__)
app.wsgi_app = ProxyFix(app.wsgi_app)
-app.secret_key = configuration.conf.get('webserver', 'SECRET_KEY')
+
+if configuration.conf.get('webserver', 'SECRET_KEY') == "temporary_key":
+log.info("SECRET_KEY for Flask App is not specified. Using a random
one.")
+app.secret_key = os.urandom(16)
+else:
+app.secret_key = configuration.conf.get('webserver', 'SECRET_KEY')
+
app.config['LOGIN_DISABLED'] = not configuration.conf.getboolean(
'webserver', 'AUTHENTICATE')
@@ -127,7 +137,6 @@ def create_app(config=None, testing=False):
def integrate_plugins():
"""Integrate plugins to the context"""
-log = LoggingMixin().log
from airflow.plugins_manager import (
admin_views, flask_blueprints, menu_links)
for v in admin_views:
[jira] [Commented] (AIRFLOW-2809) Fix security issue regarding Flask SECRET_KEY
[ https://issues.apache.org/jira/browse/AIRFLOW-2809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561060#comment-16561060 ] ASF subversion and git services commented on AIRFLOW-2809: -- Commit dfa7b26ddaca80ee8fd9915ee9f6eac50fac77f6 in incubator-airflow's branch refs/heads/master from XD-DENG [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=dfa7b26 ] [AIRFLOW-2809] Fix security issue regarding Flask SECRET_KEY It's recommended by Falsk community to use random SECRET_KEY for security reason. However, in Airflow there is a default value for secret_key and most users will ignore to change it. This may cause security concern. Closes #3651 from XD-DENG/patch-2 > Fix security issue regarding Flask SECRET_KEY > - > > Key: AIRFLOW-2809 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2809 > Project: Apache Airflow > Issue Type: Bug > Components: webserver >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Major > > h2. Background > Currently there is a configuration item *secret_key* in the configuration > .cfg file, with a default value "temporary_key". > h2. Issue > Most admins would ignore it and just use the default value "temporary_key". > However, this may be very dangerous. User may modify the cookie if they try > the default SECRET_KEY while the admin didn't change it. > In Flask documentation, it's suggested to have a SECRET_KEY which is as > random as possible ([http://flask.pocoo.org/docs/1.0/quickstart/] ). > h2. My Proposal > If Admin explicitly specified the SECRET_KEY in *.cfg* file, we use this > SECRET_KEY given by Admin. > If the default SECRET_KEY is not changed in *.cfg* file, randomly generate > SECRET_KEY. Meanwhile, print INFO to remind that a randomly generated > SECRET_KEY is used. > This solution will not affect user experience at all. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2716) Replace new Python 3.7 keywords
[ https://issues.apache.org/jira/browse/AIRFLOW-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561058#comment-16561058 ] ASF subversion and git services commented on AIRFLOW-2716: -- Commit fcd51f362c6804cb69dee8e2dd054304a0564dde in incubator-airflow's branch refs/heads/master from [~jacobhayes] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=fcd51f3 ] [AIRFLOW-2716] Replace async and await py3.7 keywords Closes #3578 from JacobHayes/py37-keywords > Replace new Python 3.7 keywords > --- > > Key: AIRFLOW-2716 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2716 > Project: Apache Airflow > Issue Type: Bug >Affects Versions: Airflow 2.0 >Reporter: Jacob Hayes >Assignee: Jacob Hayes >Priority: Major > Fix For: 2.0.0 > > > Python 3.7 added `async` and `await` as reserved keywords, so they need to be > replaced with alternative names. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2716) Replace new Python 3.7 keywords
[ https://issues.apache.org/jira/browse/AIRFLOW-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-2716. --- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request #3578 [https://github.com/apache/incubator-airflow/pull/3578] > Replace new Python 3.7 keywords > --- > > Key: AIRFLOW-2716 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2716 > Project: Apache Airflow > Issue Type: Bug >Affects Versions: Airflow 2.0 >Reporter: Jacob Hayes >Assignee: Jacob Hayes >Priority: Major > Fix For: 2.0.0 > > > Python 3.7 added `async` and `await` as reserved keywords, so they need to be > replaced with alternative names. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
incubator-airflow git commit: [AIRFLOW-2716] Replace async and await py3.7 keywords
Repository: incubator-airflow
Updated Branches:
refs/heads/master 9b7525f25 -> fcd51f362
[AIRFLOW-2716] Replace async and await py3.7 keywords
Closes #3578 from JacobHayes/py37-keywords
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/fcd51f36
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/fcd51f36
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/fcd51f36
Branch: refs/heads/master
Commit: fcd51f362c6804cb69dee8e2dd054304a0564dde
Parents: 9b7525f
Author: Jacob Hayes
Authored: Sun Jul 29 11:56:41 2018 +0200
Committer: Fokko Driesprong
Committed: Sun Jul 29 11:56:41 2018 +0200
--
UPDATING.md | 4
airflow/contrib/hooks/gcp_dataproc_hook.py| 12 +++-
airflow/contrib/operators/dataproc_operator.py| 2 +-
airflow/executors/celery_executor.py | 12 ++--
tests/contrib/operators/test_dataproc_operator.py | 8
5 files changed, 26 insertions(+), 12 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/fcd51f36/UPDATING.md
--
diff --git a/UPDATING.md b/UPDATING.md
index 2ca421d..da80f56 100644
--- a/UPDATING.md
+++ b/UPDATING.md
@@ -5,6 +5,10 @@ assists users migrating to a new version.
## Airflow Master
+### Replace DataProcHook.await calls to DataProcHook.wait
+
+The method name was changed to be compatible with the Python 3.7 async/await
keywords
+
### DAG level Access Control for new RBAC UI
Extend and enhance new Airflow RBAC UI to support DAG level ACL. Each dag now
has two permissions(one for write, one for read) associated('can_dag_edit',
'can_dag_read').
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/fcd51f36/airflow/contrib/hooks/gcp_dataproc_hook.py
--
diff --git a/airflow/contrib/hooks/gcp_dataproc_hook.py
b/airflow/contrib/hooks/gcp_dataproc_hook.py
index fc15137..8e4f32b 100644
--- a/airflow/contrib/hooks/gcp_dataproc_hook.py
+++ b/airflow/contrib/hooks/gcp_dataproc_hook.py
@@ -21,6 +21,7 @@ import time
import uuid
from apiclient.discovery import build
+from zope.deprecation import deprecation
from airflow.contrib.hooks.gcp_api_base_hook import GoogleCloudBaseHook
from airflow.utils.log.logging_mixin import LoggingMixin
@@ -224,7 +225,16 @@ class DataProcHook(GoogleCloudBaseHook):
return _DataProcJobBuilder(self.project_id, task_id, cluster_name,
job_type, properties)
-def await(self, operation):
+def wait(self, operation):
"""Awaits for Google Cloud Dataproc Operation to complete."""
submitted = _DataProcOperation(self.get_conn(), operation)
submitted.wait_for_done()
+
+
+setattr(
+DataProcHook,
+"await",
+deprecation.deprecated(
+DataProcHook.wait, "renamed to 'wait' for Python3.7 compatability"
+),
+)
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/fcd51f36/airflow/contrib/operators/dataproc_operator.py
--
diff --git a/airflow/contrib/operators/dataproc_operator.py
b/airflow/contrib/operators/dataproc_operator.py
index 23cfeb0..01d137f 100644
--- a/airflow/contrib/operators/dataproc_operator.py
+++ b/airflow/contrib/operators/dataproc_operator.py
@@ -1265,7 +1265,7 @@ class DataprocWorkflowTemplateBaseOperator(BaseOperator):
)
def execute(self, context):
-self.hook.await(self.start())
+self.hook.wait(self.start())
def start(self, context):
raise AirflowException('plese start a workflow operation')
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/fcd51f36/airflow/executors/celery_executor.py
--
diff --git a/airflow/executors/celery_executor.py
b/airflow/executors/celery_executor.py
index 6cfd2d3..481daa5 100644
--- a/airflow/executors/celery_executor.py
+++ b/airflow/executors/celery_executor.py
@@ -89,9 +89,9 @@ class CeleryExecutor(BaseExecutor):
def sync(self):
self.log.debug("Inquiring about %s celery task(s)", len(self.tasks))
-for key, async in list(self.tasks.items()):
+for key, task in list(self.tasks.items()):
try:
-state = async.state
+state = task.state
if self.last_state[key] != state:
if state == celery_states.SUCCESS:
self.success(key)
@@ -106,8 +106,8 @@ class CeleryExecutor(BaseExecutor):
del self.tasks[key]
del
