[jira] [Commented] (AIRFLOW-1794) No Exception.message in Python 3

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247149#comment-16247149
 ] 

ASF subversion and git services commented on AIRFLOW-1794:
--

Commit e2d49f3c82a55069fb1b9cac49cbc1702a3f260c in incubator-airflow's branch 
refs/heads/v1-9-stable from [~dxhuang]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=e2d49f3 ]

[AIRFLOW-1794] Remove uses of Exception.message for Python 3

Closes #2766 from dhuang/AIRFLOW-1794

(cherry picked from commit f5f7701258f858698562579cbfb4e1934056e009)
Signed-off-by: Bolke de Bruin 


> No Exception.message in Python 3
> 
>
> Key: AIRFLOW-1794
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1794
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Daniel Huang
>Assignee: Daniel Huang
>Priority: Minor
>
> [~ashb] ran into this
> {code}
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1988, in 
> wsgi_app
> response = self.full_dispatch_request()
>   File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1641, in 
> full_dispatch_request
> rv = self.handle_user_exception(e)
>   File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1544, in 
> handle_user_exception
> reraise(exc_type, exc_value, tb)
>   File "/usr/local/lib/python3.5/dist-packages/flask/_compat.py", line 33, in 
> reraise
> raise value
>   File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1639, in 
> full_dispatch_request
> rv = self.dispatch_request()
>   File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1625, in 
> dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
>   File "/usr/local/lib/python3.5/dist-packages/flask_admin/base.py", line 69, 
> in inner
> return self._run_view(f, *args, **kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/flask_admin/base.py", line 
> 368, in _run_view
> return fn(self, *args, **kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/flask_login.py", line 758, in 
> decorated_view
> return func(*args, **kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/airflow/www/utils.py", line 
> 262, in wrapper
> return f(*args, **kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/airflow/www/views.py", line 
> 715, in log
> .format(task_log_reader, e.message)]
> AttributeError: 'AttributeError' object has no attribute 'message'
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

incubator-airflow git commit: [AIRFLOW-1794] Remove uses of Exception.message for Python 3

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/v1-9-stable 1f8898e4e -> e2d49f3c8


[AIRFLOW-1794] Remove uses of Exception.message for Python 3

Closes #2766 from dhuang/AIRFLOW-1794

(cherry picked from commit f5f7701258f858698562579cbfb4e1934056e009)
Signed-off-by: Bolke de Bruin 


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/e2d49f3c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/e2d49f3c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/e2d49f3c

Branch: refs/heads/v1-9-stable
Commit: e2d49f3c82a55069fb1b9cac49cbc1702a3f260c
Parents: 1f8898e
Author: Daniel Huang 
Authored: Fri Nov 10 08:57:22 2017 +0100
Committer: Bolke de Bruin 
Committed: Fri Nov 10 08:57:59 2017 +0100

--
 airflow/hooks/webhdfs_hook.py | 2 +-
 airflow/www/views.py  | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/e2d49f3c/airflow/hooks/webhdfs_hook.py
--
diff --git a/airflow/hooks/webhdfs_hook.py b/airflow/hooks/webhdfs_hook.py
index 4510d29..35d4e33 100644
--- a/airflow/hooks/webhdfs_hook.py
+++ b/airflow/hooks/webhdfs_hook.py
@@ -61,7 +61,7 @@ class WebHDFSHook(BaseHook):
 return client
 except HdfsError as e:
 self.log.debug(
-"Read operation on namenode {nn.host} failed witg error: 
{e.message}".format(**locals())
+"Read operation on namenode {nn.host} failed with error: 
{e}".format(**locals())
 )
 nn_hosts = [c.host for c in nn_connections]
 no_nn_error = "Read operations failed on the namenodes 
below:\n{}".format("\n".join(nn_hosts))

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/e2d49f3c/airflow/www/views.py
--
diff --git a/airflow/www/views.py b/airflow/www/views.py
index 960a0d1..63af88f 100644
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -712,7 +712,7 @@ class Airflow(BaseView):
 logs = handler.read(ti)
 except AttributeError as e:
 logs = ["Task log handler {} does not support read 
logs.\n{}\n" \
-.format(task_log_reader, e.message)]
+.format(task_log_reader, str(e))]
 
 for i, log in enumerate(logs):
 if PY2 and not isinstance(log, unicode):

[jira] [Commented] (AIRFLOW-1794) No Exception.message in Python 3

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247147#comment-16247147
 ] 

ASF subversion and git services commented on AIRFLOW-1794:
--

Commit 3bc0b7d33b92758108c90c2afc411e4ea4358afe in incubator-airflow's branch 
refs/heads/v1-9-test from [~dxhuang]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=3bc0b7d ]

[AIRFLOW-1794] Remove uses of Exception.message for Python 3

Closes #2766 from dhuang/AIRFLOW-1794

(cherry picked from commit f5f7701258f858698562579cbfb4e1934056e009)
Signed-off-by: Bolke de Bruin 


> No Exception.message in Python 3
> 
>
> Key: AIRFLOW-1794
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1794
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Daniel Huang
>Assignee: Daniel Huang
>Priority: Minor
>
> [~ashb] ran into this
> {code}
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1988, in 
> wsgi_app
> response = self.full_dispatch_request()
>   File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1641, in 
> full_dispatch_request
> rv = self.handle_user_exception(e)
>   File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1544, in 
> handle_user_exception
> reraise(exc_type, exc_value, tb)
>   File "/usr/local/lib/python3.5/dist-packages/flask/_compat.py", line 33, in 
> reraise
> raise value
>   File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1639, in 
> full_dispatch_request
> rv = self.dispatch_request()
>   File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1625, in 
> dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
>   File "/usr/local/lib/python3.5/dist-packages/flask_admin/base.py", line 69, 
> in inner
> return self._run_view(f, *args, **kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/flask_admin/base.py", line 
> 368, in _run_view
> return fn(self, *args, **kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/flask_login.py", line 758, in 
> decorated_view
> return func(*args, **kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/airflow/www/utils.py", line 
> 262, in wrapper
> return f(*args, **kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/airflow/www/views.py", line 
> 715, in log
> .format(task_log_reader, e.message)]
> AttributeError: 'AttributeError' object has no attribute 'message'
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1794) No Exception.message in Python 3

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247145#comment-16247145
 ] 

ASF subversion and git services commented on AIRFLOW-1794:
--

Commit f5f7701258f858698562579cbfb4e1934056e009 in incubator-airflow's branch 
refs/heads/master from [~dxhuang]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=f5f7701 ]

[AIRFLOW-1794] Remove uses of Exception.message for Python 3

Closes #2766 from dhuang/AIRFLOW-1794


> No Exception.message in Python 3
> 
>
> Key: AIRFLOW-1794
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1794
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Daniel Huang
>Assignee: Daniel Huang
>Priority: Minor
>
> [~ashb] ran into this
> {code}
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1988, in 
> wsgi_app
> response = self.full_dispatch_request()
>   File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1641, in 
> full_dispatch_request
> rv = self.handle_user_exception(e)
>   File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1544, in 
> handle_user_exception
> reraise(exc_type, exc_value, tb)
>   File "/usr/local/lib/python3.5/dist-packages/flask/_compat.py", line 33, in 
> reraise
> raise value
>   File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1639, in 
> full_dispatch_request
> rv = self.dispatch_request()
>   File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1625, in 
> dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
>   File "/usr/local/lib/python3.5/dist-packages/flask_admin/base.py", line 69, 
> in inner
> return self._run_view(f, *args, **kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/flask_admin/base.py", line 
> 368, in _run_view
> return fn(self, *args, **kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/flask_login.py", line 758, in 
> decorated_view
> return func(*args, **kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/airflow/www/utils.py", line 
> 262, in wrapper
> return f(*args, **kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/airflow/www/views.py", line 
> 715, in log
> .format(task_log_reader, e.message)]
> AttributeError: 'AttributeError' object has no attribute 'message'
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1794) No Exception.message in Python 3

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247146#comment-16247146
 ] 

ASF subversion and git services commented on AIRFLOW-1794:
--

Commit f5f7701258f858698562579cbfb4e1934056e009 in incubator-airflow's branch 
refs/heads/master from [~dxhuang]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=f5f7701 ]

[AIRFLOW-1794] Remove uses of Exception.message for Python 3

Closes #2766 from dhuang/AIRFLOW-1794


> No Exception.message in Python 3
> 
>
> Key: AIRFLOW-1794
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1794
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Daniel Huang
>Assignee: Daniel Huang
>Priority: Minor
>
> [~ashb] ran into this
> {code}
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1988, in 
> wsgi_app
> response = self.full_dispatch_request()
>   File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1641, in 
> full_dispatch_request
> rv = self.handle_user_exception(e)
>   File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1544, in 
> handle_user_exception
> reraise(exc_type, exc_value, tb)
>   File "/usr/local/lib/python3.5/dist-packages/flask/_compat.py", line 33, in 
> reraise
> raise value
>   File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1639, in 
> full_dispatch_request
> rv = self.dispatch_request()
>   File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1625, in 
> dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
>   File "/usr/local/lib/python3.5/dist-packages/flask_admin/base.py", line 69, 
> in inner
> return self._run_view(f, *args, **kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/flask_admin/base.py", line 
> 368, in _run_view
> return fn(self, *args, **kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/flask_login.py", line 758, in 
> decorated_view
> return func(*args, **kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/airflow/www/utils.py", line 
> 262, in wrapper
> return f(*args, **kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/airflow/www/views.py", line 
> 715, in log
> .format(task_log_reader, e.message)]
> AttributeError: 'AttributeError' object has no attribute 'message'
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1794) No Exception.message in Python 3

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247148#comment-16247148
 ] 

ASF subversion and git services commented on AIRFLOW-1794:
--

Commit 3bc0b7d33b92758108c90c2afc411e4ea4358afe in incubator-airflow's branch 
refs/heads/v1-9-test from [~dxhuang]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=3bc0b7d ]

[AIRFLOW-1794] Remove uses of Exception.message for Python 3

Closes #2766 from dhuang/AIRFLOW-1794

(cherry picked from commit f5f7701258f858698562579cbfb4e1934056e009)
Signed-off-by: Bolke de Bruin 


> No Exception.message in Python 3
> 
>
> Key: AIRFLOW-1794
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1794
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Daniel Huang
>Assignee: Daniel Huang
>Priority: Minor
>
> [~ashb] ran into this
> {code}
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1988, in 
> wsgi_app
> response = self.full_dispatch_request()
>   File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1641, in 
> full_dispatch_request
> rv = self.handle_user_exception(e)
>   File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1544, in 
> handle_user_exception
> reraise(exc_type, exc_value, tb)
>   File "/usr/local/lib/python3.5/dist-packages/flask/_compat.py", line 33, in 
> reraise
> raise value
>   File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1639, in 
> full_dispatch_request
> rv = self.dispatch_request()
>   File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1625, in 
> dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
>   File "/usr/local/lib/python3.5/dist-packages/flask_admin/base.py", line 69, 
> in inner
> return self._run_view(f, *args, **kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/flask_admin/base.py", line 
> 368, in _run_view
> return fn(self, *args, **kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/flask_login.py", line 758, in 
> decorated_view
> return func(*args, **kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/airflow/www/utils.py", line 
> 262, in wrapper
> return f(*args, **kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/airflow/www/views.py", line 
> 715, in log
> .format(task_log_reader, e.message)]
> AttributeError: 'AttributeError' object has no attribute 'message'
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

incubator-airflow git commit: [AIRFLOW-1794] Remove uses of Exception.message for Python 3

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/v1-9-test a8f3efa99 -> 3bc0b7d33


[AIRFLOW-1794] Remove uses of Exception.message for Python 3

Closes #2766 from dhuang/AIRFLOW-1794

(cherry picked from commit f5f7701258f858698562579cbfb4e1934056e009)
Signed-off-by: Bolke de Bruin 


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/3bc0b7d3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/3bc0b7d3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/3bc0b7d3

Branch: refs/heads/v1-9-test
Commit: 3bc0b7d33b92758108c90c2afc411e4ea4358afe
Parents: a8f3efa
Author: Daniel Huang 
Authored: Fri Nov 10 08:57:22 2017 +0100
Committer: Bolke de Bruin 
Committed: Fri Nov 10 08:57:41 2017 +0100

--
 airflow/hooks/webhdfs_hook.py | 2 +-
 airflow/www/views.py  | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/3bc0b7d3/airflow/hooks/webhdfs_hook.py
--
diff --git a/airflow/hooks/webhdfs_hook.py b/airflow/hooks/webhdfs_hook.py
index 4510d29..35d4e33 100644
--- a/airflow/hooks/webhdfs_hook.py
+++ b/airflow/hooks/webhdfs_hook.py
@@ -61,7 +61,7 @@ class WebHDFSHook(BaseHook):
 return client
 except HdfsError as e:
 self.log.debug(
-"Read operation on namenode {nn.host} failed witg error: 
{e.message}".format(**locals())
+"Read operation on namenode {nn.host} failed with error: 
{e}".format(**locals())
 )
 nn_hosts = [c.host for c in nn_connections]
 no_nn_error = "Read operations failed on the namenodes 
below:\n{}".format("\n".join(nn_hosts))

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/3bc0b7d3/airflow/www/views.py
--
diff --git a/airflow/www/views.py b/airflow/www/views.py
index 960a0d1..63af88f 100644
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -712,7 +712,7 @@ class Airflow(BaseView):
 logs = handler.read(ti)
 except AttributeError as e:
 logs = ["Task log handler {} does not support read 
logs.\n{}\n" \
-.format(task_log_reader, e.message)]
+.format(task_log_reader, str(e))]
 
 for i, log in enumerate(logs):
 if PY2 and not isinstance(log, unicode):

incubator-airflow git commit: [AIRFLOW-1794] Remove uses of Exception.message for Python 3

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master f14f8558b -> f5f770125


[AIRFLOW-1794] Remove uses of Exception.message for Python 3

Closes #2766 from dhuang/AIRFLOW-1794


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/f5f77012
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/f5f77012
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/f5f77012

Branch: refs/heads/master
Commit: f5f7701258f858698562579cbfb4e1934056e009
Parents: f14f855
Author: Daniel Huang 
Authored: Fri Nov 10 08:57:22 2017 +0100
Committer: Bolke de Bruin 
Committed: Fri Nov 10 08:57:22 2017 +0100

--
 airflow/hooks/webhdfs_hook.py | 2 +-
 airflow/www/views.py  | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f5f77012/airflow/hooks/webhdfs_hook.py
--
diff --git a/airflow/hooks/webhdfs_hook.py b/airflow/hooks/webhdfs_hook.py
index 4510d29..35d4e33 100644
--- a/airflow/hooks/webhdfs_hook.py
+++ b/airflow/hooks/webhdfs_hook.py
@@ -61,7 +61,7 @@ class WebHDFSHook(BaseHook):
 return client
 except HdfsError as e:
 self.log.debug(
-"Read operation on namenode {nn.host} failed witg error: 
{e.message}".format(**locals())
+"Read operation on namenode {nn.host} failed with error: 
{e}".format(**locals())
 )
 nn_hosts = [c.host for c in nn_connections]
 no_nn_error = "Read operations failed on the namenodes 
below:\n{}".format("\n".join(nn_hosts))

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f5f77012/airflow/www/views.py
--
diff --git a/airflow/www/views.py b/airflow/www/views.py
index a6d788e..c7869c5 100644
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -712,7 +712,7 @@ class Airflow(BaseView):
 logs = handler.read(ti)
 except AttributeError as e:
 logs = ["Task log handler {} does not support read 
logs.\n{}\n" \
-.format(task_log_reader, e.message)]
+.format(task_log_reader, str(e))]
 
 for i, log in enumerate(logs):
 if PY2 and not isinstance(log, unicode):

[jira] [Commented] (AIRFLOW-1799) models.py raising exception with TypeError: not all arguments converted during string formatting

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247139#comment-16247139
 ] 

ASF subversion and git services commented on AIRFLOW-1799:
--

Commit 1f8898e4e958092cee49dee4d1d18fe3e5a11d0c in incubator-airflow's branch 
refs/heads/v1-9-stable from [~rbuckl]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=1f8898e ]

[AIRFLOW-1799] Fix logging line which raises errors

Closes #2777 from ryancbuckley/log-syntax-fix

(cherry picked from commit f14f8558be08b8175c0f88f8cf02be3e7583151f)
Signed-off-by: Bolke de Bruin 


> models.py raising exception with TypeError: not all arguments converted 
> during string formatting
> 
>
> Key: AIRFLOW-1799
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1799
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: operators
>Reporter: Ryan
>Assignee: Ryan
>Priority: Minor
> Fix For: 1.9.0
>
>
> My airflow install has been logging the message
> {code:java}
> Logged from file models.py, line 2159
> Traceback (most recent call last): File 
> "/usr/lib/python2.7/logging/__init__.py", line 859, in emit msg = 
> self.format(record) File "/usr/lib/python2.7/logging/__init__.py", line 732, 
> in format return fmt.format(record) File 
> "/usr/lib/python2.7/logging/__init__.py", line 471, in format record.message 
> = record.getMessage() File "/usr/lib/python2.7/logging/__init__.py", line 
> 335, in getMessage msg = msg % self.args TypeError: not all arguments 
> converted during string formatting
> {code}
> This is due to a misuse of formatting syntax



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (AIRFLOW-1799) models.py raising exception with TypeError: not all arguments converted during string formatting

[Bolke de Bruin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin resolved AIRFLOW-1799.
-
   Resolution: Fixed
Fix Version/s: 1.9.0

Issue resolved by pull request #2777
[https://github.com/apache/incubator-airflow/pull/2777]

> models.py raising exception with TypeError: not all arguments converted 
> during string formatting
> 
>
> Key: AIRFLOW-1799
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1799
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: operators
>Reporter: Ryan
>Assignee: Ryan
>Priority: Minor
> Fix For: 1.9.0
>
>
> My airflow install has been logging the message
> {code:java}
> Logged from file models.py, line 2159
> Traceback (most recent call last): File 
> "/usr/lib/python2.7/logging/__init__.py", line 859, in emit msg = 
> self.format(record) File "/usr/lib/python2.7/logging/__init__.py", line 732, 
> in format return fmt.format(record) File 
> "/usr/lib/python2.7/logging/__init__.py", line 471, in format record.message 
> = record.getMessage() File "/usr/lib/python2.7/logging/__init__.py", line 
> 335, in getMessage msg = msg % self.args TypeError: not all arguments 
> converted during string formatting
> {code}
> This is due to a misuse of formatting syntax



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

incubator-airflow git commit: [AIRFLOW-1799] Fix logging line which raises errors

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/v1-9-test 9fd765dc0 -> a8f3efa99


[AIRFLOW-1799] Fix logging line which raises errors

Closes #2777 from ryancbuckley/log-syntax-fix

(cherry picked from commit f14f8558be08b8175c0f88f8cf02be3e7583151f)
Signed-off-by: Bolke de Bruin 


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/a8f3efa9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/a8f3efa9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/a8f3efa9

Branch: refs/heads/v1-9-test
Commit: a8f3efa9929f17c913c13c4a0e1b02f7c45c4a64
Parents: 9fd765d
Author: Ryan Buckley 
Authored: Fri Nov 10 08:52:21 2017 +0100
Committer: Bolke de Bruin 
Committed: Fri Nov 10 08:52:37 2017 +0100

--
 airflow/models.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/a8f3efa9/airflow/models.py
--
diff --git a/airflow/models.py b/airflow/models.py
index b4b2325..76f879c 100755
--- a/airflow/models.py
+++ b/airflow/models.py
@@ -2161,7 +2161,7 @@ class BaseOperator(LoggingMixin):
 
 if schedule_interval:
 self.log.warning(
-"schedule_interval is used for {}, though it has "
+"schedule_interval is used for %s, though it has "
 "been deprecated as a task parameter, you need to "
 "specify it as a DAG parameter instead",
 self

incubator-airflow git commit: [AIRFLOW-1799] Fix logging line which raises errors

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/v1-9-stable 40c9c6584 -> 1f8898e4e


[AIRFLOW-1799] Fix logging line which raises errors

Closes #2777 from ryancbuckley/log-syntax-fix

(cherry picked from commit f14f8558be08b8175c0f88f8cf02be3e7583151f)
Signed-off-by: Bolke de Bruin 


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/1f8898e4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/1f8898e4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/1f8898e4

Branch: refs/heads/v1-9-stable
Commit: 1f8898e4e958092cee49dee4d1d18fe3e5a11d0c
Parents: 40c9c65
Author: Ryan Buckley 
Authored: Fri Nov 10 08:52:21 2017 +0100
Committer: Bolke de Bruin 
Committed: Fri Nov 10 08:52:51 2017 +0100

--
 airflow/models.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/1f8898e4/airflow/models.py
--
diff --git a/airflow/models.py b/airflow/models.py
index b4b2325..76f879c 100755
--- a/airflow/models.py
+++ b/airflow/models.py
@@ -2161,7 +2161,7 @@ class BaseOperator(LoggingMixin):
 
 if schedule_interval:
 self.log.warning(
-"schedule_interval is used for {}, though it has "
+"schedule_interval is used for %s, though it has "
 "been deprecated as a task parameter, you need to "
 "specify it as a DAG parameter instead",
 self

[jira] [Commented] (AIRFLOW-1799) models.py raising exception with TypeError: not all arguments converted during string formatting

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247138#comment-16247138
 ] 

ASF subversion and git services commented on AIRFLOW-1799:
--

Commit a8f3efa9929f17c913c13c4a0e1b02f7c45c4a64 in incubator-airflow's branch 
refs/heads/v1-9-test from [~rbuckl]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a8f3efa ]

[AIRFLOW-1799] Fix logging line which raises errors

Closes #2777 from ryancbuckley/log-syntax-fix

(cherry picked from commit f14f8558be08b8175c0f88f8cf02be3e7583151f)
Signed-off-by: Bolke de Bruin 


> models.py raising exception with TypeError: not all arguments converted 
> during string formatting
> 
>
> Key: AIRFLOW-1799
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1799
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: operators
>Reporter: Ryan
>Assignee: Ryan
>Priority: Minor
>
> My airflow install has been logging the message
> {code:java}
> Logged from file models.py, line 2159
> Traceback (most recent call last): File 
> "/usr/lib/python2.7/logging/__init__.py", line 859, in emit msg = 
> self.format(record) File "/usr/lib/python2.7/logging/__init__.py", line 732, 
> in format return fmt.format(record) File 
> "/usr/lib/python2.7/logging/__init__.py", line 471, in format record.message 
> = record.getMessage() File "/usr/lib/python2.7/logging/__init__.py", line 
> 335, in getMessage msg = msg % self.args TypeError: not all arguments 
> converted during string formatting
> {code}
> This is due to a misuse of formatting syntax



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1799) models.py raising exception with TypeError: not all arguments converted during string formatting

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247137#comment-16247137
 ] 

ASF subversion and git services commented on AIRFLOW-1799:
--

Commit f14f8558be08b8175c0f88f8cf02be3e7583151f in incubator-airflow's branch 
refs/heads/master from [~rbuckl]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=f14f855 ]

[AIRFLOW-1799] Fix logging line which raises errors

Closes #2777 from ryancbuckley/log-syntax-fix


> models.py raising exception with TypeError: not all arguments converted 
> during string formatting
> 
>
> Key: AIRFLOW-1799
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1799
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: operators
>Reporter: Ryan
>Assignee: Ryan
>Priority: Minor
>
> My airflow install has been logging the message
> {code:java}
> Logged from file models.py, line 2159
> Traceback (most recent call last): File 
> "/usr/lib/python2.7/logging/__init__.py", line 859, in emit msg = 
> self.format(record) File "/usr/lib/python2.7/logging/__init__.py", line 732, 
> in format return fmt.format(record) File 
> "/usr/lib/python2.7/logging/__init__.py", line 471, in format record.message 
> = record.getMessage() File "/usr/lib/python2.7/logging/__init__.py", line 
> 335, in getMessage msg = msg % self.args TypeError: not all arguments 
> converted during string formatting
> {code}
> This is due to a misuse of formatting syntax



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

incubator-airflow git commit: [AIRFLOW-1799] Fix logging line which raises errors

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master cbb00d405 -> f14f8558b


[AIRFLOW-1799] Fix logging line which raises errors

Closes #2777 from ryancbuckley/log-syntax-fix


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/f14f8558
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/f14f8558
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/f14f8558

Branch: refs/heads/master
Commit: f14f8558be08b8175c0f88f8cf02be3e7583151f
Parents: cbb00d4
Author: Ryan Buckley 
Authored: Fri Nov 10 08:52:21 2017 +0100
Committer: Bolke de Bruin 
Committed: Fri Nov 10 08:52:21 2017 +0100

--
 airflow/models.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f14f8558/airflow/models.py
--
diff --git a/airflow/models.py b/airflow/models.py
index 1686ea7..ab83563 100755
--- a/airflow/models.py
+++ b/airflow/models.py
@@ -2159,7 +2159,7 @@ class BaseOperator(LoggingMixin):
 
 if schedule_interval:
 self.log.warning(
-"schedule_interval is used for {}, though it has "
+"schedule_interval is used for %s, though it has "
 "been deprecated as a task parameter, you need to "
 "specify it as a DAG parameter instead",
 self

[jira] [Resolved] (AIRFLOW-1102) 'airflow webserver' doesn't work with gunicorn 19.4+

[Bolke de Bruin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin resolved AIRFLOW-1102.
-
   Resolution: Fixed
Fix Version/s: 1.9.0

Issue resolved by pull request #2775
[https://github.com/apache/incubator-airflow/pull/2775]

> 'airflow webserver' doesn't work with gunicorn 19.4+
> 
>
> Key: AIRFLOW-1102
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1102
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: dependencies
>Reporter: Yuu Yamashita
> Fix For: 1.9.0
>
>
> {{airflow webserver}} will show an error of "Error: 
> 'airflow.www.gunicorn_config' doesn't exist" if it invoked with gunicorn 
> 19.4+, just like the following example.
> {noformat}
> % airflow webserver
> [2017-04-11 05:34:28,945] {__init__.py:57} INFO - Using executor LocalExecutor
>      _
>  |__( )_  __/__  /  __
>   /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
> ___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
>  _/_/  |_/_/  /_//_//_/  \//|__/
> [2017-04-11 05:34:29,968] [32339] {models.py:167} INFO - Filling up the 
> DagBag from ./dags
> [2017-04-11 05:34:30,427] [32339] {connectionpool.py:735} INFO - Starting new 
> HTTPS connection (1): sts.amazonaws.com
> Running the Gunicorn Server with:
> Workers: 4 sync
> Host: 0.0.0.0:8081
> Timeout: 120
> Logfiles: - -
> =
> Error: 'airflow.www.gunicorn_config' doesn't exist
> {noformat}
> At present, it seems that airflow is configured to require gunicorn prior to 
> 19.4. It should be fixed intrinsically for future upgrade of gunicorn anyways.
> https://github.com/benoitc/gunicorn/blob/19.4/gunicorn/app/base.py#L111-L119
> Starting from {{gunicorn}} 9.4, it seems it started requiring prefix 
> "python:" for module name passed to {{-c}}. Because {{airflow webserver}} [is 
> specifying 
> airflow.www.gunicorn_config|https://github.com/apache/incubator-airflow/blob/6b1c327ee886488eedbe8a8721708b89f37d5560/airflow/bin/cli.py#L779]
>  as {{-c}} for {{gunicorn}}, it won't work with gunicorn 9.4+
> * gunicorn 9.3: 
> https://github.com/benoitc/gunicorn/blob/19.3/gunicorn/app/base.py#L111-L114
> * gunicorn 9.4: 
> https://github.com/benoitc/gunicorn/blob/19.4/gunicorn/app/base.py#L111-L119
> I've opened [a PR to gunicorn|https://github.com/benoitc/gunicorn/pull/1499] 
> to restore original behavior prior to 19.3. However, it'd be also better to 
> be fixed in {{airflow webserver}} itself.
> h5. See also
> * https://github.com/benoitc/gunicorn/pull/1068



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1102) 'airflow webserver' doesn't work with gunicorn 19.4+

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247131#comment-16247131
 ] 

ASF subversion and git services commented on AIRFLOW-1102:
--

Commit 9fd765dc0e547e00039eb2b0fbb1ed418c6abb0a in incubator-airflow's branch 
refs/heads/v1-9-test from [~bcharous]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=9fd765d ]

[AIRFLOW-1102] Upgrade Gunicorn >=19.4.0

Closes #2775 from briancharous/upgrade-gunicorn

(cherry picked from commit cbb00d4055839d8b0bb3473a9ab3942480a373f5)
Signed-off-by: Bolke de Bruin 


> 'airflow webserver' doesn't work with gunicorn 19.4+
> 
>
> Key: AIRFLOW-1102
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1102
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: dependencies
>Reporter: Yuu Yamashita
> Fix For: 1.9.0
>
>
> {{airflow webserver}} will show an error of "Error: 
> 'airflow.www.gunicorn_config' doesn't exist" if it invoked with gunicorn 
> 19.4+, just like the following example.
> {noformat}
> % airflow webserver
> [2017-04-11 05:34:28,945] {__init__.py:57} INFO - Using executor LocalExecutor
>      _
>  |__( )_  __/__  /  __
>   /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
> ___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
>  _/_/  |_/_/  /_//_//_/  \//|__/
> [2017-04-11 05:34:29,968] [32339] {models.py:167} INFO - Filling up the 
> DagBag from ./dags
> [2017-04-11 05:34:30,427] [32339] {connectionpool.py:735} INFO - Starting new 
> HTTPS connection (1): sts.amazonaws.com
> Running the Gunicorn Server with:
> Workers: 4 sync
> Host: 0.0.0.0:8081
> Timeout: 120
> Logfiles: - -
> =
> Error: 'airflow.www.gunicorn_config' doesn't exist
> {noformat}
> At present, it seems that airflow is configured to require gunicorn prior to 
> 19.4. It should be fixed intrinsically for future upgrade of gunicorn anyways.
> https://github.com/benoitc/gunicorn/blob/19.4/gunicorn/app/base.py#L111-L119
> Starting from {{gunicorn}} 9.4, it seems it started requiring prefix 
> "python:" for module name passed to {{-c}}. Because {{airflow webserver}} [is 
> specifying 
> airflow.www.gunicorn_config|https://github.com/apache/incubator-airflow/blob/6b1c327ee886488eedbe8a8721708b89f37d5560/airflow/bin/cli.py#L779]
>  as {{-c}} for {{gunicorn}}, it won't work with gunicorn 9.4+
> * gunicorn 9.3: 
> https://github.com/benoitc/gunicorn/blob/19.3/gunicorn/app/base.py#L111-L114
> * gunicorn 9.4: 
> https://github.com/benoitc/gunicorn/blob/19.4/gunicorn/app/base.py#L111-L119
> I've opened [a PR to gunicorn|https://github.com/benoitc/gunicorn/pull/1499] 
> to restore original behavior prior to 19.3. However, it'd be also better to 
> be fixed in {{airflow webserver}} itself.
> h5. See also
> * https://github.com/benoitc/gunicorn/pull/1068



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1102) 'airflow webserver' doesn't work with gunicorn 19.4+

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247132#comment-16247132
 ] 

ASF subversion and git services commented on AIRFLOW-1102:
--

Commit 40c9c6584058c1fdf78305853a24dc7cf6baaa9c in incubator-airflow's branch 
refs/heads/v1-9-stable from [~bcharous]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=40c9c65 ]

[AIRFLOW-1102] Upgrade Gunicorn >=19.4.0

Closes #2775 from briancharous/upgrade-gunicorn

(cherry picked from commit cbb00d4055839d8b0bb3473a9ab3942480a373f5)
Signed-off-by: Bolke de Bruin 


> 'airflow webserver' doesn't work with gunicorn 19.4+
> 
>
> Key: AIRFLOW-1102
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1102
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: dependencies
>Reporter: Yuu Yamashita
> Fix For: 1.9.0
>
>
> {{airflow webserver}} will show an error of "Error: 
> 'airflow.www.gunicorn_config' doesn't exist" if it invoked with gunicorn 
> 19.4+, just like the following example.
> {noformat}
> % airflow webserver
> [2017-04-11 05:34:28,945] {__init__.py:57} INFO - Using executor LocalExecutor
>      _
>  |__( )_  __/__  /  __
>   /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
> ___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
>  _/_/  |_/_/  /_//_//_/  \//|__/
> [2017-04-11 05:34:29,968] [32339] {models.py:167} INFO - Filling up the 
> DagBag from ./dags
> [2017-04-11 05:34:30,427] [32339] {connectionpool.py:735} INFO - Starting new 
> HTTPS connection (1): sts.amazonaws.com
> Running the Gunicorn Server with:
> Workers: 4 sync
> Host: 0.0.0.0:8081
> Timeout: 120
> Logfiles: - -
> =
> Error: 'airflow.www.gunicorn_config' doesn't exist
> {noformat}
> At present, it seems that airflow is configured to require gunicorn prior to 
> 19.4. It should be fixed intrinsically for future upgrade of gunicorn anyways.
> https://github.com/benoitc/gunicorn/blob/19.4/gunicorn/app/base.py#L111-L119
> Starting from {{gunicorn}} 9.4, it seems it started requiring prefix 
> "python:" for module name passed to {{-c}}. Because {{airflow webserver}} [is 
> specifying 
> airflow.www.gunicorn_config|https://github.com/apache/incubator-airflow/blob/6b1c327ee886488eedbe8a8721708b89f37d5560/airflow/bin/cli.py#L779]
>  as {{-c}} for {{gunicorn}}, it won't work with gunicorn 9.4+
> * gunicorn 9.3: 
> https://github.com/benoitc/gunicorn/blob/19.3/gunicorn/app/base.py#L111-L114
> * gunicorn 9.4: 
> https://github.com/benoitc/gunicorn/blob/19.4/gunicorn/app/base.py#L111-L119
> I've opened [a PR to gunicorn|https://github.com/benoitc/gunicorn/pull/1499] 
> to restore original behavior prior to 19.3. However, it'd be also better to 
> be fixed in {{airflow webserver}} itself.
> h5. See also
> * https://github.com/benoitc/gunicorn/pull/1068



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1102) 'airflow webserver' doesn't work with gunicorn 19.4+

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247130#comment-16247130
 ] 

ASF subversion and git services commented on AIRFLOW-1102:
--

Commit cbb00d4055839d8b0bb3473a9ab3942480a373f5 in incubator-airflow's branch 
refs/heads/master from [~bcharous]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=cbb00d4 ]

[AIRFLOW-1102] Upgrade Gunicorn >=19.4.0

Closes #2775 from briancharous/upgrade-gunicorn


> 'airflow webserver' doesn't work with gunicorn 19.4+
> 
>
> Key: AIRFLOW-1102
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1102
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: dependencies
>Reporter: Yuu Yamashita
>
> {{airflow webserver}} will show an error of "Error: 
> 'airflow.www.gunicorn_config' doesn't exist" if it invoked with gunicorn 
> 19.4+, just like the following example.
> {noformat}
> % airflow webserver
> [2017-04-11 05:34:28,945] {__init__.py:57} INFO - Using executor LocalExecutor
>      _
>  |__( )_  __/__  /  __
>   /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
> ___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
>  _/_/  |_/_/  /_//_//_/  \//|__/
> [2017-04-11 05:34:29,968] [32339] {models.py:167} INFO - Filling up the 
> DagBag from ./dags
> [2017-04-11 05:34:30,427] [32339] {connectionpool.py:735} INFO - Starting new 
> HTTPS connection (1): sts.amazonaws.com
> Running the Gunicorn Server with:
> Workers: 4 sync
> Host: 0.0.0.0:8081
> Timeout: 120
> Logfiles: - -
> =
> Error: 'airflow.www.gunicorn_config' doesn't exist
> {noformat}
> At present, it seems that airflow is configured to require gunicorn prior to 
> 19.4. It should be fixed intrinsically for future upgrade of gunicorn anyways.
> https://github.com/benoitc/gunicorn/blob/19.4/gunicorn/app/base.py#L111-L119
> Starting from {{gunicorn}} 9.4, it seems it started requiring prefix 
> "python:" for module name passed to {{-c}}. Because {{airflow webserver}} [is 
> specifying 
> airflow.www.gunicorn_config|https://github.com/apache/incubator-airflow/blob/6b1c327ee886488eedbe8a8721708b89f37d5560/airflow/bin/cli.py#L779]
>  as {{-c}} for {{gunicorn}}, it won't work with gunicorn 9.4+
> * gunicorn 9.3: 
> https://github.com/benoitc/gunicorn/blob/19.3/gunicorn/app/base.py#L111-L114
> * gunicorn 9.4: 
> https://github.com/benoitc/gunicorn/blob/19.4/gunicorn/app/base.py#L111-L119
> I've opened [a PR to gunicorn|https://github.com/benoitc/gunicorn/pull/1499] 
> to restore original behavior prior to 19.3. However, it'd be also better to 
> be fixed in {{airflow webserver}} itself.
> h5. See also
> * https://github.com/benoitc/gunicorn/pull/1068



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

incubator-airflow git commit: [AIRFLOW-1102] Upgrade Gunicorn >=19.4.0

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/v1-9-stable f7afd5a98 -> 40c9c6584


[AIRFLOW-1102] Upgrade Gunicorn >=19.4.0

Closes #2775 from briancharous/upgrade-gunicorn

(cherry picked from commit cbb00d4055839d8b0bb3473a9ab3942480a373f5)
Signed-off-by: Bolke de Bruin 


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/40c9c658
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/40c9c658
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/40c9c658

Branch: refs/heads/v1-9-stable
Commit: 40c9c6584058c1fdf78305853a24dc7cf6baaa9c
Parents: f7afd5a
Author: Brian Charous 
Authored: Fri Nov 10 08:49:41 2017 +0100
Committer: Bolke de Bruin 
Committed: Fri Nov 10 08:50:27 2017 +0100

--
 airflow/bin/cli.py | 2 +-
 setup.py   | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/40c9c658/airflow/bin/cli.py
--
diff --git a/airflow/bin/cli.py b/airflow/bin/cli.py
index 91ec49b..6d01293 100755
--- a/airflow/bin/cli.py
+++ b/airflow/bin/cli.py
@@ -726,7 +726,7 @@ def webserver(args):
 '-b', args.hostname + ':' + str(args.port),
 '-n', 'airflow-webserver',
 '-p', str(pid),
-'-c', 'airflow.www.gunicorn_config'
+'-c', 'python:airflow.www.gunicorn_config'
 ]
 
 if args.access_logfile:

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/40c9c658/setup.py
--
diff --git a/setup.py b/setup.py
index 2a87c4f..5d33fff 100644
--- a/setup.py
+++ b/setup.py
@@ -222,7 +222,7 @@ def do_setup():
 'funcsigs==1.0.0',
 'future>=0.16.0, <0.17',
 'gitpython>=2.0.2',
-'gunicorn>=19.3.0, <19.4.0',  # 19.4.? seemed to have issues
+'gunicorn>=19.4.0, <20.0',
 'jinja2>=2.7.3, <2.9.0',
 'lxml>=3.6.0, <4.0',
 'markdown>=2.5.2, <3.0',

incubator-airflow git commit: [AIRFLOW-1102] Upgrade Gunicorn >=19.4.0

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master 715602ce6 -> cbb00d405


[AIRFLOW-1102] Upgrade Gunicorn >=19.4.0

Closes #2775 from briancharous/upgrade-gunicorn


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/cbb00d40
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/cbb00d40
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/cbb00d40

Branch: refs/heads/master
Commit: cbb00d4055839d8b0bb3473a9ab3942480a373f5
Parents: 715602c
Author: Brian Charous 
Authored: Fri Nov 10 08:49:41 2017 +0100
Committer: Bolke de Bruin 
Committed: Fri Nov 10 08:49:41 2017 +0100

--
 airflow/bin/cli.py | 2 +-
 setup.py   | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/cbb00d40/airflow/bin/cli.py
--
diff --git a/airflow/bin/cli.py b/airflow/bin/cli.py
index 91ec49b..6d01293 100755
--- a/airflow/bin/cli.py
+++ b/airflow/bin/cli.py
@@ -726,7 +726,7 @@ def webserver(args):
 '-b', args.hostname + ':' + str(args.port),
 '-n', 'airflow-webserver',
 '-p', str(pid),
-'-c', 'airflow.www.gunicorn_config'
+'-c', 'python:airflow.www.gunicorn_config'
 ]
 
 if args.access_logfile:

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/cbb00d40/setup.py
--
diff --git a/setup.py b/setup.py
index 3cce836..a4fdf0e 100644
--- a/setup.py
+++ b/setup.py
@@ -224,7 +224,7 @@ def do_setup():
 'funcsigs==1.0.0',
 'future>=0.16.0, <0.17',
 'gitpython>=2.0.2',
-'gunicorn>=19.3.0, <19.4.0',  # 19.4.? seemed to have issues
+'gunicorn>=19.4.0, <20.0',
 'jinja2>=2.7.3, <2.9.0',
 'lxml>=3.6.0, <4.0',
 'markdown>=2.5.2, <3.0',

[jira] [Comment Edited] (AIRFLOW-1021) Double logging required for new users with LDAP

[Alan Ma (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16238542#comment-16238542
 ] 

Alan Ma edited comment on AIRFLOW-1021 at 11/10/17 12:03 AM:
-

Submitted a PR in your place. 
https://github.com/apache/incubator-airflow/pull/2778


was (Author: ama):
Submitted a PR in your place. 
https://github.com/apache/incubator-airflow/pull/2751

> Double logging required for new users with LDAP
> ---
>
> Key: AIRFLOW-1021
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1021
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: contrib
>Affects Versions: Airflow 1.8
>Reporter: Marcelo G. Almiron
>Assignee: Alan Ma
>
> Every user needs to login twice to access Airflow for the first time with 
> LDAP. 
> In the first trial the user is not persistent, so there is no `id` 
> associated, which leads to `None` value returned by `load_user(userid)`, 
> since `userid` is none `None`.
> A quick fix is to add the new user to the session and commit before merging.  
> That is, in module `airflow/contrib/auth/backends/ldap_auth.py`,  we can 
> change
> ```
> if not user:
> user = models.User(
> username=username,
> is_superuser=False)
> session.merge(user)
> session.commit()
> flask_login.login_user(LdapUser(user))
> ```
> by
> ```
> if not user:
> user = models.User(
> username=username,
> is_superuser=False)
> session.add(user)
> 
> session.commit()
> session.merge(user)
> flask_login.login_user(LdapUser(user))
> ```



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (AIRFLOW-1755) URL Prefix for both Flower and Web admin

[Semet (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Semet updated AIRFLOW-1755:
---
Description: 
Similar to AIRFLOW-339.
Should also fix AIRFLOW-964.

I propose in this change 2 new settings on {{airflow.cfg}} and the CLI 
arguments:
{code}
# Root URL to use for the web server
web_server_url_prefix = /airflow
...
# The root URL for Flower
flower_url_prefix = /flower
{code}

This will allow deploying airflow on only on a root FQDN, like 
{{airflow.mycompany.com}}, but also on a path.

  was:
Similar to AIRFLOW-339.
Should also fix AIRFLOW-964.

I propose in this change 2 new settings on {{airflow.cfg}} and the CLI 
arguments:
{code}
# Root URL to use for the web server
web_server_url_prefix: /flower
...
# The root URL for Flower
flower_url_prefix = /flower
{code}

This will allow deploying airflow on only on a root FQDN, like 
{{airflow.mycompany.com}}, but also on a path.


> URL Prefix for both Flower and Web admin
> 
>
> Key: AIRFLOW-1755
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1755
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: cli, core
>Affects Versions: Airflow 2.0
>Reporter: Semet
>Assignee: Semet
>Priority: Minor
>
> Similar to AIRFLOW-339.
> Should also fix AIRFLOW-964.
> I propose in this change 2 new settings on {{airflow.cfg}} and the CLI 
> arguments:
> {code}
> # Root URL to use for the web server
> web_server_url_prefix = /airflow
> ...
> # The root URL for Flower
> flower_url_prefix = /flower
> {code}
> This will allow deploying airflow on only on a root FQDN, like 
> {{airflow.mycompany.com}}, but also on a path.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (AIRFLOW-1799) models.py raising exception with TypeError: not all arguments converted during string formatting

[Ryan (JIRA)]

Ryan created AIRFLOW-1799:
-

 Summary: models.py raising exception with TypeError: not all 
arguments converted during string formatting
 Key: AIRFLOW-1799
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1799
 Project: Apache Airflow
  Issue Type: Bug
  Components: operators
Reporter: Ryan
Assignee: Ryan
Priority: Minor


Logged from file models.py, line 2159
Traceback (most recent call last): File 
"/usr/lib/python2.7/logging/__init__.py", line 859, in emit msg = 
self.format(record) File "/usr/lib/python2.7/logging/__init__.py", line 732, in 
format return fmt.format(record) File "/usr/lib/python2.7/logging/__init__.py", 
line 471, in format record.message = record.getMessage() File 
"/usr/lib/python2.7/logging/__init__.py", line 335, in getMessage msg = msg % 
self.args TypeError: not all arguments converted during string formatting

This is due to a misuse of formatting syntax



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (AIRFLOW-1799) models.py raising exception with TypeError: not all arguments converted during string formatting

[Ryan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryan updated AIRFLOW-1799:
--
Description: 
My airflow install has been logging the message
{code:java}
Logged from file models.py, line 2159
Traceback (most recent call last): File 
"/usr/lib/python2.7/logging/__init__.py", line 859, in emit msg = 
self.format(record) File "/usr/lib/python2.7/logging/__init__.py", line 732, in 
format return fmt.format(record) File "/usr/lib/python2.7/logging/__init__.py", 
line 471, in format record.message = record.getMessage() File 
"/usr/lib/python2.7/logging/__init__.py", line 335, in getMessage msg = msg % 
self.args TypeError: not all arguments converted during string formatting
{code}

This is due to a misuse of formatting syntax

  was:
Logged from file models.py, line 2159
Traceback (most recent call last): File 
"/usr/lib/python2.7/logging/__init__.py", line 859, in emit msg = 
self.format(record) File "/usr/lib/python2.7/logging/__init__.py", line 732, in 
format return fmt.format(record) File "/usr/lib/python2.7/logging/__init__.py", 
line 471, in format record.message = record.getMessage() File 
"/usr/lib/python2.7/logging/__init__.py", line 335, in getMessage msg = msg % 
self.args TypeError: not all arguments converted during string formatting

This is due to a misuse of formatting syntax


> models.py raising exception with TypeError: not all arguments converted 
> during string formatting
> 
>
> Key: AIRFLOW-1799
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1799
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: operators
>Reporter: Ryan
>Assignee: Ryan
>Priority: Minor
>
> My airflow install has been logging the message
> {code:java}
> Logged from file models.py, line 2159
> Traceback (most recent call last): File 
> "/usr/lib/python2.7/logging/__init__.py", line 859, in emit msg = 
> self.format(record) File "/usr/lib/python2.7/logging/__init__.py", line 732, 
> in format return fmt.format(record) File 
> "/usr/lib/python2.7/logging/__init__.py", line 471, in format record.message 
> = record.getMessage() File "/usr/lib/python2.7/logging/__init__.py", line 
> 335, in getMessage msg = msg % self.args TypeError: not all arguments 
> converted during string formatting
> {code}
> This is due to a misuse of formatting syntax



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1756) S3 Task Handler Cannot Read Logs With New S3Hook

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246497#comment-16246497
 ] 

ASF subversion and git services commented on AIRFLOW-1756:
--

Commit 715602ce6a78d773ca85397cf8a0fa85afe42b74 in incubator-airflow's branch 
refs/heads/master from [~ashb]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=715602c ]

[AIRFLOW-1756] Fix S3TaskHandler to work with Boto3-based S3Hook

The change from boto2 to boto3 in S3Hook caused
this to break (the
return type of `hook.get_key()` changed. There's a
better method
designed for that we should use anyway.

This wasn't caught by the tests as the mocks
weren't updated. Rather
than mocking the return of the hook I have changed
it to use "moto"
(already in use elsewhere in the tests) to mock at
the S3 layer, not
our hook.

Closes #2773 from ashb/AIRFLOW-1756-s3-logging-
boto3-fix


> S3 Task Handler Cannot Read Logs With New S3Hook
> 
>
> Key: AIRFLOW-1756
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1756
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Colin Son
>Priority: Critical
> Fix For: 1.9.0
>
>
> With the changes to the S3Hook, it seems like it cannot read the S3 task logs.
> In the `s3_read` in the S3TaskHandler.py:
> {code}
> s3_key = self.hook.get_key(remote_log_location)
> if s3_key:
> return s3_key.get_contents_as_string().decode()
> {code}
> Since the s3_key object is now a dict, you cannot call 
> `get_contents_as_string()` on a dict object. You have to use the S3Hook's 
> `read_key()` method to read the contents of the task logs now. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

incubator-airflow git commit: [AIRFLOW-1756] Fix S3TaskHandler to work with Boto3-based S3Hook

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master 28411b1e7 -> 715602ce6


[AIRFLOW-1756] Fix S3TaskHandler to work with Boto3-based S3Hook

The change from boto2 to boto3 in S3Hook caused
this to break (the
return type of `hook.get_key()` changed. There's a
better method
designed for that we should use anyway.

This wasn't caught by the tests as the mocks
weren't updated. Rather
than mocking the return of the hook I have changed
it to use "moto"
(already in use elsewhere in the tests) to mock at
the S3 layer, not
our hook.

Closes #2773 from ashb/AIRFLOW-1756-s3-logging-
boto3-fix


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/715602ce
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/715602ce
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/715602ce

Branch: refs/heads/master
Commit: 715602ce6a78d773ca85397cf8a0fa85afe42b74
Parents: 28411b1
Author: Ash Berlin-Taylor 
Authored: Thu Nov 9 21:57:04 2017 +0100
Committer: Bolke de Bruin 
Committed: Thu Nov 9 21:57:04 2017 +0100

--
 airflow/utils/log/s3_task_handler.py|  12 ++-
 tests/utils/log/test_s3_task_handler.py | 128 +++
 2 files changed, 74 insertions(+), 66 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/715602ce/airflow/utils/log/s3_task_handler.py
--
diff --git a/airflow/utils/log/s3_task_handler.py 
b/airflow/utils/log/s3_task_handler.py
index 1e56655..cfa966a 100644
--- a/airflow/utils/log/s3_task_handler.py
+++ b/airflow/utils/log/s3_task_handler.py
@@ -127,14 +127,12 @@ class S3TaskHandler(FileTaskHandler, LoggingMixin):
 :type return_error: bool
 """
 try:
-s3_key = self.hook.get_key(remote_log_location)
-if s3_key:
-return s3_key.get_contents_as_string().decode()
+return self.hook.read_key(remote_log_location)
 except:
+msg = 'Could not read logs from {}'.format(remote_log_location)
+self.log.exception(msg)
 # return error if needed
 if return_error:
-msg = 'Could not read logs from {}'.format(remote_log_location)
-self.log.error(msg)
 return msg
 
 def s3_write(self, log, remote_log_location, append=True):
@@ -149,7 +147,7 @@ class S3TaskHandler(FileTaskHandler, LoggingMixin):
 the new log is appended to any existing logs.
 :type append: bool
 """
-if append:
+if append and self.s3_log_exists(remote_log_location):
 old_log = self.s3_read(remote_log_location)
 log = '\n'.join([old_log, log]) if old_log else log
 
@@ -161,4 +159,4 @@ class S3TaskHandler(FileTaskHandler, LoggingMixin):
 encrypt=configuration.getboolean('core', 'ENCRYPT_S3_LOGS'),
 )
 except:
-self.log.error('Could not write logs to %s', remote_log_location)
+self.log.exception('Could not write logs to %s', 
remote_log_location)

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/715602ce/tests/utils/log/test_s3_task_handler.py
--
diff --git a/tests/utils/log/test_s3_task_handler.py 
b/tests/utils/log/test_s3_task_handler.py
index da879b6..b1354cd 100644
--- a/tests/utils/log/test_s3_task_handler.py
+++ b/tests/utils/log/test_s3_task_handler.py
@@ -12,46 +12,66 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from datetime import datetime
 import mock
 import unittest
 
+from airflow import configuration
 from airflow.utils.log.s3_task_handler import S3TaskHandler
+from airflow.hooks.S3_hook import S3Hook
+from airflow.models import TaskInstance, DAG
+from airflow.operators.dummy_operator import DummyOperator
 
+try:
+import boto3
+import moto
+from moto import mock_s3
+except ImportError:
+mock_s3 = None
 
+
+@unittest.skipIf(mock_s3 is None,
+ "Skipping test because moto.mock_s3 is not available")
+@mock_s3
 class TestS3TaskHandler(unittest.TestCase):
 
 def setUp(self):
 super(TestS3TaskHandler, self).setUp()
-self.remote_log_location = 'remote/log/location'
+self.remote_log_location = 's3://bucket/remote/log/location'
+self.remote_log_key = 'remote/log/location'
 self.local_log_location = 'local/log/location'
-self.s3_log_location = 's3/log/location'
-self.filename_template = ''
-self.hook_patcher = mock.patch("airflow.hooks.S3_hook.S3Hook")
-self.hook_mock = 

[jira] [Updated] (AIRFLOW-1793) DockerOperator doesn't work with docker_conn_id

[Cedrik Neumann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cedrik Neumann updated AIRFLOW-1793:

Affects Version/s: (was: 1.9.0)

> DockerOperator doesn't work with docker_conn_id
> ---
>
> Key: AIRFLOW-1793
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1793
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Cedrik Neumann
>Assignee: Cedrik Neumann
>
> The implementation of DockerOperator uses `self.base_url` when loading the 
> DockerHook instead of `self.docker_url`:
> https://github.com/apache/incubator-airflow/blob/v1-9-stable/airflow/operators/docker_operator.py#L150
> {noformat}
> [2017-11-08 16:10:13,082] {base_task_runner.py:98} INFO - Subtask:   File 
> "/src/apache-airflow/airflow/operators/docker_operator.py", line 161, in 
> execute
> [2017-11-08 16:10:13,083] {base_task_runner.py:98} INFO - Subtask: 
> self.cli = self.get_hook().get_conn()
> [2017-11-08 16:10:13,083] {base_task_runner.py:98} INFO - Subtask:   File 
> "/src/apache-airflow/airflow/operators/docker_operator.py", line 150, in 
> get_hook
> [2017-11-08 16:10:13,083] {base_task_runner.py:98} INFO - Subtask: 
> base_url=self.base_url,
> [2017-11-08 16:10:13,083] {base_task_runner.py:98} INFO - Subtask: 
> AttributeError: 'DockerOperator' object has no attribute 'base_url'
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (AIRFLOW-1793) DockerOperator doesn't work with docker_conn_id

[Cedrik Neumann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cedrik Neumann reassigned AIRFLOW-1793:
---

Assignee: Cedrik Neumann

> DockerOperator doesn't work with docker_conn_id
> ---
>
> Key: AIRFLOW-1793
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1793
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Cedrik Neumann
>Assignee: Cedrik Neumann
>
> The implementation of DockerOperator uses `self.base_url` when loading the 
> DockerHook instead of `self.docker_url`:
> https://github.com/apache/incubator-airflow/blob/v1-9-stable/airflow/operators/docker_operator.py#L150
> {noformat}
> [2017-11-08 16:10:13,082] {base_task_runner.py:98} INFO - Subtask:   File 
> "/src/apache-airflow/airflow/operators/docker_operator.py", line 161, in 
> execute
> [2017-11-08 16:10:13,083] {base_task_runner.py:98} INFO - Subtask: 
> self.cli = self.get_hook().get_conn()
> [2017-11-08 16:10:13,083] {base_task_runner.py:98} INFO - Subtask:   File 
> "/src/apache-airflow/airflow/operators/docker_operator.py", line 150, in 
> get_hook
> [2017-11-08 16:10:13,083] {base_task_runner.py:98} INFO - Subtask: 
> base_url=self.base_url,
> [2017-11-08 16:10:13,083] {base_task_runner.py:98} INFO - Subtask: 
> AttributeError: 'DockerOperator' object has no attribute 'base_url'
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1756) S3 Task Handler Cannot Read Logs With New S3Hook

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246512#comment-16246512
 ] 

ASF subversion and git services commented on AIRFLOW-1756:
--

Commit f7afd5a9858218b839fec10774dfadfd1a478969 in incubator-airflow's branch 
refs/heads/v1-9-stable from [~ashb]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=f7afd5a ]

[AIRFLOW-1756] Fix S3TaskHandler to work with Boto3-based S3Hook

The change from boto2 to boto3 in S3Hook caused
this to break (the
return type of `hook.get_key()` changed. There's a
better method
designed for that we should use anyway.

This wasn't caught by the tests as the mocks
weren't updated. Rather
than mocking the return of the hook I have changed
it to use "moto"
(already in use elsewhere in the tests) to mock at
the S3 layer, not
our hook.

Closes #2773 from ashb/AIRFLOW-1756-s3-logging-
boto3-fix

(cherry picked from commit 715602ce6a78d773ca85397cf8a0fa85afe42b74)
Signed-off-by: Bolke de Bruin 


> S3 Task Handler Cannot Read Logs With New S3Hook
> 
>
> Key: AIRFLOW-1756
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1756
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Colin Son
>Priority: Critical
> Fix For: 1.9.0
>
>
> With the changes to the S3Hook, it seems like it cannot read the S3 task logs.
> In the `s3_read` in the S3TaskHandler.py:
> {code}
> s3_key = self.hook.get_key(remote_log_location)
> if s3_key:
> return s3_key.get_contents_as_string().decode()
> {code}
> Since the s3_key object is now a dict, you cannot call 
> `get_contents_as_string()` on a dict object. You have to use the S3Hook's 
> `read_key()` method to read the contents of the task logs now. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (AIRFLOW-1756) S3 Task Handler Cannot Read Logs With New S3Hook

[Bolke de Bruin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin resolved AIRFLOW-1756.
-
Resolution: Fixed

Issue resolved by pull request #2773
[https://github.com/apache/incubator-airflow/pull/2773]

> S3 Task Handler Cannot Read Logs With New S3Hook
> 
>
> Key: AIRFLOW-1756
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1756
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Colin Son
>Priority: Critical
> Fix For: 1.9.0
>
>
> With the changes to the S3Hook, it seems like it cannot read the S3 task logs.
> In the `s3_read` in the S3TaskHandler.py:
> {code}
> s3_key = self.hook.get_key(remote_log_location)
> if s3_key:
> return s3_key.get_contents_as_string().decode()
> {code}
> Since the s3_key object is now a dict, you cannot call 
> `get_contents_as_string()` on a dict object. You have to use the S3Hook's 
> `read_key()` method to read the contents of the task logs now. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1756) S3 Task Handler Cannot Read Logs With New S3Hook

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246508#comment-16246508
 ] 

ASF subversion and git services commented on AIRFLOW-1756:
--

Commit 1f417efc8c88ac68f343e6a30ec11b98ea15eced in incubator-airflow's branch 
refs/heads/v1-9-test from [~ashb]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=1f417ef ]

[AIRFLOW-1756] Fix S3TaskHandler to work with Boto3-based S3Hook

The change from boto2 to boto3 in S3Hook caused
this to break (the
return type of `hook.get_key()` changed. There's a
better method
designed for that we should use anyway.

This wasn't caught by the tests as the mocks
weren't updated. Rather
than mocking the return of the hook I have changed
it to use "moto"
(already in use elsewhere in the tests) to mock at
the S3 layer, not
our hook.

Closes #2773 from ashb/AIRFLOW-1756-s3-logging-
boto3-fix

(cherry picked from commit 715602ce6a78d773ca85397cf8a0fa85afe42b74)
Signed-off-by: Bolke de Bruin 


> S3 Task Handler Cannot Read Logs With New S3Hook
> 
>
> Key: AIRFLOW-1756
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1756
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Colin Son
>Priority: Critical
> Fix For: 1.9.0
>
>
> With the changes to the S3Hook, it seems like it cannot read the S3 task logs.
> In the `s3_read` in the S3TaskHandler.py:
> {code}
> s3_key = self.hook.get_key(remote_log_location)
> if s3_key:
> return s3_key.get_contents_as_string().decode()
> {code}
> Since the s3_key object is now a dict, you cannot call 
> `get_contents_as_string()` on a dict object. You have to use the S3Hook's 
> `read_key()` method to read the contents of the task logs now. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

incubator-airflow git commit: [AIRFLOW-1756] Fix S3TaskHandler to work with Boto3-based S3Hook

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/v1-9-test d592f891e -> 1f417efc8


[AIRFLOW-1756] Fix S3TaskHandler to work with Boto3-based S3Hook

The change from boto2 to boto3 in S3Hook caused
this to break (the
return type of `hook.get_key()` changed. There's a
better method
designed for that we should use anyway.

This wasn't caught by the tests as the mocks
weren't updated. Rather
than mocking the return of the hook I have changed
it to use "moto"
(already in use elsewhere in the tests) to mock at
the S3 layer, not
our hook.

Closes #2773 from ashb/AIRFLOW-1756-s3-logging-
boto3-fix

(cherry picked from commit 715602ce6a78d773ca85397cf8a0fa85afe42b74)
Signed-off-by: Bolke de Bruin 


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/1f417efc
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/1f417efc
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/1f417efc

Branch: refs/heads/v1-9-test
Commit: 1f417efc8c88ac68f343e6a30ec11b98ea15eced
Parents: d592f89
Author: Ash Berlin-Taylor 
Authored: Thu Nov 9 21:57:04 2017 +0100
Committer: Bolke de Bruin 
Committed: Thu Nov 9 21:59:53 2017 +0100

--
 airflow/utils/log/s3_task_handler.py|  12 +--
 tests/utils/log/test_s3_task_handler.py | 152 +++
 2 files changed, 91 insertions(+), 73 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/1f417efc/airflow/utils/log/s3_task_handler.py
--
diff --git a/airflow/utils/log/s3_task_handler.py 
b/airflow/utils/log/s3_task_handler.py
index 1e56655..cfa966a 100644
--- a/airflow/utils/log/s3_task_handler.py
+++ b/airflow/utils/log/s3_task_handler.py
@@ -127,14 +127,12 @@ class S3TaskHandler(FileTaskHandler, LoggingMixin):
 :type return_error: bool
 """
 try:
-s3_key = self.hook.get_key(remote_log_location)
-if s3_key:
-return s3_key.get_contents_as_string().decode()
+return self.hook.read_key(remote_log_location)
 except:
+msg = 'Could not read logs from {}'.format(remote_log_location)
+self.log.exception(msg)
 # return error if needed
 if return_error:
-msg = 'Could not read logs from {}'.format(remote_log_location)
-self.log.error(msg)
 return msg
 
 def s3_write(self, log, remote_log_location, append=True):
@@ -149,7 +147,7 @@ class S3TaskHandler(FileTaskHandler, LoggingMixin):
 the new log is appended to any existing logs.
 :type append: bool
 """
-if append:
+if append and self.s3_log_exists(remote_log_location):
 old_log = self.s3_read(remote_log_location)
 log = '\n'.join([old_log, log]) if old_log else log
 
@@ -161,4 +159,4 @@ class S3TaskHandler(FileTaskHandler, LoggingMixin):
 encrypt=configuration.getboolean('core', 'ENCRYPT_S3_LOGS'),
 )
 except:
-self.log.error('Could not write logs to %s', remote_log_location)
+self.log.exception('Could not write logs to %s', 
remote_log_location)

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/1f417efc/tests/utils/log/test_s3_task_handler.py
--
diff --git a/tests/utils/log/test_s3_task_handler.py 
b/tests/utils/log/test_s3_task_handler.py
index 0438630..b1354cd 100644
--- a/tests/utils/log/test_s3_task_handler.py
+++ b/tests/utils/log/test_s3_task_handler.py
@@ -12,103 +12,123 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from datetime import datetime
 import mock
 import unittest
 
+from airflow import configuration
 from airflow.utils.log.s3_task_handler import S3TaskHandler
+from airflow.hooks.S3_hook import S3Hook
+from airflow.models import TaskInstance, DAG
+from airflow.operators.dummy_operator import DummyOperator
 
+try:
+import boto3
+import moto
+from moto import mock_s3
+except ImportError:
+mock_s3 = None
 
-@unittest.skip("Non functional S3 tests")
+
+@unittest.skipIf(mock_s3 is None,
+ "Skipping test because moto.mock_s3 is not available")
+@mock_s3
 class TestS3TaskHandler(unittest.TestCase):
 
 def setUp(self):
 super(TestS3TaskHandler, self).setUp()
-self.remote_log_location = 'remote/log/location'
-self.hook_patcher = mock.patch("airflow.hooks.S3_hook.S3Hook")
-self.hook_mock = self.hook_patcher.start()
-self.hook_inst_mock = self.hook_mock.return_value
-

[jira] [Commented] (AIRFLOW-1756) S3 Task Handler Cannot Read Logs With New S3Hook

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246513#comment-16246513
 ] 

ASF subversion and git services commented on AIRFLOW-1756:
--

Commit f7afd5a9858218b839fec10774dfadfd1a478969 in incubator-airflow's branch 
refs/heads/v1-9-stable from [~ashb]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=f7afd5a ]

[AIRFLOW-1756] Fix S3TaskHandler to work with Boto3-based S3Hook

The change from boto2 to boto3 in S3Hook caused
this to break (the
return type of `hook.get_key()` changed. There's a
better method
designed for that we should use anyway.

This wasn't caught by the tests as the mocks
weren't updated. Rather
than mocking the return of the hook I have changed
it to use "moto"
(already in use elsewhere in the tests) to mock at
the S3 layer, not
our hook.

Closes #2773 from ashb/AIRFLOW-1756-s3-logging-
boto3-fix

(cherry picked from commit 715602ce6a78d773ca85397cf8a0fa85afe42b74)
Signed-off-by: Bolke de Bruin 


> S3 Task Handler Cannot Read Logs With New S3Hook
> 
>
> Key: AIRFLOW-1756
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1756
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Colin Son
>Priority: Critical
> Fix For: 1.9.0
>
>
> With the changes to the S3Hook, it seems like it cannot read the S3 task logs.
> In the `s3_read` in the S3TaskHandler.py:
> {code}
> s3_key = self.hook.get_key(remote_log_location)
> if s3_key:
> return s3_key.get_contents_as_string().decode()
> {code}
> Since the s3_key object is now a dict, you cannot call 
> `get_contents_as_string()` on a dict object. You have to use the S3Hook's 
> `read_key()` method to read the contents of the task logs now. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1756) S3 Task Handler Cannot Read Logs With New S3Hook

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246509#comment-16246509
 ] 

ASF subversion and git services commented on AIRFLOW-1756:
--

Commit 1f417efc8c88ac68f343e6a30ec11b98ea15eced in incubator-airflow's branch 
refs/heads/v1-9-test from [~ashb]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=1f417ef ]

[AIRFLOW-1756] Fix S3TaskHandler to work with Boto3-based S3Hook

The change from boto2 to boto3 in S3Hook caused
this to break (the
return type of `hook.get_key()` changed. There's a
better method
designed for that we should use anyway.

This wasn't caught by the tests as the mocks
weren't updated. Rather
than mocking the return of the hook I have changed
it to use "moto"
(already in use elsewhere in the tests) to mock at
the S3 layer, not
our hook.

Closes #2773 from ashb/AIRFLOW-1756-s3-logging-
boto3-fix

(cherry picked from commit 715602ce6a78d773ca85397cf8a0fa85afe42b74)
Signed-off-by: Bolke de Bruin 


> S3 Task Handler Cannot Read Logs With New S3Hook
> 
>
> Key: AIRFLOW-1756
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1756
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Colin Son
>Priority: Critical
> Fix For: 1.9.0
>
>
> With the changes to the S3Hook, it seems like it cannot read the S3 task logs.
> In the `s3_read` in the S3TaskHandler.py:
> {code}
> s3_key = self.hook.get_key(remote_log_location)
> if s3_key:
> return s3_key.get_contents_as_string().decode()
> {code}
> Since the s3_key object is now a dict, you cannot call 
> `get_contents_as_string()` on a dict object. You have to use the S3Hook's 
> `read_key()` method to read the contents of the task logs now. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

incubator-airflow git commit: [AIRFLOW-1756] Fix S3TaskHandler to work with Boto3-based S3Hook

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/v1-9-stable 6b7c17d17 -> f7afd5a98


[AIRFLOW-1756] Fix S3TaskHandler to work with Boto3-based S3Hook

The change from boto2 to boto3 in S3Hook caused
this to break (the
return type of `hook.get_key()` changed. There's a
better method
designed for that we should use anyway.

This wasn't caught by the tests as the mocks
weren't updated. Rather
than mocking the return of the hook I have changed
it to use "moto"
(already in use elsewhere in the tests) to mock at
the S3 layer, not
our hook.

Closes #2773 from ashb/AIRFLOW-1756-s3-logging-
boto3-fix

(cherry picked from commit 715602ce6a78d773ca85397cf8a0fa85afe42b74)
Signed-off-by: Bolke de Bruin 


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/f7afd5a9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/f7afd5a9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/f7afd5a9

Branch: refs/heads/v1-9-stable
Commit: f7afd5a9858218b839fec10774dfadfd1a478969
Parents: 6b7c17d
Author: Ash Berlin-Taylor 
Authored: Thu Nov 9 21:57:04 2017 +0100
Committer: Bolke de Bruin 
Committed: Thu Nov 9 22:00:51 2017 +0100

--
 airflow/utils/log/s3_task_handler.py|  12 +--
 tests/utils/log/test_s3_task_handler.py | 152 +++
 2 files changed, 91 insertions(+), 73 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f7afd5a9/airflow/utils/log/s3_task_handler.py
--
diff --git a/airflow/utils/log/s3_task_handler.py 
b/airflow/utils/log/s3_task_handler.py
index 1e56655..cfa966a 100644
--- a/airflow/utils/log/s3_task_handler.py
+++ b/airflow/utils/log/s3_task_handler.py
@@ -127,14 +127,12 @@ class S3TaskHandler(FileTaskHandler, LoggingMixin):
 :type return_error: bool
 """
 try:
-s3_key = self.hook.get_key(remote_log_location)
-if s3_key:
-return s3_key.get_contents_as_string().decode()
+return self.hook.read_key(remote_log_location)
 except:
+msg = 'Could not read logs from {}'.format(remote_log_location)
+self.log.exception(msg)
 # return error if needed
 if return_error:
-msg = 'Could not read logs from {}'.format(remote_log_location)
-self.log.error(msg)
 return msg
 
 def s3_write(self, log, remote_log_location, append=True):
@@ -149,7 +147,7 @@ class S3TaskHandler(FileTaskHandler, LoggingMixin):
 the new log is appended to any existing logs.
 :type append: bool
 """
-if append:
+if append and self.s3_log_exists(remote_log_location):
 old_log = self.s3_read(remote_log_location)
 log = '\n'.join([old_log, log]) if old_log else log
 
@@ -161,4 +159,4 @@ class S3TaskHandler(FileTaskHandler, LoggingMixin):
 encrypt=configuration.getboolean('core', 'ENCRYPT_S3_LOGS'),
 )
 except:
-self.log.error('Could not write logs to %s', remote_log_location)
+self.log.exception('Could not write logs to %s', 
remote_log_location)

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f7afd5a9/tests/utils/log/test_s3_task_handler.py
--
diff --git a/tests/utils/log/test_s3_task_handler.py 
b/tests/utils/log/test_s3_task_handler.py
index 0438630..b1354cd 100644
--- a/tests/utils/log/test_s3_task_handler.py
+++ b/tests/utils/log/test_s3_task_handler.py
@@ -12,103 +12,123 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from datetime import datetime
 import mock
 import unittest
 
+from airflow import configuration
 from airflow.utils.log.s3_task_handler import S3TaskHandler
+from airflow.hooks.S3_hook import S3Hook
+from airflow.models import TaskInstance, DAG
+from airflow.operators.dummy_operator import DummyOperator
 
+try:
+import boto3
+import moto
+from moto import mock_s3
+except ImportError:
+mock_s3 = None
 
-@unittest.skip("Non functional S3 tests")
+
+@unittest.skipIf(mock_s3 is None,
+ "Skipping test because moto.mock_s3 is not available")
+@mock_s3
 class TestS3TaskHandler(unittest.TestCase):
 
 def setUp(self):
 super(TestS3TaskHandler, self).setUp()
-self.remote_log_location = 'remote/log/location'
-self.hook_patcher = mock.patch("airflow.hooks.S3_hook.S3Hook")
-self.hook_mock = self.hook_patcher.start()
-self.hook_inst_mock = self.hook_mock.return_value
-

[jira] [Commented] (AIRFLOW-1756) S3 Task Handler Cannot Read Logs With New S3Hook

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246498#comment-16246498
 ] 

ASF subversion and git services commented on AIRFLOW-1756:
--

Commit 715602ce6a78d773ca85397cf8a0fa85afe42b74 in incubator-airflow's branch 
refs/heads/master from [~ashb]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=715602c ]

[AIRFLOW-1756] Fix S3TaskHandler to work with Boto3-based S3Hook

The change from boto2 to boto3 in S3Hook caused
this to break (the
return type of `hook.get_key()` changed. There's a
better method
designed for that we should use anyway.

This wasn't caught by the tests as the mocks
weren't updated. Rather
than mocking the return of the hook I have changed
it to use "moto"
(already in use elsewhere in the tests) to mock at
the S3 layer, not
our hook.

Closes #2773 from ashb/AIRFLOW-1756-s3-logging-
boto3-fix


> S3 Task Handler Cannot Read Logs With New S3Hook
> 
>
> Key: AIRFLOW-1756
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1756
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Colin Son
>Priority: Critical
> Fix For: 1.9.0
>
>
> With the changes to the S3Hook, it seems like it cannot read the S3 task logs.
> In the `s3_read` in the S3TaskHandler.py:
> {code}
> s3_key = self.hook.get_key(remote_log_location)
> if s3_key:
> return s3_key.get_contents_as_string().decode()
> {code}
> Since the s3_key object is now a dict, you cannot call 
> `get_contents_as_string()` on a dict object. You have to use the S3Hook's 
> `read_key()` method to read the contents of the task logs now. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1797) Cannot write task logs to S3 with Python3

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246385#comment-16246385
 ] 

ASF subversion and git services commented on AIRFLOW-1797:
--

Commit 28411b1e7eddb3338a329db3e52ee09de3676784 in incubator-airflow's branch 
refs/heads/master from [~ashb]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=28411b1 ]

[AIRFLOW-1797] S3Hook.load_string didn't work on Python3

With the switch to Boto3 we now need the content
to be bytes, not a
string. On Python2 there is no difference, but for
Python3 this matters.

And since there were no real tests covering the
S3Hook I've added some
basic ones.

Closes #2771 from ashb/AIRFLOW-1797


> Cannot write task logs to S3 with Python3
> -
>
> Key: AIRFLOW-1797
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1797
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Ash Berlin-Taylor
>
> {noformat}
> Traceback (most recent call last):
>   File 
> "/usr/local/lib/python3.5/dist-packages/airflow/utils/log/s3_task_handler.py",
>  line 161, in s3_write
> encrypt=configuration.getboolean('core', 'ENCRYPT_S3_LOGS'),
>   File "/usr/local/lib/python3.5/dist-packages/airflow/hooks/S3_hook.py", 
> line 253, in load_string
> client.upload_fileobj(filelike_buffer, bucket_name, key, 
> ExtraArgs=extra_args)
>   File "/usr/local/lib/python3.5/dist-packages/boto3/s3/inject.py", line 431, 
> in upload_fileobj
> return future.result()
>   File "/usr/local/lib/python3.5/dist-packages/s3transfer/futures.py", line 
> 73, in result
> return self._coordinator.result()
>   File "/usr/local/lib/python3.5/dist-packages/s3transfer/futures.py", line 
> 233, in result
> raise self._exception
>   File "/usr/local/lib/python3.5/dist-packages/s3transfer/tasks.py", line 
> 126, in __call__
> return self._execute_main(kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/s3transfer/tasks.py", line 
> 150, in _execute_main
> return_value = self._main(**kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/s3transfer/upload.py", line 
> 679, in _main
> client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/client.py", line 312, 
> in _api_call
> return self._make_api_call(operation_name, kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/client.py", line 586, 
> in _make_api_call
> request_signer=self._request_signer, context=request_context)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/hooks.py", line 242, 
> in emit_until_response
> responses = self._emit(event_name, kwargs, stop_on_response=True)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/hooks.py", line 210, 
> in _emit
> response = handler(**kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/handlers.py", line 
> 201, in conditionally_calculate_md5
> calculate_md5(params, **kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/handlers.py", line 
> 179, in calculate_md5
> binary_md5 = _calculate_md5_from_file(body)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/handlers.py", line 
> 193, in _calculate_md5_from_file
> md5.update(chunk)
> TypeError: Unicode-objects must be encoded before hashing
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1797) Cannot write task logs to S3 with Python3

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246391#comment-16246391
 ] 

ASF subversion and git services commented on AIRFLOW-1797:
--

Commit 6b7c17d17b664c74d507dc006eb12cd023feb837 in incubator-airflow's branch 
refs/heads/v1-9-stable from [~ashb]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=6b7c17d ]

[AIRFLOW-1797] S3Hook.load_string didn't work on Python3

With the switch to Boto3 we now need the content
to be bytes, not a
string. On Python2 there is no difference, but for
Python3 this matters.

And since there were no real tests covering the
S3Hook I've added some
basic ones.

Closes #2771 from ashb/AIRFLOW-1797

(cherry picked from commit 28411b1e7eddb3338a329db3e52ee09de3676784)
Signed-off-by: Bolke de Bruin 


> Cannot write task logs to S3 with Python3
> -
>
> Key: AIRFLOW-1797
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1797
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Ash Berlin-Taylor
> Fix For: 1.9.1
>
>
> {noformat}
> Traceback (most recent call last):
>   File 
> "/usr/local/lib/python3.5/dist-packages/airflow/utils/log/s3_task_handler.py",
>  line 161, in s3_write
> encrypt=configuration.getboolean('core', 'ENCRYPT_S3_LOGS'),
>   File "/usr/local/lib/python3.5/dist-packages/airflow/hooks/S3_hook.py", 
> line 253, in load_string
> client.upload_fileobj(filelike_buffer, bucket_name, key, 
> ExtraArgs=extra_args)
>   File "/usr/local/lib/python3.5/dist-packages/boto3/s3/inject.py", line 431, 
> in upload_fileobj
> return future.result()
>   File "/usr/local/lib/python3.5/dist-packages/s3transfer/futures.py", line 
> 73, in result
> return self._coordinator.result()
>   File "/usr/local/lib/python3.5/dist-packages/s3transfer/futures.py", line 
> 233, in result
> raise self._exception
>   File "/usr/local/lib/python3.5/dist-packages/s3transfer/tasks.py", line 
> 126, in __call__
> return self._execute_main(kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/s3transfer/tasks.py", line 
> 150, in _execute_main
> return_value = self._main(**kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/s3transfer/upload.py", line 
> 679, in _main
> client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/client.py", line 312, 
> in _api_call
> return self._make_api_call(operation_name, kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/client.py", line 586, 
> in _make_api_call
> request_signer=self._request_signer, context=request_context)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/hooks.py", line 242, 
> in emit_until_response
> responses = self._emit(event_name, kwargs, stop_on_response=True)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/hooks.py", line 210, 
> in _emit
> response = handler(**kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/handlers.py", line 
> 201, in conditionally_calculate_md5
> calculate_md5(params, **kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/handlers.py", line 
> 179, in calculate_md5
> binary_md5 = _calculate_md5_from_file(body)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/handlers.py", line 
> 193, in _calculate_md5_from_file
> md5.update(chunk)
> TypeError: Unicode-objects must be encoded before hashing
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (AIRFLOW-1797) Cannot write task logs to S3 with Python3

[Bolke de Bruin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin resolved AIRFLOW-1797.
-
   Resolution: Fixed
Fix Version/s: 1.9.1

Issue resolved by pull request #2771
[https://github.com/apache/incubator-airflow/pull/2771]

> Cannot write task logs to S3 with Python3
> -
>
> Key: AIRFLOW-1797
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1797
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Ash Berlin-Taylor
> Fix For: 1.9.1
>
>
> {noformat}
> Traceback (most recent call last):
>   File 
> "/usr/local/lib/python3.5/dist-packages/airflow/utils/log/s3_task_handler.py",
>  line 161, in s3_write
> encrypt=configuration.getboolean('core', 'ENCRYPT_S3_LOGS'),
>   File "/usr/local/lib/python3.5/dist-packages/airflow/hooks/S3_hook.py", 
> line 253, in load_string
> client.upload_fileobj(filelike_buffer, bucket_name, key, 
> ExtraArgs=extra_args)
>   File "/usr/local/lib/python3.5/dist-packages/boto3/s3/inject.py", line 431, 
> in upload_fileobj
> return future.result()
>   File "/usr/local/lib/python3.5/dist-packages/s3transfer/futures.py", line 
> 73, in result
> return self._coordinator.result()
>   File "/usr/local/lib/python3.5/dist-packages/s3transfer/futures.py", line 
> 233, in result
> raise self._exception
>   File "/usr/local/lib/python3.5/dist-packages/s3transfer/tasks.py", line 
> 126, in __call__
> return self._execute_main(kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/s3transfer/tasks.py", line 
> 150, in _execute_main
> return_value = self._main(**kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/s3transfer/upload.py", line 
> 679, in _main
> client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/client.py", line 312, 
> in _api_call
> return self._make_api_call(operation_name, kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/client.py", line 586, 
> in _make_api_call
> request_signer=self._request_signer, context=request_context)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/hooks.py", line 242, 
> in emit_until_response
> responses = self._emit(event_name, kwargs, stop_on_response=True)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/hooks.py", line 210, 
> in _emit
> response = handler(**kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/handlers.py", line 
> 201, in conditionally_calculate_md5
> calculate_md5(params, **kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/handlers.py", line 
> 179, in calculate_md5
> binary_md5 = _calculate_md5_from_file(body)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/handlers.py", line 
> 193, in _calculate_md5_from_file
> md5.update(chunk)
> TypeError: Unicode-objects must be encoded before hashing
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1797) Cannot write task logs to S3 with Python3

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246392#comment-16246392
 ] 

ASF subversion and git services commented on AIRFLOW-1797:
--

Commit 6b7c17d17b664c74d507dc006eb12cd023feb837 in incubator-airflow's branch 
refs/heads/v1-9-stable from [~ashb]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=6b7c17d ]

[AIRFLOW-1797] S3Hook.load_string didn't work on Python3

With the switch to Boto3 we now need the content
to be bytes, not a
string. On Python2 there is no difference, but for
Python3 this matters.

And since there were no real tests covering the
S3Hook I've added some
basic ones.

Closes #2771 from ashb/AIRFLOW-1797

(cherry picked from commit 28411b1e7eddb3338a329db3e52ee09de3676784)
Signed-off-by: Bolke de Bruin 


> Cannot write task logs to S3 with Python3
> -
>
> Key: AIRFLOW-1797
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1797
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Ash Berlin-Taylor
> Fix For: 1.9.1
>
>
> {noformat}
> Traceback (most recent call last):
>   File 
> "/usr/local/lib/python3.5/dist-packages/airflow/utils/log/s3_task_handler.py",
>  line 161, in s3_write
> encrypt=configuration.getboolean('core', 'ENCRYPT_S3_LOGS'),
>   File "/usr/local/lib/python3.5/dist-packages/airflow/hooks/S3_hook.py", 
> line 253, in load_string
> client.upload_fileobj(filelike_buffer, bucket_name, key, 
> ExtraArgs=extra_args)
>   File "/usr/local/lib/python3.5/dist-packages/boto3/s3/inject.py", line 431, 
> in upload_fileobj
> return future.result()
>   File "/usr/local/lib/python3.5/dist-packages/s3transfer/futures.py", line 
> 73, in result
> return self._coordinator.result()
>   File "/usr/local/lib/python3.5/dist-packages/s3transfer/futures.py", line 
> 233, in result
> raise self._exception
>   File "/usr/local/lib/python3.5/dist-packages/s3transfer/tasks.py", line 
> 126, in __call__
> return self._execute_main(kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/s3transfer/tasks.py", line 
> 150, in _execute_main
> return_value = self._main(**kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/s3transfer/upload.py", line 
> 679, in _main
> client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/client.py", line 312, 
> in _api_call
> return self._make_api_call(operation_name, kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/client.py", line 586, 
> in _make_api_call
> request_signer=self._request_signer, context=request_context)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/hooks.py", line 242, 
> in emit_until_response
> responses = self._emit(event_name, kwargs, stop_on_response=True)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/hooks.py", line 210, 
> in _emit
> response = handler(**kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/handlers.py", line 
> 201, in conditionally_calculate_md5
> calculate_md5(params, **kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/handlers.py", line 
> 179, in calculate_md5
> binary_md5 = _calculate_md5_from_file(body)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/handlers.py", line 
> 193, in _calculate_md5_from_file
> md5.update(chunk)
> TypeError: Unicode-objects must be encoded before hashing
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1797) Cannot write task logs to S3 with Python3

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246388#comment-16246388
 ] 

ASF subversion and git services commented on AIRFLOW-1797:
--

Commit d592f891e58650472c8fba89bace3cce54a7972b in incubator-airflow's branch 
refs/heads/v1-9-test from [~ashb]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=d592f89 ]

[AIRFLOW-1797] S3Hook.load_string didn't work on Python3

With the switch to Boto3 we now need the content
to be bytes, not a
string. On Python2 there is no difference, but for
Python3 this matters.

And since there were no real tests covering the
S3Hook I've added some
basic ones.

Closes #2771 from ashb/AIRFLOW-1797

(cherry picked from commit 28411b1e7eddb3338a329db3e52ee09de3676784)
Signed-off-by: Bolke de Bruin 


> Cannot write task logs to S3 with Python3
> -
>
> Key: AIRFLOW-1797
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1797
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Ash Berlin-Taylor
>
> {noformat}
> Traceback (most recent call last):
>   File 
> "/usr/local/lib/python3.5/dist-packages/airflow/utils/log/s3_task_handler.py",
>  line 161, in s3_write
> encrypt=configuration.getboolean('core', 'ENCRYPT_S3_LOGS'),
>   File "/usr/local/lib/python3.5/dist-packages/airflow/hooks/S3_hook.py", 
> line 253, in load_string
> client.upload_fileobj(filelike_buffer, bucket_name, key, 
> ExtraArgs=extra_args)
>   File "/usr/local/lib/python3.5/dist-packages/boto3/s3/inject.py", line 431, 
> in upload_fileobj
> return future.result()
>   File "/usr/local/lib/python3.5/dist-packages/s3transfer/futures.py", line 
> 73, in result
> return self._coordinator.result()
>   File "/usr/local/lib/python3.5/dist-packages/s3transfer/futures.py", line 
> 233, in result
> raise self._exception
>   File "/usr/local/lib/python3.5/dist-packages/s3transfer/tasks.py", line 
> 126, in __call__
> return self._execute_main(kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/s3transfer/tasks.py", line 
> 150, in _execute_main
> return_value = self._main(**kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/s3transfer/upload.py", line 
> 679, in _main
> client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/client.py", line 312, 
> in _api_call
> return self._make_api_call(operation_name, kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/client.py", line 586, 
> in _make_api_call
> request_signer=self._request_signer, context=request_context)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/hooks.py", line 242, 
> in emit_until_response
> responses = self._emit(event_name, kwargs, stop_on_response=True)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/hooks.py", line 210, 
> in _emit
> response = handler(**kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/handlers.py", line 
> 201, in conditionally_calculate_md5
> calculate_md5(params, **kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/handlers.py", line 
> 179, in calculate_md5
> binary_md5 = _calculate_md5_from_file(body)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/handlers.py", line 
> 193, in _calculate_md5_from_file
> md5.update(chunk)
> TypeError: Unicode-objects must be encoded before hashing
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

incubator-airflow git commit: [AIRFLOW-1797] S3Hook.load_string didn't work on Python3

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/v1-9-test 8dea4a64e -> d592f891e


[AIRFLOW-1797] S3Hook.load_string didn't work on Python3

With the switch to Boto3 we now need the content
to be bytes, not a
string. On Python2 there is no difference, but for
Python3 this matters.

And since there were no real tests covering the
S3Hook I've added some
basic ones.

Closes #2771 from ashb/AIRFLOW-1797

(cherry picked from commit 28411b1e7eddb3338a329db3e52ee09de3676784)
Signed-off-by: Bolke de Bruin 


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/d592f891
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/d592f891
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/d592f891

Branch: refs/heads/v1-9-test
Commit: d592f891e58650472c8fba89bace3cce54a7972b
Parents: 8dea4a6
Author: Ash Berlin-Taylor 
Authored: Thu Nov 9 20:47:08 2017 +0100
Committer: Bolke de Bruin 
Committed: Thu Nov 9 20:47:34 2017 +0100

--
 airflow/hooks/S3_hook.py|  7 ++--
 tests/core.py   | 20 ---
 tests/hooks/test_s3_hook.py | 74 
 3 files changed, 78 insertions(+), 23 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/d592f891/airflow/hooks/S3_hook.py
--
diff --git a/airflow/hooks/S3_hook.py b/airflow/hooks/S3_hook.py
index f8052ca..b16566f 100644
--- a/airflow/hooks/S3_hook.py
+++ b/airflow/hooks/S3_hook.py
@@ -15,7 +15,7 @@
 from airflow.exceptions import AirflowException
 from airflow.contrib.hooks.aws_hook import AwsHook
 
-from six import StringIO
+from six import BytesIO
 from urllib.parse import urlparse
 import re
 import fnmatch
@@ -217,7 +217,8 @@ class S3Hook(AwsHook):
 key, 
 bucket_name=None,
 replace=False,
-encrypt=False):
+encrypt=False,
+encoding='utf-8'):
 """
 Loads a string to S3
 
@@ -247,7 +248,7 @@ class S3Hook(AwsHook):
 if encrypt:
 extra_args['ServerSideEncryption'] = "AES256"
 
-filelike_buffer = StringIO(string_data)
+filelike_buffer = BytesIO(string_data.encode(encoding))
 
 client = self.get_conn()
 client.upload_fileobj(filelike_buffer, bucket_name, key, 
ExtraArgs=extra_args)

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/d592f891/tests/core.py
--
diff --git a/tests/core.py b/tests/core.py
index c08d63d..0bd0c87 100644
--- a/tests/core.py
+++ b/tests/core.py
@@ -2386,26 +2386,6 @@ class HttpHookTest(unittest.TestCase):
 self.assertEqual(hook.base_url, 'https://localhost')
 
 
-try:
-from airflow.hooks.S3_hook import S3Hook
-except ImportError:
-S3Hook = None
-
-
-@unittest.skipIf(S3Hook is None,
- "Skipping test because S3Hook is not installed")
-class S3HookTest(unittest.TestCase):
-def setUp(self):
-configuration.load_test_config()
-self.s3_test_url = "s3://test/this/is/not/a-real-key.txt"
-
-def test_parse_s3_url(self):
-parsed = S3Hook.parse_s3_url(self.s3_test_url)
-self.assertEqual(parsed,
- ("test", "this/is/not/a-real-key.txt"),
- "Incorrect parsing of the s3 url")
-
-
 send_email_test = mock.Mock()
 
 

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/d592f891/tests/hooks/test_s3_hook.py
--
diff --git a/tests/hooks/test_s3_hook.py b/tests/hooks/test_s3_hook.py
new file mode 100644
index 000..48c9fde
--- /dev/null
+++ b/tests/hooks/test_s3_hook.py
@@ -0,0 +1,74 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import unittest
+
+from airflow import configuration
+
+try:
+from airflow.hooks.S3_hook import S3Hook
+except ImportError:
+S3Hook = None
+
+
+try:
+import boto3
+from moto import mock_s3
+except ImportError:
+mock_s3 = None
+
+
+@unittest.skipIf(S3Hook is None,
+   

[jira] [Work started] (AIRFLOW-1798) Include celery ssl configs in default template

[Daniel Huang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on AIRFLOW-1798 started by Daniel Huang.
-
> Include celery ssl configs in default template
> --
>
> Key: AIRFLOW-1798
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1798
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: configuration
>Reporter: Daniel Huang
>Assignee: Daniel Huang
>Priority: Trivial
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-646) setup.py install fails

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246082#comment-16246082
 ] 

ASF subversion and git services commented on AIRFLOW-646:
-

Commit 9425d359b0e140095b9a0afca0238e8d1d7590de in incubator-airflow's branch 
refs/heads/master from [~wrp]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=9425d35 ]

[AIRFLOW-646] Add docutils to setup_requires

python-daemon declares its docutils dependency in a setup_requires
clause, and 'python setup.py install' fails since it misses
that dependency.

Closes #2765 from wrp/docutils


> setup.py install fails
> --
>
> Key: AIRFLOW-646
> URL: https://issues.apache.org/jira/browse/AIRFLOW-646
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: Airflow 1.7.1
> Environment: OS X
> Ubuntu 16.04
> Python 2.7
> Python 3.5
>Reporter: Nick Allen
>Assignee: William Pursell
>
> Running `python setup.py install` or listing airflow as a dependency in 
> another setup.py install_requires section results in the following error:
> {quote}
> Running python-daemon-2.1.2/setup.py -q bdist_egg --dist-dir 
> /tmp/easy_install-fibs2crb/python-daemon-2.1.2/egg-dist-tmp-hbg5xgc1
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.5/site-packages/setuptools/sandbox.py", line 
> 154, in save_modules
> yield saved
>   File "/usr/local/lib/python3.5/site-packages/setuptools/sandbox.py", line 
> 195, in setup_context
> yield
>   File "/usr/local/lib/python3.5/site-packages/setuptools/sandbox.py", line 
> 243, in run_setup
> DirectorySandbox(setup_dir).run(runner)
>   File "/usr/local/lib/python3.5/site-packages/setuptools/sandbox.py", line 
> 273, in run
> return func()
>   File "/usr/local/lib/python3.5/site-packages/setuptools/sandbox.py", line 
> 242, in runner
> _execfile(setup_script, ns)
>   File "/usr/local/lib/python3.5/site-packages/setuptools/sandbox.py", line 
> 46, in _execfile
> exec(code, globals, locals)
>   File "/tmp/easy_install-fibs2crb/python-daemon-2.1.2/setup.py", line 43, in 
> 
> sys.exit(errno)
> AttributeError: module 'version' has no attribute 'ChangelogAwareDistribution'
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
>   File "setup.py", line 281, in 
> do_setup()
>   File "setup.py", line 275, in do_setup
> 'extra_clean': CleanCommand,
>   File "/usr/local/lib/python3.5/distutils/core.py", line 148, in setup
> dist.run_commands()
>   File "/usr/local/lib/python3.5/distutils/dist.py", line 955, in run_commands
> self.run_command(cmd)
>   File "/usr/local/lib/python3.5/distutils/dist.py", line 974, in run_command
> cmd_obj.run()
>   File 
> "/usr/local/lib/python3.5/site-packages/setuptools/command/install.py", line 
> 67, in run
> self.do_egg_install()
>   File 
> "/usr/local/lib/python3.5/site-packages/setuptools/command/install.py", line 
> 117, in do_egg_install
> cmd.run()
>   File 
> "/usr/local/lib/python3.5/site-packages/setuptools/command/easy_install.py", 
> line 391, in run
> self.easy_install(spec, not self.no_deps)
>   File 
> "/usr/local/lib/python3.5/site-packages/setuptools/command/easy_install.py", 
> line 621, in easy_install
> return self.install_item(None, spec, tmpdir, deps, True)
>   File 
> "/usr/local/lib/python3.5/site-packages/setuptools/command/easy_install.py", 
> line 672, in install_item
> self.process_distribution(spec, dist, deps)
>   File 
> "/usr/local/lib/python3.5/site-packages/setuptools/command/easy_install.py", 
> line 717, in process_distribution
> [requirement], self.local_index, self.easy_install
>   File "/usr/local/lib/python3.5/site-packages/pkg_resources/__init__.py", 
> line 826, in resolve
> dist = best[req.key] = env.best_match(req, ws, installer)
>   File "/usr/local/lib/python3.5/site-packages/pkg_resources/__init__.py", 
> line 1092, in best_match
> return self.obtain(req, installer)
>   File "/usr/local/lib/python3.5/site-packages/pkg_resources/__init__.py", 
> line 1104, in obtain
> return installer(requirement)
>   File 
> "/usr/local/lib/python3.5/site-packages/setuptools/command/easy_install.py", 
> line 640, in easy_install
> return self.install_item(spec, dist.location, tmpdir, deps)
>   File 
> "/usr/local/lib/python3.5/site-packages/setuptools/command/easy_install.py", 
> line 670, in install_item
> dists = self.install_eggs(spec, download, tmpdir)
>   File 
> "/usr/local/lib/python3.5/site-packages/setuptools/command/easy_install.py", 
> line 850, in install_eggs
> return self.build_and_install(setup_script, setup_base)
>   File 
> "/usr/local/lib/python3.5/site-packages/setuptools/command/easy_install.py", 
> line 1078, in build_and_install
> 

incubator-airflow git commit: [AIRFLOW-646] Add docutils to setup_requires

[criccomini]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master 3eb2dd86b -> 9425d359b


[AIRFLOW-646] Add docutils to setup_requires

python-daemon declares its docutils dependency in a setup_requires
clause, and 'python setup.py install' fails since it misses
that dependency.

Closes #2765 from wrp/docutils


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/9425d359
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/9425d359
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/9425d359

Branch: refs/heads/master
Commit: 9425d359b0e140095b9a0afca0238e8d1d7590de
Parents: 3eb2dd8
Author: William Pursell 
Authored: Thu Nov 9 09:15:02 2017 -0800
Committer: Chris Riccomini 
Committed: Thu Nov 9 09:15:12 2017 -0800

--
 setup.py | 3 +++
 1 file changed, 3 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/9425d359/setup.py
--
diff --git a/setup.py b/setup.py
index 4fac29d..3cce836 100644
--- a/setup.py
+++ b/setup.py
@@ -241,6 +241,9 @@ def do_setup():
 'thrift>=0.9.2',
 'zope.deprecation>=4.0, <5.0',
 ],
+setup_requires=[
+'docutils>=0.14, <1.0',
+],
 extras_require={
 'all': devel_all,
 'all_dbs': all_dbs,

[jira] [Commented] (AIRFLOW-1756) S3 Task Handler Cannot Read Logs With New S3Hook

[Ash Berlin-Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16245860#comment-16245860
 ] 

Ash Berlin-Taylor commented on AIRFLOW-1756:


PR to fix underlying issue and expand tests 
https://github.com/apache/incubator-airflow/pull/2773

> S3 Task Handler Cannot Read Logs With New S3Hook
> 
>
> Key: AIRFLOW-1756
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1756
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Colin Son
>Priority: Critical
> Fix For: 1.9.0
>
>
> With the changes to the S3Hook, it seems like it cannot read the S3 task logs.
> In the `s3_read` in the S3TaskHandler.py:
> {code}
> s3_key = self.hook.get_key(remote_log_location)
> if s3_key:
> return s3_key.get_contents_as_string().decode()
> {code}
> Since the s3_key object is now a dict, you cannot call 
> `get_contents_as_string()` on a dict object. You have to use the S3Hook's 
> `read_key()` method to read the contents of the task logs now. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (AIRFLOW-1797) Cannot write task logs to S3 with Python3

[Ash Berlin-Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor updated AIRFLOW-1797:
---
External issue URL: https://github.com/apache/incubator-airflow/pull/2771

> Cannot write task logs to S3 with Python3
> -
>
> Key: AIRFLOW-1797
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1797
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Ash Berlin-Taylor
>
> {noformat}
> Traceback (most recent call last):
>   File 
> "/usr/local/lib/python3.5/dist-packages/airflow/utils/log/s3_task_handler.py",
>  line 161, in s3_write
> encrypt=configuration.getboolean('core', 'ENCRYPT_S3_LOGS'),
>   File "/usr/local/lib/python3.5/dist-packages/airflow/hooks/S3_hook.py", 
> line 253, in load_string
> client.upload_fileobj(filelike_buffer, bucket_name, key, 
> ExtraArgs=extra_args)
>   File "/usr/local/lib/python3.5/dist-packages/boto3/s3/inject.py", line 431, 
> in upload_fileobj
> return future.result()
>   File "/usr/local/lib/python3.5/dist-packages/s3transfer/futures.py", line 
> 73, in result
> return self._coordinator.result()
>   File "/usr/local/lib/python3.5/dist-packages/s3transfer/futures.py", line 
> 233, in result
> raise self._exception
>   File "/usr/local/lib/python3.5/dist-packages/s3transfer/tasks.py", line 
> 126, in __call__
> return self._execute_main(kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/s3transfer/tasks.py", line 
> 150, in _execute_main
> return_value = self._main(**kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/s3transfer/upload.py", line 
> 679, in _main
> client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/client.py", line 312, 
> in _api_call
> return self._make_api_call(operation_name, kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/client.py", line 586, 
> in _make_api_call
> request_signer=self._request_signer, context=request_context)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/hooks.py", line 242, 
> in emit_until_response
> responses = self._emit(event_name, kwargs, stop_on_response=True)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/hooks.py", line 210, 
> in _emit
> response = handler(**kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/handlers.py", line 
> 201, in conditionally_calculate_md5
> calculate_md5(params, **kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/handlers.py", line 
> 179, in calculate_md5
> binary_md5 = _calculate_md5_from_file(body)
>   File "/usr/local/lib/python3.5/dist-packages/botocore/handlers.py", line 
> 193, in _calculate_md5_from_file
> md5.update(chunk)
> TypeError: Unicode-objects must be encoded before hashing
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (AIRFLOW-1790) AWS Batch Operator Suppport

[Hugo Prudente (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hugo Prudente resolved AIRFLOW-1790.

Resolution: Implemented

Pull request submitted: https://github.com/apache/incubator-airflow/pull/2762

Waiting for review.

> AWS Batch Operator Suppport
> ---
>
> Key: AIRFLOW-1790
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1790
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: operators
>Affects Versions: 1.9.0
>Reporter: Hugo Prudente
>Assignee: Hugo Prudente
>
> Add support to AWS Batch operator



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Work started] (AIRFLOW-1790) AWS Batch Operator Suppport

[Hugo Prudente (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on AIRFLOW-1790 started by Hugo Prudente.
--
> AWS Batch Operator Suppport
> ---
>
> Key: AIRFLOW-1790
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1790
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: operators
>Affects Versions: 1.9.0
>Reporter: Hugo Prudente
>Assignee: Hugo Prudente
>
> Add support to AWS Batch operator



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (AIRFLOW-1797) Cannot write task logs to S3 with Python3

[Ash Berlin-Taylor (JIRA)]

Ash Berlin-Taylor created AIRFLOW-1797:
--

 Summary: Cannot write task logs to S3 with Python3
 Key: AIRFLOW-1797
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1797
 Project: Apache Airflow
  Issue Type: Bug
Reporter: Ash Berlin-Taylor


{noformat}
Traceback (most recent call last):
  File 
"/usr/local/lib/python3.5/dist-packages/airflow/utils/log/s3_task_handler.py", 
line 161, in s3_write
encrypt=configuration.getboolean('core', 'ENCRYPT_S3_LOGS'),
  File "/usr/local/lib/python3.5/dist-packages/airflow/hooks/S3_hook.py", line 
253, in load_string
client.upload_fileobj(filelike_buffer, bucket_name, key, 
ExtraArgs=extra_args)
  File "/usr/local/lib/python3.5/dist-packages/boto3/s3/inject.py", line 431, 
in upload_fileobj
return future.result()
  File "/usr/local/lib/python3.5/dist-packages/s3transfer/futures.py", line 73, 
in result
return self._coordinator.result()
  File "/usr/local/lib/python3.5/dist-packages/s3transfer/futures.py", line 
233, in result
raise self._exception
  File "/usr/local/lib/python3.5/dist-packages/s3transfer/tasks.py", line 126, 
in __call__
return self._execute_main(kwargs)
  File "/usr/local/lib/python3.5/dist-packages/s3transfer/tasks.py", line 150, 
in _execute_main
return_value = self._main(**kwargs)
  File "/usr/local/lib/python3.5/dist-packages/s3transfer/upload.py", line 679, 
in _main
client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "/usr/local/lib/python3.5/dist-packages/botocore/client.py", line 312, 
in _api_call
return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.5/dist-packages/botocore/client.py", line 586, 
in _make_api_call
request_signer=self._request_signer, context=request_context)
  File "/usr/local/lib/python3.5/dist-packages/botocore/hooks.py", line 242, in 
emit_until_response
responses = self._emit(event_name, kwargs, stop_on_response=True)
  File "/usr/local/lib/python3.5/dist-packages/botocore/hooks.py", line 210, in 
_emit
response = handler(**kwargs)
  File "/usr/local/lib/python3.5/dist-packages/botocore/handlers.py", line 201, 
in conditionally_calculate_md5
calculate_md5(params, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/botocore/handlers.py", line 179, 
in calculate_md5
binary_md5 = _calculate_md5_from_file(body)
  File "/usr/local/lib/python3.5/dist-packages/botocore/handlers.py", line 193, 
in _calculate_md5_from_file
md5.update(chunk)
TypeError: Unicode-objects must be encoded before hashing
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (AIRFLOW-1796) Badly configured logging config can thwo exception in WWW logs view

[Ash Berlin-Taylor (JIRA)]

Ash Berlin-Taylor created AIRFLOW-1796:
--

 Summary: Badly configured  logging config can thwo exception in 
WWW logs view
 Key: AIRFLOW-1796
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1796
 Project: Apache Airflow
  Issue Type: Bug
Reporter: Ash Berlin-Taylor


It is possible to specify a custom logging config that changes the 
{{file.task}} handler to {{s3.task}} but forget to update the 
{{core.task_log_reader}} config section.

This should be validated at start time, and mentioned in the comments of the 
default logging config that the config setting needs updating to.

This ends up as the following stack trace/error if you don't set it properly 
when trying to view task logs:

{noformat}
File "/usr/local/lib/python3.5/dist-packages/airflow/www/views.py", line 712, 
in log
   logs = handler.read(ti)
AttributeError: 'NoneType' object has no attribute 'read'
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (AIRFLOW-1795) S3Hook no longer accepts s3_conn_id breaking build in ops/sensors and back-compat

[Ash Berlin-Taylor (JIRA)]

Ash Berlin-Taylor created AIRFLOW-1795:
--

 Summary: S3Hook no longer accepts s3_conn_id breaking build in 
ops/sensors and back-compat 
 Key: AIRFLOW-1795
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1795
 Project: Apache Airflow
  Issue Type: Bug
Affects Versions: 1.9.0
Reporter: Ash Berlin-Taylor
 Fix For: 1.9.0


Found whilst testing Airflow 1.9.0rc1

Previously the S3Hook accepted a parameter of {{s3_conn_id}}. As part of 
AIRFLOW-1520 we moved S3Hook to have a superclass of AWSHook, which accepts a 
{{aws_conn_id}} parameter instead.

This break back-compat generally, and more specifically it breaks the built in 
S3KeySensor which does this:

{code}
def poke(self, context):
from airflow.hooks.S3_hook import S3Hook
hook = S3Hook(s3_conn_id=self.s3_conn_id)
{code}

There are a few other instances of s3_conn_id in the code base that will also 
probably need updating/tweaking.

My first though was to add a shim mapping s3_conn_id to aws_conn_id in the 
S3Hook with a deprecation warning but the surface area with places where this 
is exposed is larger. I could add such a deprecation warning to all of these. 
Anyone have thoughts as to best way?

- Rename all instances with deprecation warnings.
- S3Hook accepts {{s3_conn_id}} and passes down to {{aws_conn_id}} in 
superclass.
- Update existing references in code base to {{aws_conn_id}}, and not in 
updating about need to update in user code. (This is my least preferred option.)

{noformat}
airflow/operators/redshift_to_s3_operator.py
33::param s3_conn_id: reference to a specific S3 connection
34::type s3_conn_id: string
51:s3_conn_id='s3_default',
62:self.s3_conn_id = s3_conn_id
69:self.s3 = S3Hook(s3_conn_id=self.s3_conn_id)

airflow/operators/s3_file_transform_operator.py
40::param source_s3_conn_id: source s3 connection
41::type source_s3_conn_id: str
44::param dest_s3_conn_id: destination s3 connection
45::type dest_s3_conn_id: str
62:source_s3_conn_id='s3_default',
63:dest_s3_conn_id='s3_default',
68:self.source_s3_conn_id = source_s3_conn_id
70:self.dest_s3_conn_id = dest_s3_conn_id
75:source_s3 = S3Hook(s3_conn_id=self.source_s3_conn_id)
76:dest_s3 = S3Hook(s3_conn_id=self.dest_s3_conn_id)

airflow/operators/s3_to_hive_operator.py
74::param s3_conn_id: source s3 connection
75::type s3_conn_id: str
102:s3_conn_id='s3_default',
119:self.s3_conn_id = s3_conn_id
130:self.s3 = S3Hook(s3_conn_id=self.s3_conn_id)

airflow/operators/sensors.py
504::param s3_conn_id: a reference to the s3 connection
505::type s3_conn_id: str
514:s3_conn_id='s3_default',
531:self.s3_conn_id = s3_conn_id
535:hook = S3Hook(s3_conn_id=self.s3_conn_id)
568:s3_conn_id='s3_default',
576:self.s3_conn_id = s3_conn_id
582:hook = S3Hook(s3_conn_id=self.s3_conn_id)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)