[
https://issues.apache.org/jira/browse/AIRFLOW-3279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Velthuis updated AIRFLOW-3279:
-----------------------------------
Description:
The documentation of how to install logging to a Google Cloud bucket is unclear.
I am now following the tutorial on the airflow page:
[https://airflow.apache.org/howto/write-logs.html]
Here I find it unclear what part of the 'logger' I have to adjust in the
`{{airflow/config_templates/airflow_local_settings.py}}`.
The adjustment states:
# Update the airflow.task and airflow.tas_runner blocks to be 'gcs.task'
instead of 'file.task'. 'loggers':
Unknown macro: \{ 'airflow.task'}
However what I find in the template is:
|'loggers': \{\| \|'airflow.processor': { \\\\\\\\| \\\\\\\\|'handlers':
['processor'], \\\\\\\\| \\\\\\\\|'level': LOG_LEVEL, \\\\\\\\|
\\\\\\\\|'propagate': False, \\\\\\\\| \\\\\\\\|},|
|'airflow.task': {
\|
\|'handlers': ['task'],
\|
\|'level': LOG_LEVEL,
\|
\|'propagate': False,
\|
\|},|
|'flask_appbuilder': {
\|
\|'handler': ['console'],
\|
\|'level': FAB_LOG_LEVEL,
\|
\|'propagate': True,
\|
\|}|
},
Since for me it is very important to do it right at the first time I hope some
clarity can be provided in what has to be adjusted in the logger. Is it only
the 'airflow.task' or more?
Furthermore, at step 6 it is a little unclear what remote_log_conn_id means. I
would propose to add a little more information to make this more clear.
The current error I am facing is:
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 16, in <module>
from airflow import configuration
File "/usr/local/lib/python2.7/site-packages/airflow/__init__.py", line 31, in
<module>
from airflow import settings
File "/usr/local/lib/python2.7/site-packages/airflow/settings.py", line 198,
in <module>
configure_logging()
File "/usr/local/lib/python2.7/site-packages/airflow/logging_config.py", line
71, in configure_logging
dictConfig(logging_config)
File "/usr/local/lib/python2.7/logging/config.py", line 794, in dictConfig
dictConfigClass(config).configure()
File "/usr/local/lib/python2.7/logging/config.py", line 568, in configure
handler = self.configure_handler(handlers[name])
File "/usr/local/lib/python2.7/logging/config.py", line 733, in
configure_handler
result = factory(**kwargs)
File
"/usr/local/lib/python2.7/site-packages/airflow/utils/log/gcs_task_handler.py",
line 30, in __init__
super(GCSTaskHandler, self).__init__(base_log_folder, filename_template)
File
"/usr/local/lib/python2.7/site-packages/airflow/utils/log/file_task_handler.py",
line 46, in __init__
self.filename_jinja_template = Template(self.filename_template)
File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 926,
in __new__
return env.from_string(source, template_class=cls)
File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 862,
in from_string
return cls.from_code(self, self.compile(source), globals, None)
File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 565,
in compile
self.handle_exception(exc_info, source_hint=source_hint)
File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 754,
in handle_exception
reraise(exc_type, exc_value, tb)
File "<unknown>", line 1, in template
jinja2.exceptions.TemplateSyntaxError: expected token ':', got '}'
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/usr/local/lib/python2.7/logging/__init__.py", line 1676, in shutdown
h.close()
File
"/usr/local/lib/python2.7/site-packages/airflow/utils/log/gcs_task_handler.py",
line 73, in close
if self.closed:
AttributeError: 'GCSTaskHandler' object has no attribute 'closed'
Error in sys.exitfunc:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/usr/local/lib/python2.7/logging/__init__.py", line 1676, in shutdown
h.close()
File
"/usr/local/lib/python2.7/site-packages/airflow/utils/log/gcs_task_handler.py",
line 73, in close
if self.closed:
AttributeError: 'GCSTaskHandler' object has no attribute 'closed'
was:
The documentation of how to install logging to a Google Cloud bucket is unclear.
I am now following the tutorial on the airflow page:
[https://airflow.apache.org/howto/write-logs.html]
Here I find it unclear what part of the 'logger' I have to adjust in the
`{{airflow/config_templates/airflow_local_settings.py}}`.
The adjustment states:
# Update the airflow.task and airflow.tas_runner blocks to be 'gcs.task'
instead of 'file.task'. 'loggers':
Unknown macro: \{ 'airflow.task'}
However what I find in the template is:
|'loggers': \{\| \|'airflow.processor': { \\\\\\\| \\\\\\\|'handlers':
['processor'], \\\\\\\| \\\\\\\|'level': LOG_LEVEL, \\\\\\\|
\\\\\\\|'propagate': False, \\\\\\\| \\\\\\\|},|
|'airflow.task': \{ \\| \\|'handlers': ['task'], \\| \\|'level': LOG_LEVEL, \\|
\\|'propagate': False, \\| \\|},|
|'flask_appbuilder': \{ \\| \\|'handler': ['console'], \\| \\|'level':
FAB_LOG_LEVEL, \\| \\|'propagate': True, \\| \\|}|
},
Since for me it is very important to do it right at the first time I hope some
clarity can be provided in what has to be adjusted in the logger. Is it only
the 'airflow.task' or more?
Furthermore, at step 6 it is a little unclear what remote_log_conn_id means. I
would propose to add a little more information to make this more clear.
> Documentation for Google Logging unclear
> ----------------------------------------
>
> Key: AIRFLOW-3279
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3279
> Project: Apache Airflow
> Issue Type: Bug
> Components: configuration, Documentation, logging
> Reporter: Paul Velthuis
> Assignee: Fokko Driesprong
> Priority: Major
>
> The documentation of how to install logging to a Google Cloud bucket is
> unclear.
> I am now following the tutorial on the airflow page:
> [https://airflow.apache.org/howto/write-logs.html]
> Here I find it unclear what part of the 'logger' I have to adjust in the
> `{{airflow/config_templates/airflow_local_settings.py}}`.
>
> The adjustment states:
>
> # Update the airflow.task and airflow.tas_runner blocks to be 'gcs.task'
> instead of 'file.task'. 'loggers':
> Unknown macro: \{ 'airflow.task'}
>
> However what I find in the template is:
> |'loggers': \{\| \|'airflow.processor': { \\\\\\\\| \\\\\\\\|'handlers':
> ['processor'], \\\\\\\\| \\\\\\\\|'level': LOG_LEVEL, \\\\\\\\|
> \\\\\\\\|'propagate': False, \\\\\\\\| \\\\\\\\|},|
> |'airflow.task': {
> \|
> \|'handlers': ['task'],
> \|
> \|'level': LOG_LEVEL,
> \|
> \|'propagate': False,
> \|
> \|},|
> |'flask_appbuilder': {
> \|
> \|'handler': ['console'],
> \|
> \|'level': FAB_LOG_LEVEL,
> \|
> \|'propagate': True,
> \|
> \|}|
> },
>
> Since for me it is very important to do it right at the first time I hope
> some clarity can be provided in what has to be adjusted in the logger. Is it
> only the 'airflow.task' or more?
> Furthermore, at step 6 it is a little unclear what remote_log_conn_id means.
> I would propose to add a little more information to make this more clear.
>
> The current error I am facing is:
> Traceback (most recent call last):
> File "/usr/local/bin/airflow", line 16, in <module>
> from airflow import configuration
> File "/usr/local/lib/python2.7/site-packages/airflow/__init__.py", line 31,
> in <module>
> from airflow import settings
> File "/usr/local/lib/python2.7/site-packages/airflow/settings.py", line 198,
> in <module>
> configure_logging()
> File "/usr/local/lib/python2.7/site-packages/airflow/logging_config.py",
> line 71, in configure_logging
> dictConfig(logging_config)
> File "/usr/local/lib/python2.7/logging/config.py", line 794, in dictConfig
> dictConfigClass(config).configure()
> File "/usr/local/lib/python2.7/logging/config.py", line 568, in configure
> handler = self.configure_handler(handlers[name])
> File "/usr/local/lib/python2.7/logging/config.py", line 733, in
> configure_handler
> result = factory(**kwargs)
> File
> "/usr/local/lib/python2.7/site-packages/airflow/utils/log/gcs_task_handler.py",
> line 30, in __init__
> super(GCSTaskHandler, self).__init__(base_log_folder, filename_template)
> File
> "/usr/local/lib/python2.7/site-packages/airflow/utils/log/file_task_handler.py",
> line 46, in __init__
> self.filename_jinja_template = Template(self.filename_template)
> File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line
> 926, in __new__
> return env.from_string(source, template_class=cls)
> File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line
> 862, in from_string
> return cls.from_code(self, self.compile(source), globals, None)
> File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line
> 565, in compile
> self.handle_exception(exc_info, source_hint=source_hint)
> File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line
> 754, in handle_exception
> reraise(exc_type, exc_value, tb)
> File "<unknown>", line 1, in template
> jinja2.exceptions.TemplateSyntaxError: expected token ':', got '}'
> Error in atexit._run_exitfuncs:
> Traceback (most recent call last):
> File "/usr/local/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
> func(*targs, **kargs)
> File "/usr/local/lib/python2.7/logging/__init__.py", line 1676, in shutdown
> h.close()
> File
> "/usr/local/lib/python2.7/site-packages/airflow/utils/log/gcs_task_handler.py",
> line 73, in close
> if self.closed:
> AttributeError: 'GCSTaskHandler' object has no attribute 'closed'
> Error in sys.exitfunc:
> Traceback (most recent call last):
> File "/usr/local/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
> func(*targs, **kargs)
> File "/usr/local/lib/python2.7/logging/__init__.py", line 1676, in shutdown
> h.close()
> File
> "/usr/local/lib/python2.7/site-packages/airflow/utils/log/gcs_task_handler.py",
> line 73, in close
> if self.closed:
> AttributeError: 'GCSTaskHandler' object has no attribute 'closed'
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
