Hi Feng,
Thanks for your help! Got it, will try to push on the python based logging
config.
I'm trying to set-up the GCS logging on airflow v1-9-stable and my
logging_config.py seems to be causing a python import error, caused by
'from airflow import configuration'
"Initialize database...
Unable to load the config, contains a configuration error.
Traceback (most recent call last):
File "/usr/lib/python3.5/logging/config.py", line 384, in resolve
self.importer(used)
ImportError: No module named
'airflow.utils.log.logging_mixin.RedirectStdHandler';
'airflow.utils.log.logging_mixin' is not a package
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3.5/logging/config.py", line 558, in configure
handler = self.configure_handler(handlers[name])
File "/usr/lib/python3.5/logging/config.py", line 708, in
configure_handler
klass = self.resolve(cname)
File "/usr/lib/python3.5/logging/config.py", line 391, in resolve
raise v
File "/usr/lib/python3.5/logging/config.py", line 384, in resolve
self.importer(used)
ValueError: Cannot resolve
'airflow.utils.log.logging_mixin.RedirectStdHandler': No module named
'airflow.utils.log.logging_mixin.RedirectStdHandler';
'airflow.utils.log.logging_mixin' is not a package
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 16, in <module>
from airflow import configuration
File "/usr/local/lib/python3.5/dist-packages/airflow/__init__.py", line
31, in <module>
from airflow import settings
File "/usr/local/lib/python3.5/dist-packages/airflow/settings.py", line
148, in <module>
configure_logging()
File "/usr/local/lib/python3.5/dist-packages/airflow/logging_config.py",
line 75, in configure_logging
raise e
File "/usr/local/lib/python3.5/dist-packages/airflow/logging_config.py",
line 70, in configure_logging
dictConfig(logging_config)
File "/usr/lib/python3.5/logging/config.py", line 795, in dictConfig
dictConfigClass(config).configure()
File "/usr/lib/python3.5/logging/config.py", line 566, in configure
'%r: %s' % (name, e))
ValueError: Unable to configure handler 'console': Cannot resolve
'airflow.utils.log.logging_mixin.RedirectStdHandler': No module named
'airflow.utils.log.logging_mixin.RedirectStdHandler';
'airflow.utils.log.logging_mixin' is not a package
HTTP/1.1 200 OK
Unable to load the config, contains a configuration error.
Traceback (most recent call last):
File "/usr/lib/python3.5/logging/config.py", line 384, in resolve
self.importer(used)
ImportError: No module named
'airflow.utils.log.logging_mixin.RedirectStdHandler';
'airflow.utils.log.logging_mixin' is not a package
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3.5/logging/config.py", line 558, in configure
handler = self.configure_handler(handlers[name])
File "/usr/lib/python3.5/logging/config.py", line 708, in
configure_handler
klass = self.resolve(cname)
File "/usr/lib/python3.5/logging/config.py", line 391, in resolve
raise v
File "/usr/lib/python3.5/logging/config.py", line 384, in resolve
self.importer(used)
ValueError: Cannot resolve
'airflow.utils.log.logging_mixin.RedirectStdHandler': No module named
'airflow.utils.log.logging_mixin.RedirectStdHandler';
'airflow.utils.log.logging_mixin' is not a package
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 16, in <module>
from airflow import configuration
File "/usr/local/lib/python3.5/dist-packages/airflow/__init__.py", line
31, in <module>
from airflow import settings
File "/usr/local/lib/python3.5/dist-packages/airflow/settings.py", line
148, in <module>
configure_logging()
File "/usr/local/lib/python3.5/dist-packages/airflow/logging_config.py",
line 75, in configure_logging
raise e
File "/usr/local/lib/python3.5/dist-packages/airflow/logging_config.py",
line 70, in configure_logging
dictConfig(logging_config)
File "/usr/lib/python3.5/logging/config.py", line 795, in dictConfig
dictConfigClass(config).configure()
File "/usr/lib/python3.5/logging/config.py", line 566, in configure
'%r: %s' % (name, e))
ValueError: Unable to configure handler 'console': Cannot resolve
'airflow.utils.log.logging_mixin.RedirectStdHandler': No module named
'airflow.utils.log.logging_mixin.RedirectStdHandler';
'airflow.utils.log.logging_mixin' is not a package"
Have you encountered this before?
On Mon, Dec 18, 2017 at 8:53 PM, Feng Lu <fen...@google.com.invalid> wrote:
> Hi Kevin,
>
> Kindly see my reply inline:
>
> On Mon, Dec 18, 2017 at 3:28 PM, Kevin Lam <ke...@fathomhealth.co> wrote:
>
> > Hi,
> >
> > I'm trying to get airflow to use GCS for logging purposes and had a few
> > questions.
> >
> > We're currently using Airflow 1.9rc2, running in a Kubernetes Airflow
> > deployment (similar to https://github.com/mumoshu/kube-airflow)
> >
> > 1/ Seems like the logging code has been going through some changes in the
> > recent versions. What's the correct way to set up GCS for logging? Is it
> by
> > just specifying remote_base_log_folder and remote_log_conn_id in
> > airflow.cfg? Or by following this guide:
> > http://airflow.readthedocs.io/en/latest/integration.html#gcp, using the
> > python based logging config? Is there an Airflow version that we should
> use
> > to be most stable?
> >
> The python based logging config is the right place to make changes, in our
> test setup, we override the airflow_local_settings.py similarly to the link
> you pasted.
> You may also want to config: [core]task_log_reader = gcs.task
>
>
> >
> > 2/ Is there a way to encode the connection for GCS in a file so that one
> > doesn't have to open the webserver and create it from the admin panel?
> It'd
> > be nice if the GCS connection would be automatically created.
> >
> Unfortunately GCS connection ties to some GCP project and is impossible to
> pre-populate.
> Airflow1.9 should fix the gcp connection type issue (
> https://github.com/apache/incubator-airflow/commit/
> 2f107d8a30910fd025774004d5c4c95407ed55c5),
> so you can use airflow connections CLI directly.
>
>
> >
> > Thanks in advance for your help!
> >
>
