We are using v1.9rc6 and it seems to work fine with directly editing the airflow/config_templates/airflow_local_settings.py
On Tue, Dec 19, 2017 at 2:44 PM, Kevin Lam <[email protected]> wrote: > Looks like it might be related to > https://github.com/apache/incubator-airflow/commit/ > 02ff8ae35dd16e6f23d29d7b24a5fb9c09d0b7a4? > Why isn't this fix on the v1-9 branches? Should I be using master instead? > > On Tue, Dec 19, 2017 at 5:37 PM, Kevin Lam <[email protected]> wrote: > > > Hi Feng, > > > > Thanks for your help! Got it, will try to push on the python based > logging > > config. > > > > I'm trying to set-up the GCS logging on airflow v1-9-stable and my > > logging_config.py seems to be causing a python import error, caused by > > 'from airflow import configuration' > > > > "Initialize database... > > Unable to load the config, contains a configuration error. > > Traceback (most recent call last): > > File "/usr/lib/python3.5/logging/config.py", line 384, in resolve > > self.importer(used) > > ImportError: No module named 'airflow.utils.log.logging_ > mixin.RedirectStdHandler'; > > 'airflow.utils.log.logging_mixin' is not a package > > > > The above exception was the direct cause of the following exception: > > > > Traceback (most recent call last): > > File "/usr/lib/python3.5/logging/config.py", line 558, in configure > > handler = self.configure_handler(handlers[name]) > > File "/usr/lib/python3.5/logging/config.py", line 708, in > > configure_handler > > klass = self.resolve(cname) > > File "/usr/lib/python3.5/logging/config.py", line 391, in resolve > > raise v > > File "/usr/lib/python3.5/logging/config.py", line 384, in resolve > > self.importer(used) > > ValueError: Cannot resolve 'airflow.utils.log.logging_ > mixin.RedirectStdHandler': > > No module named 'airflow.utils.log.logging_mixin.RedirectStdHandler'; > > 'airflow.utils.log.logging_mixin' is not a package > > > > During handling of the above exception, another exception occurred: > > > > Traceback (most recent call last): > > File "/usr/local/bin/airflow", line 16, in <module> > > from airflow import configuration > > File "/usr/local/lib/python3.5/dist-packages/airflow/__init__.py", > line > > 31, in <module> > > from airflow import settings > > File "/usr/local/lib/python3.5/dist-packages/airflow/settings.py", > line > > 148, in <module> > > configure_logging() > > File "/usr/local/lib/python3.5/dist-packages/airflow/logging_ > config.py", > > line 75, in configure_logging > > raise e > > File "/usr/local/lib/python3.5/dist-packages/airflow/logging_ > config.py", > > line 70, in configure_logging > > dictConfig(logging_config) > > File "/usr/lib/python3.5/logging/config.py", line 795, in dictConfig > > dictConfigClass(config).configure() > > File "/usr/lib/python3.5/logging/config.py", line 566, in configure > > '%r: %s' % (name, e)) > > ValueError: Unable to configure handler 'console': Cannot resolve > > 'airflow.utils.log.logging_mixin.RedirectStdHandler': No module named > > 'airflow.utils.log.logging_mixin.RedirectStdHandler'; > > 'airflow.utils.log.logging_mixin' is not a package > > HTTP/1.1 200 OK > > Unable to load the config, contains a configuration error. > > Traceback (most recent call last): > > File "/usr/lib/python3.5/logging/config.py", line 384, in resolve > > self.importer(used) > > ImportError: No module named 'airflow.utils.log.logging_ > mixin.RedirectStdHandler'; > > 'airflow.utils.log.logging_mixin' is not a package > > > > The above exception was the direct cause of the following exception: > > > > Traceback (most recent call last): > > File "/usr/lib/python3.5/logging/config.py", line 558, in configure > > handler = self.configure_handler(handlers[name]) > > File "/usr/lib/python3.5/logging/config.py", line 708, in > > configure_handler > > klass = self.resolve(cname) > > File "/usr/lib/python3.5/logging/config.py", line 391, in resolve > > raise v > > File "/usr/lib/python3.5/logging/config.py", line 384, in resolve > > self.importer(used) > > ValueError: Cannot resolve 'airflow.utils.log.logging_ > mixin.RedirectStdHandler': > > No module named 'airflow.utils.log.logging_mixin.RedirectStdHandler'; > > 'airflow.utils.log.logging_mixin' is not a package > > > > During handling of the above exception, another exception occurred: > > > > Traceback (most recent call last): > > File "/usr/local/bin/airflow", line 16, in <module> > > from airflow import configuration > > File "/usr/local/lib/python3.5/dist-packages/airflow/__init__.py", > line > > 31, in <module> > > from airflow import settings > > File "/usr/local/lib/python3.5/dist-packages/airflow/settings.py", > line > > 148, in <module> > > configure_logging() > > File "/usr/local/lib/python3.5/dist-packages/airflow/logging_ > config.py", > > line 75, in configure_logging > > raise e > > File "/usr/local/lib/python3.5/dist-packages/airflow/logging_ > config.py", > > line 70, in configure_logging > > dictConfig(logging_config) > > File "/usr/lib/python3.5/logging/config.py", line 795, in dictConfig > > dictConfigClass(config).configure() > > File "/usr/lib/python3.5/logging/config.py", line 566, in configure > > '%r: %s' % (name, e)) > > ValueError: Unable to configure handler 'console': Cannot resolve > > 'airflow.utils.log.logging_mixin.RedirectStdHandler': No module named > > 'airflow.utils.log.logging_mixin.RedirectStdHandler'; > > 'airflow.utils.log.logging_mixin' is not a package" > > > > Have you encountered this before? > > > > On Mon, Dec 18, 2017 at 8:53 PM, Feng Lu <[email protected]> > > wrote: > > > >> Hi Kevin, > >> > >> Kindly see my reply inline: > >> > >> On Mon, Dec 18, 2017 at 3:28 PM, Kevin Lam <[email protected]> > wrote: > >> > >> > Hi, > >> > > >> > I'm trying to get airflow to use GCS for logging purposes and had a > few > >> > questions. > >> > > >> > We're currently using Airflow 1.9rc2, running in a Kubernetes Airflow > >> > deployment (similar to https://github.com/mumoshu/kube-airflow) > >> > > >> > 1/ Seems like the logging code has been going through some changes in > >> the > >> > recent versions. What's the correct way to set up GCS for logging? Is > >> it by > >> > just specifying remote_base_log_folder and remote_log_conn_id in > >> > airflow.cfg? Or by following this guide: > >> > http://airflow.readthedocs.io/en/latest/integration.html#gcp, using > the > >> > python based logging config? Is there an Airflow version that we > should > >> use > >> > to be most stable? > >> > > >> The python based logging config is the right place to make changes, in > our > >> test setup, we override the airflow_local_settings.py similarly to the > >> link > >> you pasted. > >> You may also want to config: [core]task_log_reader = gcs.task > >> > >> > >> > > >> > 2/ Is there a way to encode the connection for GCS in a file so that > one > >> > doesn't have to open the webserver and create it from the admin panel? > >> It'd > >> > be nice if the GCS connection would be automatically created. > >> > > >> Unfortunately GCS connection ties to some GCP project and is impossible > to > >> pre-populate. > >> Airflow1.9 should fix the gcp connection type issue ( > >> https://github.com/apache/incubator-airflow/commit/2f107d8a3 > >> 0910fd025774004d5c4c95407ed55c5), > >> so you can use airflow connections CLI directly. > >> > >> > >> > > >> > Thanks in advance for your help! > >> > > >> > > > > >
