Hi Andy, Thanks for reaching out. We are debugging the new logging, and input from the community is highly appreciated.
If you are using Python 2, you'll need to put an empty __init__.py in all the directory, so ~/airflow/plugins/__init__.py, this needs to be empty. Could you share your airflow_local_settings.py? If there are any GCS credentials, please remove them. Please check that you've renamed the DEFAULT_LOGGING_CONFIG variable to LOGGING_CONFIG, this might not be evident from the updating.md. Cheers, Fokko 2017-10-17 12:00 GMT+02:00 Andrew Maguire <[email protected]>: > Hi, > > I've updated to 1.9 but am having trouble setting the logging_config_class > class path in the airflow.cfg file. > > Currently i have below files in {AIRFLOW_HOME}/plugins > > [image: image.png] > Where airflow_logging_settings.py is just a copy of this file > <https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/airflow_local_settings.py> > but with the GCS stuff uncommented and a line added for GCS_LOG_FOLDER to > be pulled from airflow.cfg just like BASE_LOG_FOLDER > > then in my {AIRFLOW_HOME}/airflow.cfg file i have this following line to > set up the log stuff: > > # The folder where airflow should store its log files > # This path must be absolute > base_log_folder = {AIRFLOW_HOME}/logs > > gcs_log_folder = gs://pmc-airflow/logs > > # Airflow can store logs remotely in AWS S3 or Google Cloud Storage. Users > # must supply an Airflow connection id that provides access to the storage > # location. > remote_log_conn_id = my_gcp_connection > encrypt_s3_logs = False > > # Logging level > logging_level = INFO > > # Logging class > # Specify the class that will specify the logging configuration > # This class has to be on the python classpath > logging_config_class = plugins.airflow_logging_settings.LOGGING_CONFIG > > # Log format > log_format = [%%(asctime)s] {{%%(filename)s:%%(lineno)d}} %%(levelname)s - > %%(message)s > simple_log_format = %%(asctime)s %%(levelname)s - %%(message)s > > > However if i run "airflow list_dags" i now get: > > andrew_maguire@airflow-server:~/airflow$ airflow list_dags > Traceback (most recent call last): > File "/usr/local/bin/airflow", line 16, in <module> > from airflow import configuration > File "/usr/local/lib/python2.7/dist-packages/airflow/__init__.py", line 31, > in <module> > from airflow import settings > File "/usr/local/lib/python2.7/dist-packages/airflow/settings.py", line > 148, in <module> > configure_logging() > File "/usr/local/lib/python2.7/dist-packages/airflow/logging_config.py", > line 47, in configure_logging > 'Unable to load custom logging from {}'.format(logging_class_path) > ImportError: Unable to load custom logging from > plugins.airflow_logging_settings.LOGGING_CONFIG > > If i go back into my airflow.cfg and comment out the line: > > logging_config_class = plugins.airflow_logging_settings.LOGGING_CONFIG > > Things work again but i'm only doing local logging. > > So am sure i'm doing something wrong here in how i'm setting that line in the > airflow.cfg file. > > So what i did was. > > 1. create a folder {AIRFLOW_HOME}/plugins - i did this as updating.md > <https://github.com/apache/incubator-airflow/blob/master/UPDATING.md> > mentioned that "The logging configuration file that contains the > configuration needs te on the the PYTHONPATH, for example in ~/airflow/dags > or ~/airflow/plugins. These directories are loaded by default". > > 2. copy this file > <https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/airflow_local_settings.py> > into {AIRFLOW_HOME}/plugins with the GCS changes i mentioned. > > 3. create an __init__.py file in {AIRFLOW_HOME}/plugins - do i need to put > anyting in partiucular in here? > > 4. update {AIRFLOW_HOME}/airflow.cfg as above. > > Can someone help me figure out where i went wrong? > > I'm hesitant to change or add anytihng to the pythonpath as an not 100% sure > what i'm doing. So was hoping to just drop the logging config file somewhere > it would automatically be picked up. And i'm also not really sure about > python packages and class paths etc so kinda feeling my way through it but > not confident. > > Cheers > > Andy > > >
