Thank you for the response Maxime. We will consider the points you made when making the changes and may come back with further thoughts as we work on it.
I can't say I am too familiar with Kibana, Logstash and Elasticsearch but I am assuming Airflow would still produce logs and you would be using Logstash to process the logs? Thanks, Luke Maycock OLIVER WYMAN luke.mayc...@affiliate.oliverwyman.com<mailto:luke.mayc...@affiliate.oliverwyman.com> www.oliverwyman.com<http://www.oliverwyman.com/> ________________________________ From: Maxime Beauchemin <maximebeauche...@gmail.com> Sent: 14 October 2016 17:02 To: firstname.lastname@example.org Subject: Re: Airflow Logging Another consideration is for configuration as code in `settings.py` in-place or in conjunction with `airflow.cfg` to allow more dynamic configuration, like passing your own custom log handler[s]. Many folks at the meetup (including Airbnb) spoke about their intent to ship the logging to Kibana and/or other logging backend. Side note about `airflow.cfg` vs `settings.py`: it would be great to move completely away from the cfgs for Airflow 2.0. It turns out that Python's ConfigParser may be one of the worst module in the standard library and the cfg not-so-standard standard isn't great. We have many instances where we allow for people to pass objects or function as configuration parameters anyways, so let's keep the whole configuration in one place! Max On Fri, Oct 14, 2016 at 7:43 AM, Maycock, Luke < luke.mayc...@affiliate.oliverwyman.com> wrote: > Carrying on from the below. We believe it would be useful to have a number > of additional configuration options for logs as below. Do you think there > are any important options we have missed or do you have any other feedback? > If so, please let us know. > > > Global > > CLEAR_INHERITED_LOGGING_SETTINGS (Boolean) > Clears any handlers, inherited from the root logger, from the Airflow > logger object and its children. > > LOG_TO_DEBUG_FILE (Boolean) > > Turn on/off logging of all events to a file. > > LOG_TO_ERROR_FILE (Boolean) > Turn on/off the logging of all errors to a file. > > LOG_TO_CONSOLE (Boolean) > Turn on/off the logging of all events to the console. > > Per log file > The Python logging library, being used for logging in the Airflow code, > has the ability to rotate log files based on time: > > > • TimedRotatingFileHandler - (https://docs.python.org/2.6/ > library/logging.html#timedrotatingfilehandler) - rotates based on the > product of when and interval (see below). > > The TimedRotatingFileHandler takes the following arguments: > > * Filename – the filename > * When – used to specify the type of interval > * Interval – defines the interval > * BackupCount - If backupCount is nonzero, at most backupCount files > will be kept, and if more would be created when rollover occurs, the oldest > one is deleted. The deletion logic uses the interval to determine which > files to delete, so changing the interval may leave old files lying around. > * Encoding - if encoding is not ‘None’, it is used to open the file > with that encoding. > * Delay - if delay is true, then file opening is deferred until the > first call to emit() – I don’t think this is suitable to expose as an > Airflow configuration option. > * UTC - if the UTC argument is true, times in UTC will be used; > otherwise local time is used. > I believe all of the above, except ‘delay’, should be exposed as Airflow > configuration. > > > Cheers, > Luke Maycock > OLIVER WYMAN > luke.mayc...@affiliate.oliverwyman.com<mailto:luke. > mayc...@affiliate.oliverwyman.com> > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > ________________________________ > From: Maycock, Luke > Sent: 13 October 2016 14:52 > To: email@example.com > Subject: Airflow Logging > > > Hi All, > > We (owlabs - fork: https://github.com/owlabs/incubator-airflow) have a > high level design for how to improve the logging throughout the Airflow > code to be more consistent, maintainable and extensible. We'd really > appreciate any feedback on the design. > > Design for Consolidating Logging Setup: > > * Create a Class in the utils\logging.py to instantiate and handle > setup of an "airflow" logger object. This allows us to easily find all > setup code for further modification and extension in the future. > * This class would be where any general logging configuration (from > Airflow.cfg) would be applied. > * Instantiate this class in this file, so that importing it allows easy > control of the logging setup from anywhere else in the application. > * Move all setup of the logging in general to this class with easy to > call control methods to turn forms of logging on and off (e.g. > console/debugFile/errorFile). > * Move any other helper functions related to logging that are still > required into the utils\logging.py so that we can easily find them for > modification and extension in the future. > * Ensure that all logging throughout the application is done via the > "airflow" logger object, or by a child of this logger object. This allows > us to: > * Leave all settings of the root logger object alone, so that we are > friendly to any parent or child processes that are not part of the > application, allowing them to safely manage their own overriding logging > setup. > * Modify the settings of logging throughout the entire application > via a single simple point of control. > > Cheers, > Luke Maycock > OLIVER WYMAN > luke.mayc...@affiliate.oliverwyman.com<mailto:luke. > mayc...@affiliate.oliverwyman.com> > www.oliverwyman.com<http://www.oliverwyman.com/> > > > ________________________________ > This e-mail and any attachments may be confidential or legally privileged. > If you received this message in error or are not the intended recipient, > you should destroy the e-mail message and any attachments or copies, and > you are prohibited from retaining, distributing, disclosing or using any > information contained herein. Please inform us of the erroneous delivery by > return e-mail. Thank you for your cooperation. > ________________________________ This e-mail and any attachments may be confidential or legally privileged. If you received this message in error or are not the intended recipient, you should destroy the e-mail message and any attachments or copies, and you are prohibited from retaining, distributing, disclosing or using any information contained herein. Please inform us of the erroneous delivery by return e-mail. Thank you for your cooperation.