My requirement would indeed be that I would be able to add my own logging handler (I did the same in my Luigi days), I included a python log handler that logged to Google Cloud Logging.
But I also like the current logging to Cloud storage. So my ideal logging setup would be: All of Airflow (scheduler + webserver + worker) -> Log to custom log handler Output of DAG -> Log to customer log handler + Cloud *storage* for easy access in Airflow It's currently a nice-to-have. On Tue, Oct 18, 2016 at 6:13 PM Maycock, Luke < luke.mayc...@affiliate.oliverwyman.com> wrote: > Thank you for the response Maxime. We will consider the points you made > when making the changes and may come back with further thoughts as we work > on it. > > > I can't say I am too familiar with Kibana, Logstash and Elasticsearch but > I am assuming Airflow would still produce logs and you would be using > Logstash to process the logs? > > > Thanks, > Luke Maycock > OLIVER WYMAN > luke.mayc...@affiliate.oliverwyman.com<mailto: > luke.mayc...@affiliate.oliverwyman.com> > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > ________________________________ > From: Maxime Beauchemin <maximebeauche...@gmail.com> > Sent: 14 October 2016 17:02 > To: dev@airflow.incubator.apache.org > Subject: Re: Airflow Logging > > Another consideration is for configuration as code in `settings.py` > in-place or in conjunction with `airflow.cfg` to allow more dynamic > configuration, like passing your own custom log handler[s]. Many folks at > the meetup (including Airbnb) spoke about their intent to ship the logging > to Kibana and/or other logging backend. > > Side note about `airflow.cfg` vs `settings.py`: it would be great to move > completely away from the cfgs for Airflow 2.0. It turns out that Python's > ConfigParser may be one of the worst module in the standard library and the > cfg not-so-standard standard isn't great. We have many instances where we > allow for people to pass objects or function as configuration parameters > anyways, so let's keep the whole configuration in one place! > > Max > > On Fri, Oct 14, 2016 at 7:43 AM, Maycock, Luke < > luke.mayc...@affiliate.oliverwyman.com> wrote: > > > Carrying on from the below. We believe it would be useful to have a > number > > of additional configuration options for logs as below. Do you think there > > are any important options we have missed or do you have any other > feedback? > > If so, please let us know. > > > > > > Global > > > > CLEAR_INHERITED_LOGGING_SETTINGS (Boolean) > > Clears any handlers, inherited from the root logger, from the Airflow > > logger object and its children. > > > > LOG_TO_DEBUG_FILE (Boolean) > > > > Turn on/off logging of all events to a file. > > > > LOG_TO_ERROR_FILE (Boolean) > > Turn on/off the logging of all errors to a file. > > > > LOG_TO_CONSOLE (Boolean) > > Turn on/off the logging of all events to the console. > > > > Per log file > > The Python logging library, being used for logging in the Airflow code, > > has the ability to rotate log files based on time: > > > > > > • TimedRotatingFileHandler - (https://docs.python.org/2.6/ > > library/logging.html#timedrotatingfilehandler) - rotates based on the > > product of when and interval (see below). > > > > The TimedRotatingFileHandler takes the following arguments: > > > > * Filename – the filename > > * When – used to specify the type of interval > > * Interval – defines the interval > > * BackupCount - If backupCount is nonzero, at most backupCount files > > will be kept, and if more would be created when rollover occurs, the > oldest > > one is deleted. The deletion logic uses the interval to determine which > > files to delete, so changing the interval may leave old files lying > around. > > * Encoding - if encoding is not ‘None’, it is used to open the file > > with that encoding. > > * Delay - if delay is true, then file opening is deferred until the > > first call to emit() – I don’t think this is suitable to expose as an > > Airflow configuration option. > > * UTC - if the UTC argument is true, times in UTC will be used; > > otherwise local time is used. > > I believe all of the above, except ‘delay’, should be exposed as Airflow > > configuration. > > > > > > Cheers, > > Luke Maycock > > OLIVER WYMAN > > luke.mayc...@affiliate.oliverwyman.com<mailto:luke. > > mayc...@affiliate.oliverwyman.com> > > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > > > > > ________________________________ > > From: Maycock, Luke > > Sent: 13 October 2016 14:52 > > To: dev@airflow.incubator.apache.org > > Subject: Airflow Logging > > > > > > Hi All, > > > > We (owlabs - fork: https://github.com/owlabs/incubator-airflow) have a > > high level design for how to improve the logging throughout the Airflow > > code to be more consistent, maintainable and extensible. We'd really > > appreciate any feedback on the design. > > > > Design for Consolidating Logging Setup: > > > > * Create a Class in the utils\logging.py to instantiate and handle > > setup of an "airflow" logger object. This allows us to easily find all > > setup code for further modification and extension in the future. > > * This class would be where any general logging configuration (from > > Airflow.cfg) would be applied. > > * Instantiate this class in this file, so that importing it allows > easy > > control of the logging setup from anywhere else in the application. > > * Move all setup of the logging in general to this class with easy to > > call control methods to turn forms of logging on and off (e.g. > > console/debugFile/errorFile). > > * Move any other helper functions related to logging that are still > > required into the utils\logging.py so that we can easily find them for > > modification and extension in the future. > > * Ensure that all logging throughout the application is done via the > > "airflow" logger object, or by a child of this logger object. This allows > > us to: > > * Leave all settings of the root logger object alone, so that we > are > > friendly to any parent or child processes that are not part of the > > application, allowing them to safely manage their own overriding logging > > setup. > > * Modify the settings of logging throughout the entire application > > via a single simple point of control. > > > > Cheers, > > Luke Maycock > > OLIVER WYMAN > > luke.mayc...@affiliate.oliverwyman.com<mailto:luke. > > mayc...@affiliate.oliverwyman.com> > > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > > > ________________________________ > > This e-mail and any attachments may be confidential or legally > privileged. > > If you received this message in error or are not the intended recipient, > > you should destroy the e-mail message and any attachments or copies, and > > you are prohibited from retaining, distributing, disclosing or using any > > information contained herein. Please inform us of the erroneous delivery > by > > return e-mail. Thank you for your cooperation. > > > > ________________________________ > This e-mail and any attachments may be confidential or legally privileged. > If you received this message in error or are not the intended recipient, > you should destroy the e-mail message and any attachments or copies, and > you are prohibited from retaining, distributing, disclosing or using any > information contained herein. Please inform us of the erroneous delivery by > return e-mail. Thank you for your cooperation. >
