Re: [openstack-dev] Logging format: let's discuss a bit about default format, format configuration and so on
Excerpts from Juan Antonio Osorio's message of 2017-11-04 00:11:47 +0200: > > On 3 Nov 2017 19:59, "Doug Hellmann" wrote: > > > > Excerpts from Cédric Jeanneret's message of 2017-11-01 14:54:34 +0100: > > > Dear Stackers, > > > > > > While working on my locally deployed Openstack (Pike using TripleO), I > > > was a bit struggling with the logging part. Currently, all logs are > > > pushed to per-service files, in the standard format "one line per > > > entry", plain flat text. > > > > > > It's nice, but if one is wanting to push and index those logs in an ELK, > > > the current, default format isn't really good. > > > > > > After some discussions about oslo.log, it appears it provides a nice > > > JSONFormatter handler¹ one might want to use for each (python) service > > > using oslo central library. > > > > > > A JSON format is really cool, as it's easy to parse for machines, and it > > > can be on a multi-line without any bit issue - this is especially > > > important for stack traces, as their output is multi-line without real > > > way to have a common delimiter so that we can re-format it and feed it > > > to any log parser (logstash, fluentd, …). > > > > > > After some more talks, olso.log will not provide a unified interface in > > > order to output all received logs as JSON - this makes sens, as that > > > would mean "rewrite almost all the python logging management > > > interface"², and that's pretty useless, since (all?) services have their > > > own "logging.conf" file. > > > > > > That said… to the main purpose of that mail: > > > > > > - Default format for logs > > > A first question would be "are we all OK with the default output format" > > > - I'm pretty sure "humans" are happy with that, as it's really > > > convenient to read and grep. But on a "standard" Openstack deploy, I'm > > > pretty sure one does not have only one controller, one ceph node and one > > > compute. Hence comes the log centralization, and with that, the log > > > indexation and treatments. > > > > > > For that, one might argue "I'm using plain files on my logger, and > > > grep-ing -r in them". That's a way to do things, and for that, plain, > > > flat logs are great. > > > > > > But… I'm pretty sure I'm not the only one wanting to use some kind of > > > ELK cluster for that kind of purpose. So the right question is: what > > > about switching the default log format to JSON? On my part, I don't see > > > "cons", only "pros", but my judgment is of course biased, as I'm "alone > > > in my corner". But what about you, Community? > > > > > > - Provide a way to configure the output format/handler > > > While poking around in the puppet modules code, I didn't find any way to > > > set the output handler for the logs. For example, in puppet-nova³ we can > > > set a lot of things, but not the useful handler for the output. > > > > > > It would be really cool to get, for each puppet module, the capability > > > to set the handler so that one can just push some stuff in hiera, and > > > Voilà, we have JSON logs. > > > > > > Doing so would allow people to chose between the default (current) > > > output, and something more "computable". > > > > Using the JSON formatter currently requires setting a logging > > configuration file using the standard library configuration format > > and fully specifying things like log levels, handlers, and output > > destination. Would it make sense to add an option in oslo.log to > > give deployers an easier way to enable the JSON formatter? > > > > This would actually be very useful. Great! Let me know if you would like to help figuring out where to do that in the library code. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Logging format: let's discuss a bit about default format, format configuration and so on
+1 From: Juan Antonio Osorio [jaosor...@gmail.com] Sent: Friday, November 03, 2017 3:11 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] Logging format: let's discuss a bit about default format, format configuration and so on On 3 Nov 2017 19:59, "Doug Hellmann" mailto:d...@doughellmann.com>> wrote: Excerpts from Cédric Jeanneret's message of 2017-11-01 14:54:34 +0100: > Dear Stackers, > > While working on my locally deployed Openstack (Pike using TripleO), I > was a bit struggling with the logging part. Currently, all logs are > pushed to per-service files, in the standard format "one line per > entry", plain flat text. > > It's nice, but if one is wanting to push and index those logs in an ELK, > the current, default format isn't really good. > > After some discussions about oslo.log, it appears it provides a nice > JSONFormatter handler¹ one might want to use for each (python) service > using oslo central library. > > A JSON format is really cool, as it's easy to parse for machines, and it > can be on a multi-line without any bit issue - this is especially > important for stack traces, as their output is multi-line without real > way to have a common delimiter so that we can re-format it and feed it > to any log parser (logstash, fluentd, …). > > After some more talks, olso.log will not provide a unified interface in > order to output all received logs as JSON - this makes sens, as that > would mean "rewrite almost all the python logging management > interface"², and that's pretty useless, since (all?) services have their > own "logging.conf" file. > > That said… to the main purpose of that mail: > > - Default format for logs > A first question would be "are we all OK with the default output format" > - I'm pretty sure "humans" are happy with that, as it's really > convenient to read and grep. But on a "standard" Openstack deploy, I'm > pretty sure one does not have only one controller, one ceph node and one > compute. Hence comes the log centralization, and with that, the log > indexation and treatments. > > For that, one might argue "I'm using plain files on my logger, and > grep-ing -r in them". That's a way to do things, and for that, plain, > flat logs are great. > > But… I'm pretty sure I'm not the only one wanting to use some kind of > ELK cluster for that kind of purpose. So the right question is: what > about switching the default log format to JSON? On my part, I don't see > "cons", only "pros", but my judgment is of course biased, as I'm "alone > in my corner". But what about you, Community? > > - Provide a way to configure the output format/handler > While poking around in the puppet modules code, I didn't find any way to > set the output handler for the logs. For example, in puppet-nova³ we can > set a lot of things, but not the useful handler for the output. > > It would be really cool to get, for each puppet module, the capability > to set the handler so that one can just push some stuff in hiera, and > Voilà, we have JSON logs. > > Doing so would allow people to chose between the default (current) > output, and something more "computable". Using the JSON formatter currently requires setting a logging configuration file using the standard library configuration format and fully specifying things like log levels, handlers, and output destination. Would it make sense to add an option in oslo.log to give deployers an easier way to enable the JSON formatter? This would actually be very useful. Doug > > Of course, either proposal will require a nice code change in all puppet > modules (add a new parameter for the foo::logging class, and use that > new param in the configuration file, and so on), but at least people > will be able to actually chose. > > So, before opening an issue on each launchpad project (that would be… > long), I'd rather open the discussion in here and, eventually, come to > some nice, acceptable and accepted solution that would make the > Openstack Community happy :). > > Any thoughts? > > Thank you for your attention, feedback and wonderful support for that > monster project :). > > Cheers, > > C. > > > ¹ > https://github.com/openstack/oslo.log/blob/master/oslo_log/formatters.py#L166-L235 > ² > http://eavesdrop.openstack.org/irclogs/%23openstack-oslo/%23openstack-oslo.2017-11-01.log.html#t2017-11-01T13:23:14 > ³ https://github.com/openstack/puppet-nova/blob/master/manifests/logging.pp > _
Re: [openstack-dev] Logging format: let's discuss a bit about default format, format configuration and so on
On 3 Nov 2017 19:59, "Doug Hellmann" wrote: Excerpts from Cédric Jeanneret's message of 2017-11-01 14:54:34 +0100: > Dear Stackers, > > While working on my locally deployed Openstack (Pike using TripleO), I > was a bit struggling with the logging part. Currently, all logs are > pushed to per-service files, in the standard format "one line per > entry", plain flat text. > > It's nice, but if one is wanting to push and index those logs in an ELK, > the current, default format isn't really good. > > After some discussions about oslo.log, it appears it provides a nice > JSONFormatter handler¹ one might want to use for each (python) service > using oslo central library. > > A JSON format is really cool, as it's easy to parse for machines, and it > can be on a multi-line without any bit issue - this is especially > important for stack traces, as their output is multi-line without real > way to have a common delimiter so that we can re-format it and feed it > to any log parser (logstash, fluentd, …). > > After some more talks, olso.log will not provide a unified interface in > order to output all received logs as JSON - this makes sens, as that > would mean "rewrite almost all the python logging management > interface"², and that's pretty useless, since (all?) services have their > own "logging.conf" file. > > That said… to the main purpose of that mail: > > - Default format for logs > A first question would be "are we all OK with the default output format" > - I'm pretty sure "humans" are happy with that, as it's really > convenient to read and grep. But on a "standard" Openstack deploy, I'm > pretty sure one does not have only one controller, one ceph node and one > compute. Hence comes the log centralization, and with that, the log > indexation and treatments. > > For that, one might argue "I'm using plain files on my logger, and > grep-ing -r in them". That's a way to do things, and for that, plain, > flat logs are great. > > But… I'm pretty sure I'm not the only one wanting to use some kind of > ELK cluster for that kind of purpose. So the right question is: what > about switching the default log format to JSON? On my part, I don't see > "cons", only "pros", but my judgment is of course biased, as I'm "alone > in my corner". But what about you, Community? > > - Provide a way to configure the output format/handler > While poking around in the puppet modules code, I didn't find any way to > set the output handler for the logs. For example, in puppet-nova³ we can > set a lot of things, but not the useful handler for the output. > > It would be really cool to get, for each puppet module, the capability > to set the handler so that one can just push some stuff in hiera, and > Voilà, we have JSON logs. > > Doing so would allow people to chose between the default (current) > output, and something more "computable". Using the JSON formatter currently requires setting a logging configuration file using the standard library configuration format and fully specifying things like log levels, handlers, and output destination. Would it make sense to add an option in oslo.log to give deployers an easier way to enable the JSON formatter? This would actually be very useful. Doug > > Of course, either proposal will require a nice code change in all puppet > modules (add a new parameter for the foo::logging class, and use that > new param in the configuration file, and so on), but at least people > will be able to actually chose. > > So, before opening an issue on each launchpad project (that would be… > long), I'd rather open the discussion in here and, eventually, come to > some nice, acceptable and accepted solution that would make the > Openstack Community happy :). > > Any thoughts? > > Thank you for your attention, feedback and wonderful support for that > monster project :). > > Cheers, > > C. > > > ¹ > https://github.com/openstack/oslo.log/blob/master/oslo_log/ formatters.py#L166-L235 > ² > http://eavesdrop.openstack.org/irclogs/%23openstack-oslo/ %23openstack-oslo.2017-11-01.log.html#t2017-11-01T13:23:14 > ³ https://github.com/openstack/puppet-nova/blob/master/ manifests/logging.pp > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Logging format: let's discuss a bit about default format, format configuration and so on
Excerpts from Cédric Jeanneret's message of 2017-11-01 14:54:34 +0100: > Dear Stackers, > > While working on my locally deployed Openstack (Pike using TripleO), I > was a bit struggling with the logging part. Currently, all logs are > pushed to per-service files, in the standard format "one line per > entry", plain flat text. > > It's nice, but if one is wanting to push and index those logs in an ELK, > the current, default format isn't really good. > > After some discussions about oslo.log, it appears it provides a nice > JSONFormatter handler¹ one might want to use for each (python) service > using oslo central library. > > A JSON format is really cool, as it's easy to parse for machines, and it > can be on a multi-line without any bit issue - this is especially > important for stack traces, as their output is multi-line without real > way to have a common delimiter so that we can re-format it and feed it > to any log parser (logstash, fluentd, …). > > After some more talks, olso.log will not provide a unified interface in > order to output all received logs as JSON - this makes sens, as that > would mean "rewrite almost all the python logging management > interface"², and that's pretty useless, since (all?) services have their > own "logging.conf" file. > > That said… to the main purpose of that mail: > > - Default format for logs > A first question would be "are we all OK with the default output format" > - I'm pretty sure "humans" are happy with that, as it's really > convenient to read and grep. But on a "standard" Openstack deploy, I'm > pretty sure one does not have only one controller, one ceph node and one > compute. Hence comes the log centralization, and with that, the log > indexation and treatments. > > For that, one might argue "I'm using plain files on my logger, and > grep-ing -r in them". That's a way to do things, and for that, plain, > flat logs are great. > > But… I'm pretty sure I'm not the only one wanting to use some kind of > ELK cluster for that kind of purpose. So the right question is: what > about switching the default log format to JSON? On my part, I don't see > "cons", only "pros", but my judgment is of course biased, as I'm "alone > in my corner". But what about you, Community? > > - Provide a way to configure the output format/handler > While poking around in the puppet modules code, I didn't find any way to > set the output handler for the logs. For example, in puppet-nova³ we can > set a lot of things, but not the useful handler for the output. > > It would be really cool to get, for each puppet module, the capability > to set the handler so that one can just push some stuff in hiera, and > Voilà, we have JSON logs. > > Doing so would allow people to chose between the default (current) > output, and something more "computable". Using the JSON formatter currently requires setting a logging configuration file using the standard library configuration format and fully specifying things like log levels, handlers, and output destination. Would it make sense to add an option in oslo.log to give deployers an easier way to enable the JSON formatter? Doug > > Of course, either proposal will require a nice code change in all puppet > modules (add a new parameter for the foo::logging class, and use that > new param in the configuration file, and so on), but at least people > will be able to actually chose. > > So, before opening an issue on each launchpad project (that would be… > long), I'd rather open the discussion in here and, eventually, come to > some nice, acceptable and accepted solution that would make the > Openstack Community happy :). > > Any thoughts? > > Thank you for your attention, feedback and wonderful support for that > monster project :). > > Cheers, > > C. > > > ¹ > https://github.com/openstack/oslo.log/blob/master/oslo_log/formatters.py#L166-L235 > ² > http://eavesdrop.openstack.org/irclogs/%23openstack-oslo/%23openstack-oslo.2017-11-01.log.html#t2017-11-01T13:23:14 > ³ https://github.com/openstack/puppet-nova/blob/master/manifests/logging.pp > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Logging format: let's discuss a bit about default format, format configuration and so on
On Thu, Nov 2, 2017 at 11:15 AM, Bogdan Dobrelya wrote: > Hi. Few comments inline. > > > Dear Stackers, >> While working on my locally deployed Openstack (Pike using TripleO), I >> was a bit struggling with the logging part. Currently, all logs are >> pushed to per-service files, in the standard format "one line per >> entry", plain flat text. >> >> It's nice, but if one is wanting to push and index those logs in an ELK, >> the current, default format isn't really good. >> >> After some discussions about oslo.log, it appears it provides a nice >> JSONFormatter handler¹ one might want to use for each (python) service >> using oslo central library. >> >> A JSON format is really cool, as it's easy to parse for machines, and it >> can be on a multi-line without any bit issue - this is especially >> important for stack traces, as their output is multi-line without real >> way to have a common delimiter so that we can re-format it and feed it >> to any log parser (logstash, fluentd, …). >> >> After some more talks, olso.log will not provide a unified interface in >> order to output all received logs as JSON - this makes sens, as that >> would mean "rewrite almost all the python logging management >> interface"², and that's pretty useless, since (all?) services have their >> own "logging.conf" file. >> >> That said… to the main purpose of that mail: >> >> - Default format for logs >> A first question would be "are we all OK with the default output format" >> - I'm pretty sure "humans" are happy with that, as it's really >> convenient to read and grep. But on a "standard" Openstack deploy, I'm >> pretty sure one does not have only one controller, one ceph node and one >> compute. Hence comes the log centralization, and with that, the log >> indexation and treatments. >> >> For that, one might argue "I'm using plain files on my logger, and >> grep-ing -r in them". That's a way to do things, and for that, plain, >> flat logs are great. >> >> But… I'm pretty sure I'm not the only one wanting to use some kind of >> ELK cluster for that kind of purpose. So the right question is: what >> about switching the default log format to JSON? On my part, I don't see >> "cons", only "pros", but my judgment is of course biased, as I'm "alone >> in my corner". But what about you, Community? >> >> - Provide a way to configure the output format/handler >> While poking around in the puppet modules code, I didn't find any way to >> set the output handler for the logs. For example, in puppet-nova³ we can >> set a lot of things, but not the useful handler for the output. >> > Right, I was also looking into this and am yet to find how it's done. Any idea how? >> It would be really cool to get, for each puppet module, the capability >> to set the handler so that one can just push some stuff in hiera, and >> Voilà, we have JSON logs. >> >> Doing so would allow people to chose between the default (current) >> output, and something more "computable". >> > > This is being implemented (for Q) in the scope of this blueprint [0]. > By default, containers will keep the current logging behaviour, which is > writing logs into bind-mounted host-path directories, and for those > services that can syslog only, sending the logs to the host journald via > syslog (bind-mounted /dev/log). And if the stdout/stderr feature enabled, > side-car containers will make sure everything is captured in journald and > tagged properly, with multiline entries as well. And journald can dump > messages as JSON as well. I hope that works! Please send you comments to > the aforementioned bp's spec. > > [0] https://blueprints.launchpad.net/tripleo/+spec/logging-stdout-rsyslog Correct, as Bogdan mentioned, that will hopefully come as part of the work of that blueprint. The plan so far is to keep the logs as they currently are and then put them in JSON format in an rsyslog forwarder (the implementation is still to come). However, one thing that would be great would be to have oslo.log already provide JSON output, that way we would need way less processing. And most importantly, it would make StackTrace processing easier. Right now it's a pain to reconstruct python exceptions as they end up being forwarded as several log lines, and having it as one line, or as JSON output already would be great. Any idea if this is possible already via oslo.log? > > > >> Of course, either proposal will require a nice code change in all puppet >> modules (add a new parameter for the foo::logging class, and use that >> new param in the configuration file, and so on), but at least people >> will be able to actually chose. >> >> So, before opening an issue on each launchpad project (that would be… >> long), I'd rather open the discussion in here and, eventually, come to >> some nice, acceptable and accepted solution that would make the >> Openstack Community happy :). >> >> Any thoughts? >> >> Thank you for your attention, feedback and wonderful support for that >> monster project :). >> >
Re: [openstack-dev] Logging format: let's discuss a bit about default format, format configuration and so on
Hi. Few comments inline. Dear Stackers, While working on my locally deployed Openstack (Pike using TripleO), I was a bit struggling with the logging part. Currently, all logs are pushed to per-service files, in the standard format "one line per entry", plain flat text. It's nice, but if one is wanting to push and index those logs in an ELK, the current, default format isn't really good. After some discussions about oslo.log, it appears it provides a nice JSONFormatter handler¹ one might want to use for each (python) service using oslo central library. A JSON format is really cool, as it's easy to parse for machines, and it can be on a multi-line without any bit issue - this is especially important for stack traces, as their output is multi-line without real way to have a common delimiter so that we can re-format it and feed it to any log parser (logstash, fluentd, …). After some more talks, olso.log will not provide a unified interface in order to output all received logs as JSON - this makes sens, as that would mean "rewrite almost all the python logging management interface"², and that's pretty useless, since (all?) services have their own "logging.conf" file. That said… to the main purpose of that mail: - Default format for logs A first question would be "are we all OK with the default output format" - I'm pretty sure "humans" are happy with that, as it's really convenient to read and grep. But on a "standard" Openstack deploy, I'm pretty sure one does not have only one controller, one ceph node and one compute. Hence comes the log centralization, and with that, the log indexation and treatments. For that, one might argue "I'm using plain files on my logger, and grep-ing -r in them". That's a way to do things, and for that, plain, flat logs are great. But… I'm pretty sure I'm not the only one wanting to use some kind of ELK cluster for that kind of purpose. So the right question is: what about switching the default log format to JSON? On my part, I don't see "cons", only "pros", but my judgment is of course biased, as I'm "alone in my corner". But what about you, Community? - Provide a way to configure the output format/handler While poking around in the puppet modules code, I didn't find any way to set the output handler for the logs. For example, in puppet-nova³ we can set a lot of things, but not the useful handler for the output. It would be really cool to get, for each puppet module, the capability to set the handler so that one can just push some stuff in hiera, and Voilà, we have JSON logs. Doing so would allow people to chose between the default (current) output, and something more "computable". This is being implemented (for Q) in the scope of this blueprint [0]. By default, containers will keep the current logging behaviour, which is writing logs into bind-mounted host-path directories, and for those services that can syslog only, sending the logs to the host journald via syslog (bind-mounted /dev/log). And if the stdout/stderr feature enabled, side-car containers will make sure everything is captured in journald and tagged properly, with multiline entries as well. And journald can dump messages as JSON as well. I hope that works! Please send you comments to the aforementioned bp's spec. [0] https://blueprints.launchpad.net/tripleo/+spec/logging-stdout-rsyslog Of course, either proposal will require a nice code change in all puppet modules (add a new parameter for the foo::logging class, and use that new param in the configuration file, and so on), but at least people will be able to actually chose. So, before opening an issue on each launchpad project (that would be… long), I'd rather open the discussion in here and, eventually, come to some nice, acceptable and accepted solution that would make the Openstack Community happy :). Any thoughts? Thank you for your attention, feedback and wonderful support for that monster project :). Cheers, C. ¹ https://github.com/openstack/oslo.log/blob/master/oslo_log/formatters.py#L166-L235 ² http://eavesdrop.openstack.org/irclogs/%23openstack-oslo/%23openstack-oslo.2017-11-01.log.html#t2017-11-01T13:23:14 ³ https://github.com/openstack/puppet-nova/blob/master/manifests/logging.pp -- Cédric Jeanneret Senior Linux System Administrator Infrastructure Solutions Camptocamp SA PSE-A / EPFL, 1015 Lausanne Phone: +41 21 619 10 32 Office: +41 21 619 10 02 Email: cedric.jeanne...@camptocamp.com -- Best regards, Bogdan Dobrelya, Irc #bogdando __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
