Re: [openstack-dev] [oslo][tripleo][puppet] Logging format: let's discuss a bit about default format, format configuration and so on
Excerpts from Juan Antonio Osorio's message of 2017-11-06 15:18:33 +0200: > Proposed a patch to be able to enable the JSON formatter via an oslo.log > configuration parameter: > > https://review.openstack.org/#/c/517882/ That's approved, thanks! We should make sure we have a release note before we tag a new release of the library, though. Could you do that in another patch? Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][tripleo][puppet] Logging format: let's discuss a bit about default format, format configuration and so on
Proposed a patch to be able to enable the JSON formatter via an oslo.log configuration parameter: https://review.openstack.org/#/c/517882/ On Mon, Nov 6, 2017 at 9:43 AM, Cédric Jeanneret < cedric.jeanne...@camptocamp.com> wrote: > On 11/06/2017 08:36 AM, Juan Antonio Osorio wrote: > > Giving this a bit more thought; I'm slightly more inclined on merely > adding > > the JSON formatter option to the standard oslo.log configuration options. > > Fine for me. > > > > > This is because we right now have the ability to pass oslo.cfg options > via > > the CLI, and we would loose that with the advanced logging configuration > > file. The aforementioned option is something that we're using for > > containerized openstack services. > > OK - I'll check on my own if I can get something nice with the > logging.conf file. But that won't be for "tomorrow", as I'm not > full-time on openstack (unfortunately :(). Both solutions have their > pros and cons in the end. > > > > > I'll look into adding the ability to turn that handler on/off from > oslo.log. > > Ping me if I can help :). And big thanks for that coming change! > > > > > > > > > On Mon, Nov 6, 2017 at 8:34 AM, Juan Antonio Osorio> > > wrote: > > > >> > >> > >> On Mon, Nov 6, 2017 at 8:04 AM, Cédric Jeanneret < > >> cedric.jeanne...@camptocamp.com> wrote: > >> > >>> On 11/04/2017 07:08 AM, Doug Hellmann wrote: > Excerpts from Juan Antonio Osorio's message of 2017-11-04 00:11:47 > >>> +0200: > >> On 3 Nov 2017 19:59, "Doug Hellmann" wrote: > >> > >> Excerpts from Cédric Jeanneret's message of 2017-11-01 14:54:34 > +0100: > >>> Dear Stackers, > >>> > >>> While working on my locally deployed Openstack (Pike using > TripleO), > >>> I > >>> was a bit struggling with the logging part. Currently, all logs are > >>> pushed to per-service files, in the standard format "one line per > >>> entry", plain flat text. > >>> > >>> It's nice, but if one is wanting to push and index those logs in an > >>> ELK, > >>> the current, default format isn't really good. > >>> > >>> After some discussions about oslo.log, it appears it provides a > nice > >>> JSONFormatter handler¹ one might want to use for each (python) > >>> service > >>> using oslo central library. > >>> > >>> A JSON format is really cool, as it's easy to parse for machines, > >>> and it > >>> can be on a multi-line without any bit issue - this is especially > >>> important for stack traces, as their output is multi-line without > >>> real > >>> way to have a common delimiter so that we can re-format it and feed > >>> it > >>> to any log parser (logstash, fluentd, …). > >>> > >>> After some more talks, olso.log will not provide a unified > interface > >>> in > >>> order to output all received logs as JSON - this makes sens, as > that > >>> would mean "rewrite almost all the python logging management > >>> interface"², and that's pretty useless, since (all?) services have > >>> their > >>> own "logging.conf" file. > >>> > >>> That said… to the main purpose of that mail: > >>> > >>> - Default format for logs > >>> A first question would be "are we all OK with the default output > >>> format" > >>> - I'm pretty sure "humans" are happy with that, as it's really > >>> convenient to read and grep. But on a "standard" Openstack deploy, > >>> I'm > >>> pretty sure one does not have only one controller, one ceph node > and > >>> one > >>> compute. Hence comes the log centralization, and with that, the log > >>> indexation and treatments. > >>> > >>> For that, one might argue "I'm using plain files on my logger, and > >>> grep-ing -r in them". That's a way to do things, and for that, > plain, > >>> flat logs are great. > >>> > >>> But… I'm pretty sure I'm not the only one wanting to use some kind > of > >>> ELK cluster for that kind of purpose. So the right question is: > what > >>> about switching the default log format to JSON? On my part, I don't > >>> see > >>> "cons", only "pros", but my judgment is of course biased, as I'm > >>> "alone > >>> in my corner". But what about you, Community? > >>> > >>> - Provide a way to configure the output format/handler > >>> While poking around in the puppet modules code, I didn't find any > >>> way to > >>> set the output handler for the logs. For example, in puppet-nova³ > we > >>> can > >>> set a lot of things, but not the useful handler for the output. > >>> > >>> It would be really cool to get, for each puppet module, the > >>> capability > >>> to set the handler so that one can just push some stuff in hiera, > and > >>> Voilà, we have JSON logs. > >>> > >>> Doing so would allow people to chose between the default (current) > >>> output, and something more "computable". > >> > >> Using
Re: [openstack-dev] [oslo][tripleo][puppet] Logging format: let's discuss a bit about default format, format configuration and so on
On 11/06/2017 08:36 AM, Juan Antonio Osorio wrote: > Giving this a bit more thought; I'm slightly more inclined on merely adding > the JSON formatter option to the standard oslo.log configuration options. Fine for me. > > This is because we right now have the ability to pass oslo.cfg options via > the CLI, and we would loose that with the advanced logging configuration > file. The aforementioned option is something that we're using for > containerized openstack services. OK - I'll check on my own if I can get something nice with the logging.conf file. But that won't be for "tomorrow", as I'm not full-time on openstack (unfortunately :(). Both solutions have their pros and cons in the end. > > I'll look into adding the ability to turn that handler on/off from oslo.log. Ping me if I can help :). And big thanks for that coming change! > > > > On Mon, Nov 6, 2017 at 8:34 AM, Juan Antonio Osorio> wrote: > >> >> >> On Mon, Nov 6, 2017 at 8:04 AM, Cédric Jeanneret < >> cedric.jeanne...@camptocamp.com> wrote: >> >>> On 11/04/2017 07:08 AM, Doug Hellmann wrote: Excerpts from Juan Antonio Osorio's message of 2017-11-04 00:11:47 >>> +0200: >> On 3 Nov 2017 19:59, "Doug Hellmann" wrote: >> >> Excerpts from Cédric Jeanneret's message of 2017-11-01 14:54:34 +0100: >>> Dear Stackers, >>> >>> While working on my locally deployed Openstack (Pike using TripleO), >>> I >>> was a bit struggling with the logging part. Currently, all logs are >>> pushed to per-service files, in the standard format "one line per >>> entry", plain flat text. >>> >>> It's nice, but if one is wanting to push and index those logs in an >>> ELK, >>> the current, default format isn't really good. >>> >>> After some discussions about oslo.log, it appears it provides a nice >>> JSONFormatter handler¹ one might want to use for each (python) >>> service >>> using oslo central library. >>> >>> A JSON format is really cool, as it's easy to parse for machines, >>> and it >>> can be on a multi-line without any bit issue - this is especially >>> important for stack traces, as their output is multi-line without >>> real >>> way to have a common delimiter so that we can re-format it and feed >>> it >>> to any log parser (logstash, fluentd, …). >>> >>> After some more talks, olso.log will not provide a unified interface >>> in >>> order to output all received logs as JSON - this makes sens, as that >>> would mean "rewrite almost all the python logging management >>> interface"², and that's pretty useless, since (all?) services have >>> their >>> own "logging.conf" file. >>> >>> That said… to the main purpose of that mail: >>> >>> - Default format for logs >>> A first question would be "are we all OK with the default output >>> format" >>> - I'm pretty sure "humans" are happy with that, as it's really >>> convenient to read and grep. But on a "standard" Openstack deploy, >>> I'm >>> pretty sure one does not have only one controller, one ceph node and >>> one >>> compute. Hence comes the log centralization, and with that, the log >>> indexation and treatments. >>> >>> For that, one might argue "I'm using plain files on my logger, and >>> grep-ing -r in them". That's a way to do things, and for that, plain, >>> flat logs are great. >>> >>> But… I'm pretty sure I'm not the only one wanting to use some kind of >>> ELK cluster for that kind of purpose. So the right question is: what >>> about switching the default log format to JSON? On my part, I don't >>> see >>> "cons", only "pros", but my judgment is of course biased, as I'm >>> "alone >>> in my corner". But what about you, Community? >>> >>> - Provide a way to configure the output format/handler >>> While poking around in the puppet modules code, I didn't find any >>> way to >>> set the output handler for the logs. For example, in puppet-nova³ we >>> can >>> set a lot of things, but not the useful handler for the output. >>> >>> It would be really cool to get, for each puppet module, the >>> capability >>> to set the handler so that one can just push some stuff in hiera, and >>> Voilà, we have JSON logs. >>> >>> Doing so would allow people to chose between the default (current) >>> output, and something more "computable". >> >> Using the JSON formatter currently requires setting a logging >> configuration file using the standard library configuration format >> and fully specifying things like log levels, handlers, and output >> destination. Would it make sense to add an option in oslo.log to >> give deployers an easier way to enable the JSON formatter? >> > > This would actually be very useful. Great! Let me know if you would like to help figuring out where to do
Re: [openstack-dev] [oslo][tripleo][puppet] Logging format: let's discuss a bit about default format, format configuration and so on
Giving this a bit more thought; I'm slightly more inclined on merely adding the JSON formatter option to the standard oslo.log configuration options. This is because we right now have the ability to pass oslo.cfg options via the CLI, and we would loose that with the advanced logging configuration file. The aforementioned option is something that we're using for containerized openstack services. I'll look into adding the ability to turn that handler on/off from oslo.log. On Mon, Nov 6, 2017 at 8:34 AM, Juan Antonio Osoriowrote: > > > On Mon, Nov 6, 2017 at 8:04 AM, Cédric Jeanneret < > cedric.jeanne...@camptocamp.com> wrote: > >> On 11/04/2017 07:08 AM, Doug Hellmann wrote: >> > Excerpts from Juan Antonio Osorio's message of 2017-11-04 00:11:47 >> +0200: >> >>> On 3 Nov 2017 19:59, "Doug Hellmann" wrote: >> >>> >> >>> Excerpts from Cédric Jeanneret's message of 2017-11-01 14:54:34 +0100: >> Dear Stackers, >> >> While working on my locally deployed Openstack (Pike using TripleO), >> I >> was a bit struggling with the logging part. Currently, all logs are >> pushed to per-service files, in the standard format "one line per >> entry", plain flat text. >> >> It's nice, but if one is wanting to push and index those logs in an >> ELK, >> the current, default format isn't really good. >> >> After some discussions about oslo.log, it appears it provides a nice >> JSONFormatter handler¹ one might want to use for each (python) >> service >> using oslo central library. >> >> A JSON format is really cool, as it's easy to parse for machines, >> and it >> can be on a multi-line without any bit issue - this is especially >> important for stack traces, as their output is multi-line without >> real >> way to have a common delimiter so that we can re-format it and feed >> it >> to any log parser (logstash, fluentd, …). >> >> After some more talks, olso.log will not provide a unified interface >> in >> order to output all received logs as JSON - this makes sens, as that >> would mean "rewrite almost all the python logging management >> interface"², and that's pretty useless, since (all?) services have >> their >> own "logging.conf" file. >> >> That said… to the main purpose of that mail: >> >> - Default format for logs >> A first question would be "are we all OK with the default output >> format" >> - I'm pretty sure "humans" are happy with that, as it's really >> convenient to read and grep. But on a "standard" Openstack deploy, >> I'm >> pretty sure one does not have only one controller, one ceph node and >> one >> compute. Hence comes the log centralization, and with that, the log >> indexation and treatments. >> >> For that, one might argue "I'm using plain files on my logger, and >> grep-ing -r in them". That's a way to do things, and for that, plain, >> flat logs are great. >> >> But… I'm pretty sure I'm not the only one wanting to use some kind of >> ELK cluster for that kind of purpose. So the right question is: what >> about switching the default log format to JSON? On my part, I don't >> see >> "cons", only "pros", but my judgment is of course biased, as I'm >> "alone >> in my corner". But what about you, Community? >> >> - Provide a way to configure the output format/handler >> While poking around in the puppet modules code, I didn't find any >> way to >> set the output handler for the logs. For example, in puppet-nova³ we >> can >> set a lot of things, but not the useful handler for the output. >> >> It would be really cool to get, for each puppet module, the >> capability >> to set the handler so that one can just push some stuff in hiera, and >> Voilà, we have JSON logs. >> >> Doing so would allow people to chose between the default (current) >> output, and something more "computable". >> >>> >> >>> Using the JSON formatter currently requires setting a logging >> >>> configuration file using the standard library configuration format >> >>> and fully specifying things like log levels, handlers, and output >> >>> destination. Would it make sense to add an option in oslo.log to >> >>> give deployers an easier way to enable the JSON formatter? >> >>> >> >> >> >> This would actually be very useful. >> > >> > Great! Let me know if you would like to help figuring out where to do >> > that in the library code. >> >> There's already some (not really documented) feature allowing oslo.log >> to load a configuration file. In fact, there's already one existing in >> the keystone tree (/etc/keystone/logging.conf - altougn not loaded) - we >> might base something "casual" and "generic" on that one. >> >> I think all wsgi/python services should configure the logging through >> that file so that
Re: [openstack-dev] [oslo][tripleo][puppet] Logging format: let's discuss a bit about default format, format configuration and so on
On Mon, Nov 6, 2017 at 8:04 AM, Cédric Jeanneret < cedric.jeanne...@camptocamp.com> wrote: > On 11/04/2017 07:08 AM, Doug Hellmann wrote: > > Excerpts from Juan Antonio Osorio's message of 2017-11-04 00:11:47 +0200: > >>> On 3 Nov 2017 19:59, "Doug Hellmann"wrote: > >>> > >>> Excerpts from Cédric Jeanneret's message of 2017-11-01 14:54:34 +0100: > Dear Stackers, > > While working on my locally deployed Openstack (Pike using TripleO), I > was a bit struggling with the logging part. Currently, all logs are > pushed to per-service files, in the standard format "one line per > entry", plain flat text. > > It's nice, but if one is wanting to push and index those logs in an > ELK, > the current, default format isn't really good. > > After some discussions about oslo.log, it appears it provides a nice > JSONFormatter handler¹ one might want to use for each (python) service > using oslo central library. > > A JSON format is really cool, as it's easy to parse for machines, and > it > can be on a multi-line without any bit issue - this is especially > important for stack traces, as their output is multi-line without real > way to have a common delimiter so that we can re-format it and feed it > to any log parser (logstash, fluentd, …). > > After some more talks, olso.log will not provide a unified interface > in > order to output all received logs as JSON - this makes sens, as that > would mean "rewrite almost all the python logging management > interface"², and that's pretty useless, since (all?) services have > their > own "logging.conf" file. > > That said… to the main purpose of that mail: > > - Default format for logs > A first question would be "are we all OK with the default output > format" > - I'm pretty sure "humans" are happy with that, as it's really > convenient to read and grep. But on a "standard" Openstack deploy, I'm > pretty sure one does not have only one controller, one ceph node and > one > compute. Hence comes the log centralization, and with that, the log > indexation and treatments. > > For that, one might argue "I'm using plain files on my logger, and > grep-ing -r in them". That's a way to do things, and for that, plain, > flat logs are great. > > But… I'm pretty sure I'm not the only one wanting to use some kind of > ELK cluster for that kind of purpose. So the right question is: what > about switching the default log format to JSON? On my part, I don't > see > "cons", only "pros", but my judgment is of course biased, as I'm > "alone > in my corner". But what about you, Community? > > - Provide a way to configure the output format/handler > While poking around in the puppet modules code, I didn't find any way > to > set the output handler for the logs. For example, in puppet-nova³ we > can > set a lot of things, but not the useful handler for the output. > > It would be really cool to get, for each puppet module, the capability > to set the handler so that one can just push some stuff in hiera, and > Voilà, we have JSON logs. > > Doing so would allow people to chose between the default (current) > output, and something more "computable". > >>> > >>> Using the JSON formatter currently requires setting a logging > >>> configuration file using the standard library configuration format > >>> and fully specifying things like log levels, handlers, and output > >>> destination. Would it make sense to add an option in oslo.log to > >>> give deployers an easier way to enable the JSON formatter? > >>> > >> > >> This would actually be very useful. > > > > Great! Let me know if you would like to help figuring out where to do > > that in the library code. > > There's already some (not really documented) feature allowing oslo.log > to load a configuration file. In fact, there's already one existing in > the keystone tree (/etc/keystone/logging.conf - altougn not loaded) - we > might base something "casual" and "generic" on that one. > > I think all wsgi/python services should configure the logging through > that file so that it's clearer and cleaner. But that part is maybe a bit > too big and "aggressive" :). And the logging configuration isn't that > friendly, to be honest, at least I have some issues with its doc ;). > > But I think we might come to something nice and cool. It would allow > anyone to push their own log "formatter", in the end. > So you (oslo.log) wouldn't need to work with format output requests > anymore, just provide two basics (plain and json), and let users play > with the logging configuration (and even output things in XML if they > want) ;). > > Regarding oslo.log: apparently, adding the following entry in the > service configuration file should do it: >
Re: [openstack-dev] [oslo][tripleo][puppet] Logging format: let's discuss a bit about default format, format configuration and so on
On 11/04/2017 07:08 AM, Doug Hellmann wrote: > Excerpts from Juan Antonio Osorio's message of 2017-11-04 00:11:47 +0200: >>> On 3 Nov 2017 19:59, "Doug Hellmann"wrote: >>> >>> Excerpts from Cédric Jeanneret's message of 2017-11-01 14:54:34 +0100: Dear Stackers, While working on my locally deployed Openstack (Pike using TripleO), I was a bit struggling with the logging part. Currently, all logs are pushed to per-service files, in the standard format "one line per entry", plain flat text. It's nice, but if one is wanting to push and index those logs in an ELK, the current, default format isn't really good. After some discussions about oslo.log, it appears it provides a nice JSONFormatter handler¹ one might want to use for each (python) service using oslo central library. A JSON format is really cool, as it's easy to parse for machines, and it can be on a multi-line without any bit issue - this is especially important for stack traces, as their output is multi-line without real way to have a common delimiter so that we can re-format it and feed it to any log parser (logstash, fluentd, …). After some more talks, olso.log will not provide a unified interface in order to output all received logs as JSON - this makes sens, as that would mean "rewrite almost all the python logging management interface"², and that's pretty useless, since (all?) services have their own "logging.conf" file. That said… to the main purpose of that mail: - Default format for logs A first question would be "are we all OK with the default output format" - I'm pretty sure "humans" are happy with that, as it's really convenient to read and grep. But on a "standard" Openstack deploy, I'm pretty sure one does not have only one controller, one ceph node and one compute. Hence comes the log centralization, and with that, the log indexation and treatments. For that, one might argue "I'm using plain files on my logger, and grep-ing -r in them". That's a way to do things, and for that, plain, flat logs are great. But… I'm pretty sure I'm not the only one wanting to use some kind of ELK cluster for that kind of purpose. So the right question is: what about switching the default log format to JSON? On my part, I don't see "cons", only "pros", but my judgment is of course biased, as I'm "alone in my corner". But what about you, Community? - Provide a way to configure the output format/handler While poking around in the puppet modules code, I didn't find any way to set the output handler for the logs. For example, in puppet-nova³ we can set a lot of things, but not the useful handler for the output. It would be really cool to get, for each puppet module, the capability to set the handler so that one can just push some stuff in hiera, and Voilà, we have JSON logs. Doing so would allow people to chose between the default (current) output, and something more "computable". >>> >>> Using the JSON formatter currently requires setting a logging >>> configuration file using the standard library configuration format >>> and fully specifying things like log levels, handlers, and output >>> destination. Would it make sense to add an option in oslo.log to >>> give deployers an easier way to enable the JSON formatter? >>> >> >> This would actually be very useful. > > Great! Let me know if you would like to help figuring out where to do > that in the library code. There's already some (not really documented) feature allowing oslo.log to load a configuration file. In fact, there's already one existing in the keystone tree (/etc/keystone/logging.conf - altougn not loaded) - we might base something "casual" and "generic" on that one. I think all wsgi/python services should configure the logging through that file so that it's clearer and cleaner. But that part is maybe a bit too big and "aggressive" :). And the logging configuration isn't that friendly, to be honest, at least I have some issues with its doc ;). But I think we might come to something nice and cool. It would allow anyone to push their own log "formatter", in the end. So you (oslo.log) wouldn't need to work with format output requests anymore, just provide two basics (plain and json), and let users play with the logging configuration (and even output things in XML if they want) ;). Regarding oslo.log: apparently, adding the following entry in the service configuration file should do it: log-config-append¹ Can anyone confirm that? ¹ https://github.com/openstack/oslo.log/blob/master/oslo_log/_options.py#L44 > > Doug > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe:
Re: [openstack-dev] [oslo][tripleo][puppet] Logging format: let's discuss a bit about default format, format configuration and so on
On Fri, Nov 3, 2017 at 12:21 AM, Cédric Jeanneretwrote: > On 11/02/2017 05:18 PM, Ben Nemec wrote: >> Adding tags for the relevant projects. I _think_ this is mostly >> directed at Puppet/TripleO, but Oslo is obviously relevant as well. > > Thank you! First mail in there, wasn't really sure how to do that. > >> >> On 11/01/2017 08:54 AM, Cédric Jeanneret wrote: >>> Dear Stackers, >>> >>> While working on my locally deployed Openstack (Pike using TripleO), I >>> was a bit struggling with the logging part. Currently, all logs are >>> pushed to per-service files, in the standard format "one line per >>> entry", plain flat text. >>> >>> It's nice, but if one is wanting to push and index those logs in an ELK, >>> the current, default format isn't really good. >>> >>> After some discussions about oslo.log, it appears it provides a nice >>> JSONFormatter handler¹ one might want to use for each (python) service >>> using oslo central library. >>> >>> A JSON format is really cool, as it's easy to parse for machines, and it >>> can be on a multi-line without any bit issue - this is especially >>> important for stack traces, as their output is multi-line without real >>> way to have a common delimiter so that we can re-format it and feed it >>> to any log parser (logstash, fluentd, …). >>> >>> After some more talks, olso.log will not provide a unified interface in >>> order to output all received logs as JSON - this makes sens, as that >>> would mean "rewrite almost all the python logging management >>> interface"², and that's pretty useless, since (all?) services have their >>> own "logging.conf" file. >>> >>> That said… to the main purpose of that mail: >>> >>> - Default format for logs >>> A first question would be "are we all OK with the default output format" >>> - I'm pretty sure "humans" are happy with that, as it's really >>> convenient to read and grep. But on a "standard" Openstack deploy, I'm >>> pretty sure one does not have only one controller, one ceph node and one >>> compute. Hence comes the log centralization, and with that, the log >>> indexation and treatments. >>> >>> For that, one might argue "I'm using plain files on my logger, and >>> grep-ing -r in them". That's a way to do things, and for that, plain, >>> flat logs are great. >>> >>> But… I'm pretty sure I'm not the only one wanting to use some kind of >>> ELK cluster for that kind of purpose. So the right question is: what >>> about switching the default log format to JSON? On my part, I don't see >>> "cons", only "pros", but my judgment is of course biased, as I'm "alone >>> in my corner". But what about you, Community? >> >> The major con I see is that we don't require an ELK stack and reading >> JSON logs if you don't have one of those is probably worse, although I >> will admit I haven't spent much time reading our JSON-formatted logs. My >> experience with things that log in JSON has not generally been positive >> though. It's just not as human-readable. > > We're on the same line on that - hence the following proposal. I was > pretty sure switching the default format was a bad thing, but I had to > submit it in order to be complete ;). > Let's focus on the other one, as it's more friendly for everyone. > >> >> The other problem with changing log format defaults is that many people >> have already set up filters and processing based on the existing log >> format. There are significant user impacts to changing that default. >> >>> >>> - Provide a way to configure the output format/handler >>> While poking around in the puppet modules code, I didn't find any way to >>> set the output handler for the logs. For example, in puppet-nova³ we can >>> set a lot of things, but not the useful handler for the output. >>> >>> It would be really cool to get, for each puppet module, the capability >>> to set the handler so that one can just push some stuff in hiera, and >>> Voilà, we have JSON logs. >>> >>> Doing so would allow people to chose between the default (current) >>> output, and something more "computable". >> >> I think we should do this regardless. There are valid arguments for >> people to want both log formats IMHO and we should allow people to use >> what they want. > > If I understand the things correctly, that would require a "small" > change in every puppet modules so that it configures the service with > the proper log output. Any way to automate something on that? It might > be worth to do some PoC on a specific module maybe? > So for the puppet modules, the available logging configuration lives in puppet-oslo so if there's a configuration around that that needs to be updated[0] then that is the first change. From there you would need to go through each module to expose this new configuration in the logging classes. For example puppet-nova[1]. You could probably write a script to modify each but I'm not sure they are completely consistent between classes. I think they are but it would
Re: [openstack-dev] [oslo][tripleo][puppet] Logging format: let's discuss a bit about default format, format configuration and so on
On 11/02/2017 05:18 PM, Ben Nemec wrote: > Adding tags for the relevant projects. I _think_ this is mostly > directed at Puppet/TripleO, but Oslo is obviously relevant as well. Thank you! First mail in there, wasn't really sure how to do that. > > On 11/01/2017 08:54 AM, Cédric Jeanneret wrote: >> Dear Stackers, >> >> While working on my locally deployed Openstack (Pike using TripleO), I >> was a bit struggling with the logging part. Currently, all logs are >> pushed to per-service files, in the standard format "one line per >> entry", plain flat text. >> >> It's nice, but if one is wanting to push and index those logs in an ELK, >> the current, default format isn't really good. >> >> After some discussions about oslo.log, it appears it provides a nice >> JSONFormatter handler¹ one might want to use for each (python) service >> using oslo central library. >> >> A JSON format is really cool, as it's easy to parse for machines, and it >> can be on a multi-line without any bit issue - this is especially >> important for stack traces, as their output is multi-line without real >> way to have a common delimiter so that we can re-format it and feed it >> to any log parser (logstash, fluentd, …). >> >> After some more talks, olso.log will not provide a unified interface in >> order to output all received logs as JSON - this makes sens, as that >> would mean "rewrite almost all the python logging management >> interface"², and that's pretty useless, since (all?) services have their >> own "logging.conf" file. >> >> That said… to the main purpose of that mail: >> >> - Default format for logs >> A first question would be "are we all OK with the default output format" >> - I'm pretty sure "humans" are happy with that, as it's really >> convenient to read and grep. But on a "standard" Openstack deploy, I'm >> pretty sure one does not have only one controller, one ceph node and one >> compute. Hence comes the log centralization, and with that, the log >> indexation and treatments. >> >> For that, one might argue "I'm using plain files on my logger, and >> grep-ing -r in them". That's a way to do things, and for that, plain, >> flat logs are great. >> >> But… I'm pretty sure I'm not the only one wanting to use some kind of >> ELK cluster for that kind of purpose. So the right question is: what >> about switching the default log format to JSON? On my part, I don't see >> "cons", only "pros", but my judgment is of course biased, as I'm "alone >> in my corner". But what about you, Community? > > The major con I see is that we don't require an ELK stack and reading > JSON logs if you don't have one of those is probably worse, although I > will admit I haven't spent much time reading our JSON-formatted logs. My > experience with things that log in JSON has not generally been positive > though. It's just not as human-readable. We're on the same line on that - hence the following proposal. I was pretty sure switching the default format was a bad thing, but I had to submit it in order to be complete ;). Let's focus on the other one, as it's more friendly for everyone. > > The other problem with changing log format defaults is that many people > have already set up filters and processing based on the existing log > format. There are significant user impacts to changing that default. > >> >> - Provide a way to configure the output format/handler >> While poking around in the puppet modules code, I didn't find any way to >> set the output handler for the logs. For example, in puppet-nova³ we can >> set a lot of things, but not the useful handler for the output. >> >> It would be really cool to get, for each puppet module, the capability >> to set the handler so that one can just push some stuff in hiera, and >> Voilà, we have JSON logs. >> >> Doing so would allow people to chose between the default (current) >> output, and something more "computable". > > I think we should do this regardless. There are valid arguments for > people to want both log formats IMHO and we should allow people to use > what they want. If I understand the things correctly, that would require a "small" change in every puppet modules so that it configures the service with the proper log output. Any way to automate something on that? It might be worth to do some PoC on a specific module maybe? > >> >> Of course, either proposal will require a nice code change in all puppet >> modules (add a new parameter for the foo::logging class, and use that >> new param in the configuration file, and so on), but at least people >> will be able to actually chose. >> >> So, before opening an issue on each launchpad project (that would be… >> long), I'd rather open the discussion in here and, eventually, come to >> some nice, acceptable and accepted solution that would make the >> Openstack Community happy :). >> >> Any thoughts? >> >> Thank you for your attention, feedback and wonderful support for that >> monster project :). >> >> Cheers, >> >> C. >> >>
Re: [openstack-dev] [oslo][tripleo][puppet] Logging format: let's discuss a bit about default format, format configuration and so on
Adding tags for the relevant projects. I _think_ this is mostly directed at Puppet/TripleO, but Oslo is obviously relevant as well. On 11/01/2017 08:54 AM, Cédric Jeanneret wrote: Dear Stackers, While working on my locally deployed Openstack (Pike using TripleO), I was a bit struggling with the logging part. Currently, all logs are pushed to per-service files, in the standard format "one line per entry", plain flat text. It's nice, but if one is wanting to push and index those logs in an ELK, the current, default format isn't really good. After some discussions about oslo.log, it appears it provides a nice JSONFormatter handler¹ one might want to use for each (python) service using oslo central library. A JSON format is really cool, as it's easy to parse for machines, and it can be on a multi-line without any bit issue - this is especially important for stack traces, as their output is multi-line without real way to have a common delimiter so that we can re-format it and feed it to any log parser (logstash, fluentd, …). After some more talks, olso.log will not provide a unified interface in order to output all received logs as JSON - this makes sens, as that would mean "rewrite almost all the python logging management interface"², and that's pretty useless, since (all?) services have their own "logging.conf" file. That said… to the main purpose of that mail: - Default format for logs A first question would be "are we all OK with the default output format" - I'm pretty sure "humans" are happy with that, as it's really convenient to read and grep. But on a "standard" Openstack deploy, I'm pretty sure one does not have only one controller, one ceph node and one compute. Hence comes the log centralization, and with that, the log indexation and treatments. For that, one might argue "I'm using plain files on my logger, and grep-ing -r in them". That's a way to do things, and for that, plain, flat logs are great. But… I'm pretty sure I'm not the only one wanting to use some kind of ELK cluster for that kind of purpose. So the right question is: what about switching the default log format to JSON? On my part, I don't see "cons", only "pros", but my judgment is of course biased, as I'm "alone in my corner". But what about you, Community? The major con I see is that we don't require an ELK stack and reading JSON logs if you don't have one of those is probably worse, although I will admit I haven't spent much time reading our JSON-formatted logs. My experience with things that log in JSON has not generally been positive though. It's just not as human-readable. The other problem with changing log format defaults is that many people have already set up filters and processing based on the existing log format. There are significant user impacts to changing that default. - Provide a way to configure the output format/handler While poking around in the puppet modules code, I didn't find any way to set the output handler for the logs. For example, in puppet-nova³ we can set a lot of things, but not the useful handler for the output. It would be really cool to get, for each puppet module, the capability to set the handler so that one can just push some stuff in hiera, and Voilà, we have JSON logs. Doing so would allow people to chose between the default (current) output, and something more "computable". I think we should do this regardless. There are valid arguments for people to want both log formats IMHO and we should allow people to use what they want. Of course, either proposal will require a nice code change in all puppet modules (add a new parameter for the foo::logging class, and use that new param in the configuration file, and so on), but at least people will be able to actually chose. So, before opening an issue on each launchpad project (that would be… long), I'd rather open the discussion in here and, eventually, come to some nice, acceptable and accepted solution that would make the Openstack Community happy :). Any thoughts? Thank you for your attention, feedback and wonderful support for that monster project :). Cheers, C. ¹ https://github.com/openstack/oslo.log/blob/master/oslo_log/formatters.py#L166-L235 ² http://eavesdrop.openstack.org/irclogs/%23openstack-oslo/%23openstack-oslo.2017-11-01.log.html#t2017-11-01T13:23:14 ³ https://github.com/openstack/puppet-nova/blob/master/manifests/logging.pp __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev