Re: [openstack-dev] [oslo][tripleo][puppet] Logging format: let's discuss a bit about default format, format configuration and so on

[Doug Hellmann]

Excerpts from Juan Antonio Osorio's message of 2017-11-06 15:18:33 +0200:
> Proposed a patch to be able to enable the JSON formatter via an oslo.log
> configuration parameter:
> 
> https://review.openstack.org/#/c/517882/

That's approved, thanks!

We should make sure we have a release note before we tag a new release
of the library, though. Could you do that in another patch?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][tripleo][puppet] Logging format: let's discuss a bit about default format, format configuration and so on

[Juan Antonio Osorio]

Proposed a patch to be able to enable the JSON formatter via an oslo.log
configuration parameter:

https://review.openstack.org/#/c/517882/

On Mon, Nov 6, 2017 at 9:43 AM, Cédric Jeanneret <
cedric.jeanne...@camptocamp.com> wrote:

> On 11/06/2017 08:36 AM, Juan Antonio Osorio wrote:
> > Giving this a bit more thought; I'm slightly more inclined on merely
> adding
> > the JSON formatter option to the standard oslo.log configuration options.
>
> Fine for me.
>
> >
> > This is because we right now have the ability to pass oslo.cfg options
> via
> > the CLI, and we would loose that with the advanced logging configuration
> > file. The aforementioned option is something that we're using for
> > containerized openstack services.
>
> OK - I'll check on my own if I can get something nice with the
> logging.conf file. But that won't be for "tomorrow", as I'm not
> full-time on openstack (unfortunately :(). Both solutions have their
> pros and cons in the end.
>
> >
> > I'll look into adding the ability to turn that handler on/off from
> oslo.log.
>
> Ping me if I can help :). And big thanks for that coming change!
>
> >
> >
> >
> > On Mon, Nov 6, 2017 at 8:34 AM, Juan Antonio Osorio  >
> > wrote:
> >
> >>
> >>
> >> On Mon, Nov 6, 2017 at 8:04 AM, Cédric Jeanneret <
> >> cedric.jeanne...@camptocamp.com> wrote:
> >>
> >>> On 11/04/2017 07:08 AM, Doug Hellmann wrote:
>  Excerpts from Juan Antonio Osorio's message of 2017-11-04 00:11:47
> >>> +0200:
> >> On 3 Nov 2017 19:59, "Doug Hellmann"  wrote:
> >>
> >> Excerpts from Cédric Jeanneret's message of 2017-11-01 14:54:34
> +0100:
> >>> Dear Stackers,
> >>>
> >>> While working on my locally deployed Openstack (Pike using
> TripleO),
> >>> I
> >>> was a bit struggling with the logging part. Currently, all logs are
> >>> pushed to per-service files, in the standard format "one line per
> >>> entry", plain flat text.
> >>>
> >>> It's nice, but if one is wanting to push and index those logs in an
> >>> ELK,
> >>> the current, default format isn't really good.
> >>>
> >>> After some discussions about oslo.log, it appears it provides a
> nice
> >>> JSONFormatter handler¹ one might want to use for each (python)
> >>> service
> >>> using oslo central library.
> >>>
> >>> A JSON format is really cool, as it's easy to parse for machines,
> >>> and it
> >>> can be on a multi-line without any bit issue - this is especially
> >>> important for stack traces, as their output is multi-line without
> >>> real
> >>> way to have a common delimiter so that we can re-format it and feed
> >>> it
> >>> to any log parser (logstash, fluentd, …).
> >>>
> >>> After some more talks, olso.log will not provide a unified
> interface
> >>> in
> >>> order to output all received logs as JSON - this makes sens, as
> that
> >>> would mean "rewrite almost all the python logging management
> >>> interface"², and that's pretty useless, since (all?) services have
> >>> their
> >>> own "logging.conf" file.
> >>>
> >>> That said… to the main purpose of that mail:
> >>>
> >>> - Default format for logs
> >>> A first question would be "are we all OK with the default output
> >>> format"
> >>> - I'm pretty sure "humans" are happy with that, as it's really
> >>> convenient to read and grep. But on a "standard" Openstack deploy,
> >>> I'm
> >>> pretty sure one does not have only one controller, one ceph node
> and
> >>> one
> >>> compute. Hence comes the log centralization, and with that, the log
> >>> indexation and treatments.
> >>>
> >>> For that, one might argue "I'm using plain files on my logger, and
> >>> grep-ing -r in them". That's a way to do things, and for that,
> plain,
> >>> flat logs are great.
> >>>
> >>> But… I'm pretty sure I'm not the only one wanting to use some kind
> of
> >>> ELK cluster for that kind of purpose. So the right question is:
> what
> >>> about switching the default log format to JSON? On my part, I don't
> >>> see
> >>> "cons", only "pros", but my judgment is of course biased, as I'm
> >>> "alone
> >>> in my corner". But what about you, Community?
> >>>
> >>> - Provide a way to configure the output format/handler
> >>> While poking around in the puppet modules code, I didn't find any
> >>> way to
> >>> set the output handler for the logs. For example, in puppet-nova³
> we
> >>> can
> >>> set a lot of things, but not the useful handler for the output.
> >>>
> >>> It would be really cool to get, for each puppet module, the
> >>> capability
> >>> to set the handler so that one can just push some stuff in hiera,
> and
> >>> Voilà, we have JSON logs.
> >>>
> >>> Doing so would allow people to chose between the default  (current)
> >>> output, and something more "computable".
> >>
> >> Using 

Re: [openstack-dev] [oslo][tripleo][puppet] Logging format: let's discuss a bit about default format, format configuration and so on

[Cédric Jeanneret]

On 11/06/2017 08:36 AM, Juan Antonio Osorio wrote:
> Giving this a bit more thought; I'm slightly more inclined on merely adding
> the JSON formatter option to the standard oslo.log configuration options.

Fine for me.

> 
> This is because we right now have the ability to pass oslo.cfg options via
> the CLI, and we would loose that with the advanced logging configuration
> file. The aforementioned option is something that we're using for
> containerized openstack services.

OK - I'll check on my own if I can get something nice with the
logging.conf file. But that won't be for "tomorrow", as I'm not
full-time on openstack (unfortunately :(). Both solutions have their
pros and cons in the end.

> 
> I'll look into adding the ability to turn that handler on/off from oslo.log.

Ping me if I can help :). And big thanks for that coming change!

> 
> 
> 
> On Mon, Nov 6, 2017 at 8:34 AM, Juan Antonio Osorio 
> wrote:
> 
>>
>>
>> On Mon, Nov 6, 2017 at 8:04 AM, Cédric Jeanneret <
>> cedric.jeanne...@camptocamp.com> wrote:
>>
>>> On 11/04/2017 07:08 AM, Doug Hellmann wrote:
 Excerpts from Juan Antonio Osorio's message of 2017-11-04 00:11:47
>>> +0200:
>> On 3 Nov 2017 19:59, "Doug Hellmann"  wrote:
>>
>> Excerpts from Cédric Jeanneret's message of 2017-11-01 14:54:34 +0100:
>>> Dear Stackers,
>>>
>>> While working on my locally deployed Openstack (Pike using TripleO),
>>> I
>>> was a bit struggling with the logging part. Currently, all logs are
>>> pushed to per-service files, in the standard format "one line per
>>> entry", plain flat text.
>>>
>>> It's nice, but if one is wanting to push and index those logs in an
>>> ELK,
>>> the current, default format isn't really good.
>>>
>>> After some discussions about oslo.log, it appears it provides a nice
>>> JSONFormatter handler¹ one might want to use for each (python)
>>> service
>>> using oslo central library.
>>>
>>> A JSON format is really cool, as it's easy to parse for machines,
>>> and it
>>> can be on a multi-line without any bit issue - this is especially
>>> important for stack traces, as their output is multi-line without
>>> real
>>> way to have a common delimiter so that we can re-format it and feed
>>> it
>>> to any log parser (logstash, fluentd, …).
>>>
>>> After some more talks, olso.log will not provide a unified interface
>>> in
>>> order to output all received logs as JSON - this makes sens, as that
>>> would mean "rewrite almost all the python logging management
>>> interface"², and that's pretty useless, since (all?) services have
>>> their
>>> own "logging.conf" file.
>>>
>>> That said… to the main purpose of that mail:
>>>
>>> - Default format for logs
>>> A first question would be "are we all OK with the default output
>>> format"
>>> - I'm pretty sure "humans" are happy with that, as it's really
>>> convenient to read and grep. But on a "standard" Openstack deploy,
>>> I'm
>>> pretty sure one does not have only one controller, one ceph node and
>>> one
>>> compute. Hence comes the log centralization, and with that, the log
>>> indexation and treatments.
>>>
>>> For that, one might argue "I'm using plain files on my logger, and
>>> grep-ing -r in them". That's a way to do things, and for that, plain,
>>> flat logs are great.
>>>
>>> But… I'm pretty sure I'm not the only one wanting to use some kind of
>>> ELK cluster for that kind of purpose. So the right question is: what
>>> about switching the default log format to JSON? On my part, I don't
>>> see
>>> "cons", only "pros", but my judgment is of course biased, as I'm
>>> "alone
>>> in my corner". But what about you, Community?
>>>
>>> - Provide a way to configure the output format/handler
>>> While poking around in the puppet modules code, I didn't find any
>>> way to
>>> set the output handler for the logs. For example, in puppet-nova³ we
>>> can
>>> set a lot of things, but not the useful handler for the output.
>>>
>>> It would be really cool to get, for each puppet module, the
>>> capability
>>> to set the handler so that one can just push some stuff in hiera, and
>>> Voilà, we have JSON logs.
>>>
>>> Doing so would allow people to chose between the default  (current)
>>> output, and something more "computable".
>>
>> Using the JSON formatter currently requires setting a logging
>> configuration file using the standard library configuration format
>> and fully specifying things like log levels, handlers, and output
>> destination. Would it make sense to add an option in oslo.log to
>> give deployers an easier way to enable the JSON formatter?
>>
>
> This would actually be very useful.

 Great! Let me know if you would like to help figuring out where to do
 

Re: [openstack-dev] [oslo][tripleo][puppet] Logging format: let's discuss a bit about default format, format configuration and so on

[Juan Antonio Osorio]

Giving this a bit more thought; I'm slightly more inclined on merely adding
the JSON formatter option to the standard oslo.log configuration options.

This is because we right now have the ability to pass oslo.cfg options via
the CLI, and we would loose that with the advanced logging configuration
file. The aforementioned option is something that we're using for
containerized openstack services.

I'll look into adding the ability to turn that handler on/off from oslo.log.



On Mon, Nov 6, 2017 at 8:34 AM, Juan Antonio Osorio 
wrote:

>
>
> On Mon, Nov 6, 2017 at 8:04 AM, Cédric Jeanneret <
> cedric.jeanne...@camptocamp.com> wrote:
>
>> On 11/04/2017 07:08 AM, Doug Hellmann wrote:
>> > Excerpts from Juan Antonio Osorio's message of 2017-11-04 00:11:47
>> +0200:
>> >>> On 3 Nov 2017 19:59, "Doug Hellmann"  wrote:
>> >>>
>> >>> Excerpts from Cédric Jeanneret's message of 2017-11-01 14:54:34 +0100:
>>  Dear Stackers,
>> 
>>  While working on my locally deployed Openstack (Pike using TripleO),
>> I
>>  was a bit struggling with the logging part. Currently, all logs are
>>  pushed to per-service files, in the standard format "one line per
>>  entry", plain flat text.
>> 
>>  It's nice, but if one is wanting to push and index those logs in an
>> ELK,
>>  the current, default format isn't really good.
>> 
>>  After some discussions about oslo.log, it appears it provides a nice
>>  JSONFormatter handler¹ one might want to use for each (python)
>> service
>>  using oslo central library.
>> 
>>  A JSON format is really cool, as it's easy to parse for machines,
>> and it
>>  can be on a multi-line without any bit issue - this is especially
>>  important for stack traces, as their output is multi-line without
>> real
>>  way to have a common delimiter so that we can re-format it and feed
>> it
>>  to any log parser (logstash, fluentd, …).
>> 
>>  After some more talks, olso.log will not provide a unified interface
>> in
>>  order to output all received logs as JSON - this makes sens, as that
>>  would mean "rewrite almost all the python logging management
>>  interface"², and that's pretty useless, since (all?) services have
>> their
>>  own "logging.conf" file.
>> 
>>  That said… to the main purpose of that mail:
>> 
>>  - Default format for logs
>>  A first question would be "are we all OK with the default output
>> format"
>>  - I'm pretty sure "humans" are happy with that, as it's really
>>  convenient to read and grep. But on a "standard" Openstack deploy,
>> I'm
>>  pretty sure one does not have only one controller, one ceph node and
>> one
>>  compute. Hence comes the log centralization, and with that, the log
>>  indexation and treatments.
>> 
>>  For that, one might argue "I'm using plain files on my logger, and
>>  grep-ing -r in them". That's a way to do things, and for that, plain,
>>  flat logs are great.
>> 
>>  But… I'm pretty sure I'm not the only one wanting to use some kind of
>>  ELK cluster for that kind of purpose. So the right question is: what
>>  about switching the default log format to JSON? On my part, I don't
>> see
>>  "cons", only "pros", but my judgment is of course biased, as I'm
>> "alone
>>  in my corner". But what about you, Community?
>> 
>>  - Provide a way to configure the output format/handler
>>  While poking around in the puppet modules code, I didn't find any
>> way to
>>  set the output handler for the logs. For example, in puppet-nova³ we
>> can
>>  set a lot of things, but not the useful handler for the output.
>> 
>>  It would be really cool to get, for each puppet module, the
>> capability
>>  to set the handler so that one can just push some stuff in hiera, and
>>  Voilà, we have JSON logs.
>> 
>>  Doing so would allow people to chose between the default  (current)
>>  output, and something more "computable".
>> >>>
>> >>> Using the JSON formatter currently requires setting a logging
>> >>> configuration file using the standard library configuration format
>> >>> and fully specifying things like log levels, handlers, and output
>> >>> destination. Would it make sense to add an option in oslo.log to
>> >>> give deployers an easier way to enable the JSON formatter?
>> >>>
>> >>
>> >> This would actually be very useful.
>> >
>> > Great! Let me know if you would like to help figuring out where to do
>> > that in the library code.
>>
>> There's already some (not really documented) feature allowing oslo.log
>> to load a configuration file. In fact, there's already one existing in
>> the keystone tree (/etc/keystone/logging.conf - altougn not loaded) - we
>> might base something "casual" and "generic" on that one.
>>
>> I think all wsgi/python services should configure the logging through
>> that file so that 

Re: [openstack-dev] [oslo][tripleo][puppet] Logging format: let's discuss a bit about default format, format configuration and so on

[Juan Antonio Osorio]

On Mon, Nov 6, 2017 at 8:04 AM, Cédric Jeanneret <
cedric.jeanne...@camptocamp.com> wrote:

> On 11/04/2017 07:08 AM, Doug Hellmann wrote:
> > Excerpts from Juan Antonio Osorio's message of 2017-11-04 00:11:47 +0200:
> >>> On 3 Nov 2017 19:59, "Doug Hellmann"  wrote:
> >>>
> >>> Excerpts from Cédric Jeanneret's message of 2017-11-01 14:54:34 +0100:
>  Dear Stackers,
> 
>  While working on my locally deployed Openstack (Pike using TripleO), I
>  was a bit struggling with the logging part. Currently, all logs are
>  pushed to per-service files, in the standard format "one line per
>  entry", plain flat text.
> 
>  It's nice, but if one is wanting to push and index those logs in an
> ELK,
>  the current, default format isn't really good.
> 
>  After some discussions about oslo.log, it appears it provides a nice
>  JSONFormatter handler¹ one might want to use for each (python) service
>  using oslo central library.
> 
>  A JSON format is really cool, as it's easy to parse for machines, and
> it
>  can be on a multi-line without any bit issue - this is especially
>  important for stack traces, as their output is multi-line without real
>  way to have a common delimiter so that we can re-format it and feed it
>  to any log parser (logstash, fluentd, …).
> 
>  After some more talks, olso.log will not provide a unified interface
> in
>  order to output all received logs as JSON - this makes sens, as that
>  would mean "rewrite almost all the python logging management
>  interface"², and that's pretty useless, since (all?) services have
> their
>  own "logging.conf" file.
> 
>  That said… to the main purpose of that mail:
> 
>  - Default format for logs
>  A first question would be "are we all OK with the default output
> format"
>  - I'm pretty sure "humans" are happy with that, as it's really
>  convenient to read and grep. But on a "standard" Openstack deploy, I'm
>  pretty sure one does not have only one controller, one ceph node and
> one
>  compute. Hence comes the log centralization, and with that, the log
>  indexation and treatments.
> 
>  For that, one might argue "I'm using plain files on my logger, and
>  grep-ing -r in them". That's a way to do things, and for that, plain,
>  flat logs are great.
> 
>  But… I'm pretty sure I'm not the only one wanting to use some kind of
>  ELK cluster for that kind of purpose. So the right question is: what
>  about switching the default log format to JSON? On my part, I don't
> see
>  "cons", only "pros", but my judgment is of course biased, as I'm
> "alone
>  in my corner". But what about you, Community?
> 
>  - Provide a way to configure the output format/handler
>  While poking around in the puppet modules code, I didn't find any way
> to
>  set the output handler for the logs. For example, in puppet-nova³ we
> can
>  set a lot of things, but not the useful handler for the output.
> 
>  It would be really cool to get, for each puppet module, the capability
>  to set the handler so that one can just push some stuff in hiera, and
>  Voilà, we have JSON logs.
> 
>  Doing so would allow people to chose between the default  (current)
>  output, and something more "computable".
> >>>
> >>> Using the JSON formatter currently requires setting a logging
> >>> configuration file using the standard library configuration format
> >>> and fully specifying things like log levels, handlers, and output
> >>> destination. Would it make sense to add an option in oslo.log to
> >>> give deployers an easier way to enable the JSON formatter?
> >>>
> >>
> >> This would actually be very useful.
> >
> > Great! Let me know if you would like to help figuring out where to do
> > that in the library code.
>
> There's already some (not really documented) feature allowing oslo.log
> to load a configuration file. In fact, there's already one existing in
> the keystone tree (/etc/keystone/logging.conf - altougn not loaded) - we
> might base something "casual" and "generic" on that one.
>
> I think all wsgi/python services should configure the logging through
> that file so that it's clearer and cleaner. But that part is maybe a bit
> too big and "aggressive" :). And the logging configuration isn't that
> friendly, to be honest, at least I have some issues with its doc ;).
>
> But I think we might come to something nice and cool. It would allow
> anyone to push  their own log "formatter", in the end.
> So you (oslo.log) wouldn't need to work with format output requests
> anymore, just provide two basics (plain and json), and let users play
> with the logging configuration (and even output things in XML if they
> want) ;).
>
> Regarding oslo.log: apparently, adding the following entry in the
> service configuration file should do it:
> 

Re: [openstack-dev] [oslo][tripleo][puppet] Logging format: let's discuss a bit about default format, format configuration and so on

[Cédric Jeanneret]

On 11/04/2017 07:08 AM, Doug Hellmann wrote:
> Excerpts from Juan Antonio Osorio's message of 2017-11-04 00:11:47 +0200:
>>> On 3 Nov 2017 19:59, "Doug Hellmann"  wrote:
>>>
>>> Excerpts from Cédric Jeanneret's message of 2017-11-01 14:54:34 +0100:
 Dear Stackers,

 While working on my locally deployed Openstack (Pike using TripleO), I
 was a bit struggling with the logging part. Currently, all logs are
 pushed to per-service files, in the standard format "one line per
 entry", plain flat text.

 It's nice, but if one is wanting to push and index those logs in an ELK,
 the current, default format isn't really good.

 After some discussions about oslo.log, it appears it provides a nice
 JSONFormatter handler¹ one might want to use for each (python) service
 using oslo central library.

 A JSON format is really cool, as it's easy to parse for machines, and it
 can be on a multi-line without any bit issue - this is especially
 important for stack traces, as their output is multi-line without real
 way to have a common delimiter so that we can re-format it and feed it
 to any log parser (logstash, fluentd, …).

 After some more talks, olso.log will not provide a unified interface in
 order to output all received logs as JSON - this makes sens, as that
 would mean "rewrite almost all the python logging management
 interface"², and that's pretty useless, since (all?) services have their
 own "logging.conf" file.

 That said… to the main purpose of that mail:

 - Default format for logs
 A first question would be "are we all OK with the default output format"
 - I'm pretty sure "humans" are happy with that, as it's really
 convenient to read and grep. But on a "standard" Openstack deploy, I'm
 pretty sure one does not have only one controller, one ceph node and one
 compute. Hence comes the log centralization, and with that, the log
 indexation and treatments.

 For that, one might argue "I'm using plain files on my logger, and
 grep-ing -r in them". That's a way to do things, and for that, plain,
 flat logs are great.

 But… I'm pretty sure I'm not the only one wanting to use some kind of
 ELK cluster for that kind of purpose. So the right question is: what
 about switching the default log format to JSON? On my part, I don't see
 "cons", only "pros", but my judgment is of course biased, as I'm "alone
 in my corner". But what about you, Community?

 - Provide a way to configure the output format/handler
 While poking around in the puppet modules code, I didn't find any way to
 set the output handler for the logs. For example, in puppet-nova³ we can
 set a lot of things, but not the useful handler for the output.

 It would be really cool to get, for each puppet module, the capability
 to set the handler so that one can just push some stuff in hiera, and
 Voilà, we have JSON logs.

 Doing so would allow people to chose between the default  (current)
 output, and something more "computable".
>>>
>>> Using the JSON formatter currently requires setting a logging
>>> configuration file using the standard library configuration format
>>> and fully specifying things like log levels, handlers, and output
>>> destination. Would it make sense to add an option in oslo.log to
>>> give deployers an easier way to enable the JSON formatter?
>>>
>>
>> This would actually be very useful.
> 
> Great! Let me know if you would like to help figuring out where to do
> that in the library code.

There's already some (not really documented) feature allowing oslo.log
to load a configuration file. In fact, there's already one existing in
the keystone tree (/etc/keystone/logging.conf - altougn not loaded) - we
might base something "casual" and "generic" on that one.

I think all wsgi/python services should configure the logging through
that file so that it's clearer and cleaner. But that part is maybe a bit
too big and "aggressive" :). And the logging configuration isn't that
friendly, to be honest, at least I have some issues with its doc ;).

But I think we might come to something nice and cool. It would allow
anyone to push  their own log "formatter", in the end.
So you (oslo.log) wouldn't need to work with format output requests
anymore, just provide two basics (plain and json), and let users play
with the logging configuration (and even output things in XML if they
want) ;).

Regarding oslo.log: apparently, adding the following entry in the
service configuration file should do it:
log-config-append¹

Can anyone confirm that?


¹ https://github.com/openstack/oslo.log/blob/master/oslo_log/_options.py#L44

> 
> Doug
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 

Re: [openstack-dev] [oslo][tripleo][puppet] Logging format: let's discuss a bit about default format, format configuration and so on

[Alex Schultz]

On Fri, Nov 3, 2017 at 12:21 AM, Cédric Jeanneret
 wrote:
> On 11/02/2017 05:18 PM, Ben Nemec wrote:
>> Adding tags for the relevant projects.  I _think_ this is mostly
>> directed at Puppet/TripleO, but Oslo is obviously relevant as well.
>
> Thank you! First mail in there, wasn't really sure how to do that.
>
>>
>> On 11/01/2017 08:54 AM, Cédric Jeanneret wrote:
>>> Dear Stackers,
>>>
>>> While working on my locally deployed Openstack (Pike using TripleO), I
>>> was a bit struggling with the logging part. Currently, all logs are
>>> pushed to per-service files, in the standard format "one line per
>>> entry", plain flat text.
>>>
>>> It's nice, but if one is wanting to push and index those logs in an ELK,
>>> the current, default format isn't really good.
>>>
>>> After some discussions about oslo.log, it appears it provides a nice
>>> JSONFormatter handler¹ one might want to use for each (python) service
>>> using oslo central library.
>>>
>>> A JSON format is really cool, as it's easy to parse for machines, and it
>>> can be on a multi-line without any bit issue - this is especially
>>> important for stack traces, as their output is multi-line without real
>>> way to have a common delimiter so that we can re-format it and feed it
>>> to any log parser (logstash, fluentd, …).
>>>
>>> After some more talks, olso.log will not provide a unified interface in
>>> order to output all received logs as JSON - this makes sens, as that
>>> would mean "rewrite almost all the python logging management
>>> interface"², and that's pretty useless, since (all?) services have their
>>> own "logging.conf" file.
>>>
>>> That said… to the main purpose of that mail:
>>>
>>> - Default format for logs
>>> A first question would be "are we all OK with the default output format"
>>> - I'm pretty sure "humans" are happy with that, as it's really
>>> convenient to read and grep. But on a "standard" Openstack deploy, I'm
>>> pretty sure one does not have only one controller, one ceph node and one
>>> compute. Hence comes the log centralization, and with that, the log
>>> indexation and treatments.
>>>
>>> For that, one might argue "I'm using plain files on my logger, and
>>> grep-ing -r in them". That's a way to do things, and for that, plain,
>>> flat logs are great.
>>>
>>> But… I'm pretty sure I'm not the only one wanting to use some kind of
>>> ELK cluster for that kind of purpose. So the right question is: what
>>> about switching the default log format to JSON? On my part, I don't see
>>> "cons", only "pros", but my judgment is of course biased, as I'm "alone
>>> in my corner". But what about you, Community?
>>
>> The major con I see is that we don't require an ELK stack and reading
>> JSON logs if you don't have one of those is probably worse, although I
>> will admit I haven't spent much time reading our JSON-formatted logs. My
>> experience with things that log in JSON has not generally been positive
>> though.  It's just not as human-readable.
>
> We're on the same line on that - hence the following proposal. I was
> pretty sure switching the default format was a bad thing, but I had to
> submit it in order to be complete ;).
> Let's focus on the other one, as it's more friendly for everyone.
>
>>
>> The other problem with changing log format defaults is that many people
>> have already set up filters and processing based on the existing log
>> format.  There are significant user impacts to changing that default.
>>
>>>
>>> - Provide a way to configure the output format/handler
>>> While poking around in the puppet modules code, I didn't find any way to
>>> set the output handler for the logs. For example, in puppet-nova³ we can
>>> set a lot of things, but not the useful handler for the output.
>>>
>>> It would be really cool to get, for each puppet module, the capability
>>> to set the handler so that one can just push some stuff in hiera, and
>>> Voilà, we have JSON logs.
>>>
>>> Doing so would allow people to chose between the default  (current)
>>> output, and something more "computable".
>>
>> I think we should do this regardless.  There are valid arguments for
>> people to want both log formats IMHO and we should allow people to use
>> what they want.
>
> If I understand the things correctly, that would require a "small"
> change in every puppet modules so that it configures the service with
> the proper log output. Any way to automate something on that? It might
> be worth to do some PoC on a specific module maybe?
>

So for the puppet modules, the available logging configuration lives
in puppet-oslo so if there's a configuration around that that needs to
be updated[0] then that is the first change. From there you would need
to go through each module to expose this new configuration in the
logging classes. For example puppet-nova[1].  You could probably write
a script to modify each but I'm not sure they are completely
consistent between classes. I think they are but it would 

Re: [openstack-dev] [oslo][tripleo][puppet] Logging format: let's discuss a bit about default format, format configuration and so on

[Cédric Jeanneret]

On 11/02/2017 05:18 PM, Ben Nemec wrote:
> Adding tags for the relevant projects.  I _think_ this is mostly
> directed at Puppet/TripleO, but Oslo is obviously relevant as well.

Thank you! First mail in there, wasn't really sure how to do that.

> 
> On 11/01/2017 08:54 AM, Cédric Jeanneret wrote:
>> Dear Stackers,
>>
>> While working on my locally deployed Openstack (Pike using TripleO), I
>> was a bit struggling with the logging part. Currently, all logs are
>> pushed to per-service files, in the standard format "one line per
>> entry", plain flat text.
>>
>> It's nice, but if one is wanting to push and index those logs in an ELK,
>> the current, default format isn't really good.
>>
>> After some discussions about oslo.log, it appears it provides a nice
>> JSONFormatter handler¹ one might want to use for each (python) service
>> using oslo central library.
>>
>> A JSON format is really cool, as it's easy to parse for machines, and it
>> can be on a multi-line without any bit issue - this is especially
>> important for stack traces, as their output is multi-line without real
>> way to have a common delimiter so that we can re-format it and feed it
>> to any log parser (logstash, fluentd, …).
>>
>> After some more talks, olso.log will not provide a unified interface in
>> order to output all received logs as JSON - this makes sens, as that
>> would mean "rewrite almost all the python logging management
>> interface"², and that's pretty useless, since (all?) services have their
>> own "logging.conf" file.
>>
>> That said… to the main purpose of that mail:
>>
>> - Default format for logs
>> A first question would be "are we all OK with the default output format"
>> - I'm pretty sure "humans" are happy with that, as it's really
>> convenient to read and grep. But on a "standard" Openstack deploy, I'm
>> pretty sure one does not have only one controller, one ceph node and one
>> compute. Hence comes the log centralization, and with that, the log
>> indexation and treatments.
>>
>> For that, one might argue "I'm using plain files on my logger, and
>> grep-ing -r in them". That's a way to do things, and for that, plain,
>> flat logs are great.
>>
>> But… I'm pretty sure I'm not the only one wanting to use some kind of
>> ELK cluster for that kind of purpose. So the right question is: what
>> about switching the default log format to JSON? On my part, I don't see
>> "cons", only "pros", but my judgment is of course biased, as I'm "alone
>> in my corner". But what about you, Community?
> 
> The major con I see is that we don't require an ELK stack and reading
> JSON logs if you don't have one of those is probably worse, although I
> will admit I haven't spent much time reading our JSON-formatted logs. My
> experience with things that log in JSON has not generally been positive
> though.  It's just not as human-readable.

We're on the same line on that - hence the following proposal. I was
pretty sure switching the default format was a bad thing, but I had to
submit it in order to be complete ;).
Let's focus on the other one, as it's more friendly for everyone.

> 
> The other problem with changing log format defaults is that many people
> have already set up filters and processing based on the existing log
> format.  There are significant user impacts to changing that default.
> 
>>
>> - Provide a way to configure the output format/handler
>> While poking around in the puppet modules code, I didn't find any way to
>> set the output handler for the logs. For example, in puppet-nova³ we can
>> set a lot of things, but not the useful handler for the output.
>>
>> It would be really cool to get, for each puppet module, the capability
>> to set the handler so that one can just push some stuff in hiera, and
>> Voilà, we have JSON logs.
>>
>> Doing so would allow people to chose between the default  (current)
>> output, and something more "computable".
> 
> I think we should do this regardless.  There are valid arguments for
> people to want both log formats IMHO and we should allow people to use
> what they want.

If I understand the things correctly, that would require a "small"
change in every puppet modules so that it configures the service with
the proper log output. Any way to automate something on that? It might
be worth to do some PoC on a specific module maybe?


> 
>>
>> Of course, either proposal will require a nice code change in all puppet
>> modules (add a new parameter for the foo::logging class, and use that
>> new param in the configuration file, and so on), but at least people
>> will be able to actually chose.
>>
>> So, before opening an issue on each launchpad project (that would be…
>> long), I'd rather open the discussion in here and, eventually, come to
>> some nice, acceptable and accepted solution that would make the
>> Openstack Community happy :).
>>
>> Any thoughts?
>>
>> Thank you for your attention, feedback and wonderful support for that
>> monster project :).
>>
>> Cheers,
>>
>> C.
>>
>>

Re: [openstack-dev] [oslo][tripleo][puppet] Logging format: let's discuss a bit about default format, format configuration and so on

[Ben Nemec]

Adding tags for the relevant projects.  I _think_ this is mostly 
directed at Puppet/TripleO, but Oslo is obviously relevant as well.


On 11/01/2017 08:54 AM, Cédric Jeanneret wrote:

Dear Stackers,

While working on my locally deployed Openstack (Pike using TripleO), I
was a bit struggling with the logging part. Currently, all logs are
pushed to per-service files, in the standard format "one line per
entry", plain flat text.

It's nice, but if one is wanting to push and index those logs in an ELK,
the current, default format isn't really good.

After some discussions about oslo.log, it appears it provides a nice
JSONFormatter handler¹ one might want to use for each (python) service
using oslo central library.

A JSON format is really cool, as it's easy to parse for machines, and it
can be on a multi-line without any bit issue - this is especially
important for stack traces, as their output is multi-line without real
way to have a common delimiter so that we can re-format it and feed it
to any log parser (logstash, fluentd, …).

After some more talks, olso.log will not provide a unified interface in
order to output all received logs as JSON - this makes sens, as that
would mean "rewrite almost all the python logging management
interface"², and that's pretty useless, since (all?) services have their
own "logging.conf" file.

That said… to the main purpose of that mail:

- Default format for logs
A first question would be "are we all OK with the default output format"
- I'm pretty sure "humans" are happy with that, as it's really
convenient to read and grep. But on a "standard" Openstack deploy, I'm
pretty sure one does not have only one controller, one ceph node and one
compute. Hence comes the log centralization, and with that, the log
indexation and treatments.

For that, one might argue "I'm using plain files on my logger, and
grep-ing -r in them". That's a way to do things, and for that, plain,
flat logs are great.

But… I'm pretty sure I'm not the only one wanting to use some kind of
ELK cluster for that kind of purpose. So the right question is: what
about switching the default log format to JSON? On my part, I don't see
"cons", only "pros", but my judgment is of course biased, as I'm "alone
in my corner". But what about you, Community?


The major con I see is that we don't require an ELK stack and reading 
JSON logs if you don't have one of those is probably worse, although I 
will admit I haven't spent much time reading our JSON-formatted logs. 
My experience with things that log in JSON has not generally been 
positive though.  It's just not as human-readable.


The other problem with changing log format defaults is that many people 
have already set up filters and processing based on the existing log 
format.  There are significant user impacts to changing that default.




- Provide a way to configure the output format/handler
While poking around in the puppet modules code, I didn't find any way to
set the output handler for the logs. For example, in puppet-nova³ we can
set a lot of things, but not the useful handler for the output.

It would be really cool to get, for each puppet module, the capability
to set the handler so that one can just push some stuff in hiera, and
Voilà, we have JSON logs.

Doing so would allow people to chose between the default  (current)
output, and something more "computable".


I think we should do this regardless.  There are valid arguments for 
people to want both log formats IMHO and we should allow people to use 
what they want.




Of course, either proposal will require a nice code change in all puppet
modules (add a new parameter for the foo::logging class, and use that
new param in the configuration file, and so on), but at least people
will be able to actually chose.

So, before opening an issue on each launchpad project (that would be…
long), I'd rather open the discussion in here and, eventually, come to
some nice, acceptable and accepted solution that would make the
Openstack Community happy :).

Any thoughts?

Thank you for your attention, feedback and wonderful support for that
monster project :).

Cheers,

C.


¹
https://github.com/openstack/oslo.log/blob/master/oslo_log/formatters.py#L166-L235
²
http://eavesdrop.openstack.org/irclogs/%23openstack-oslo/%23openstack-oslo.2017-11-01.log.html#t2017-11-01T13:23:14
³ https://github.com/openstack/puppet-nova/blob/master/manifests/logging.pp




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev