PR created (not complete yet, not ready for merge): https://github.com/rsyslog/rsyslog/pull/931
Have a quick look to get a feel for namespaced reporting of dyn-stats counters. On Fri, Mar 25, 2016 at 4:25 PM, singh.janmejay <[email protected]> wrote: > Cool. Will fix both stats representation and foreach in one PR. > On Mar 25, 2016 3:10 PM, "Rainer Gerhards" <[email protected]> wrote: > >> ACK! Please go, let's set aside my probably too strong concerns. I am >> perfectly happy when you say it's a small change and glad you do it! >> >> Rainer >> >> Sent from phone, thus brief. >> Am 25.03.2016 06:21 schrieb "David Lang" <[email protected]>: >> >> > As I understand Rainer, go . >> > >> > David Lang >> > >> > On Fri, 25 Mar 2016, singh.janmejay wrote: >> > >> > Makes sense. So final call on foreach? Go or no-go? >> >> On Mar 25, 2016 2:38 AM, "David Lang" <[email protected]> wrote: >> >> >> >> Just a note, the dynastats output line needs this as well as the bucket >> >>> line. >> >>> >> >>> This isn't remote-input, but it needs the same ability to iterate over >> >>> things. >> >>> >> >>> example: >> >>> >> >>> >> >>> >> >>> >> {"name":"global","origin":"dynstats","message_framing.ops_overflow":0,"message_framing.new_metric_add":5,"message_framing.no_metric":0,"message_framing.metrics_purged":0,"message_framing.ops_ignored":0,"message_framing.purge_triggered":0,"msgs_per_host.ops_overflow":0,"msgs_per_host.new_metric_add":0,"msgs_per_host.no_metric":0,"msgs_per_host.metrics_purged":0,"msgs_per_host.ops_ignored":0,"msgs_per_host.purge_triggered":0,"msgs_per_edge_relay.ops_overflow":0,"msgs_per_edge_relay.new_metric_add":0,"msgs_per_edge_relay.no_metric":0,"msgs_per_edge_relay.metrics_purged":0,"msgs_per_edge_relay.ops_ignored":0,"msgs_per_edge_relay.purge_triggered":0,"msgs_per_core_relay.ops_overflow":0,"msgs_per_core_relay.new_metric_add":0,"msgs_per_core_relay.no_metric":0,"msgs_per_core_relay.metrics_purged":0,"msgs_per_core_relay.ops_ignored":0,"msgs_per_core_relay.purge_triggered":0,"msgs_per_program.ops_overflow":0,"msgs_per_program.new_metric_add":0,"msgs_per_program.no_metric":0,"msgs_pe >> ! >> >>> >> >> r_ >> > >> >> p! >> >> >> >>> >> >>> >> >>> >> rogram.metrics_purged":0,"msgs_per_program.ops_ignored":0,"msgs_per_program.purge_triggered":0,"msgs_per_tag.ops_overflow":0,"msgs_per_tag.new_metric_add":0,"msgs_per_tag.no_metric":0,"msgs_per_tag.metrics_purged":0,"msgs_per_tag.ops_ignored":0,"msgs_per_tag.purge_triggered":0} >> >>> >> >>> David Lang >> >>> >> >>> On Wed, 23 Mar 2016, singh.janmejay wrote: >> >>> >> >>> Date: Wed, 23 Mar 2016 09:10:47 +0530 >> >>> >> >>>> From: singh.janmejay <[email protected]> >> >>>> Reply-To: rsyslog-users <[email protected]> >> >>>> To: rsyslog-users <[email protected]> >> >>>> Subject: Re: [rsyslog] dynastats JSON output need work >> >>>> >> >>>> Agree, if most of us like it, I can implement foreach for objects. As >> of >> >>>> now I use omprog to call a script which uses jq and awk to make it >> >>>> opentsdb >> >>>> and redis friendly. I could have used omredis if foreach worked for >> >>>> objects. >> >>>> >> >>>> Let us conclude. >> >>>> >> >>>> @Rainer any thoughts on foreach for object? >> >>>> On Mar 23, 2016 1:56 AM, "David Lang" <[email protected]> wrote: >> >>>> >> >>>> On Wed, 23 Mar 2016, singh.janmejay wrote: >> >>>> >> >>>>> >> >>>>> Makes sense. >> >>>>> >> >>>>> >> >>>>>> ..."counters":["a":80,"b":67] won't work, I think you meant >> >>>>>> ..."counters": [{"a" : 80}, {"b": 67}]. >> >>>>>> >> >>>>>> >> >>>>>> ahh, showing my lack of knowledge of JSON :-) talking things >> through: >> >>>>> >> >>>>> so our choices are: >> >>>>> >> >>>>> "counters": [{"a":80},{"b":67}] >> >>>>> vs >> >>>>> "counters": {"a":80,"b":67} >> >>>>> >> >>>>> running these through jq gives >> >>>>> >> >>>>> { >> >>>>> "counters": [ >> >>>>> { >> >>>>> "a": 80 >> >>>>> }, >> >>>>> { >> >>>>> "b": 67 >> >>>>> } >> >>>>> ] >> >>>>> } >> >>>>> >> >>>>> vs >> >>>>> >> >>>>> { >> >>>>> "counters": { >> >>>>> "a": 80, >> >>>>> "b": 67 >> >>>>> } >> >>>>> } >> >>>>> >> >>>>> or to reference a >> >>>>> .counters[0].a >> >>>>> vs >> >>>>> .counters.a >> >>>>> >> >>>>> The latter seems to clearly be the cleaner and more logical one to >> me. >> >>>>> >> >>>>> Does this seem sane? >> >>>>> >> >>>>> But to use it, we need a variation of foreach that gives the object >> >>>>> name >> >>>>> as well as the object contents. >> >>>>> >> >>>>> David Lang >> >>>>> >> >>>>> So it boils down to foreach handling object or not. >> >>>>> >> >>>>> >> >>>>>> Thoughts? >> >>>>>> >> >>>>>> On Wed, Mar 23, 2016 at 1:18 AM, David Lang <[email protected]> wrote: >> >>>>>> >> >>>>>> On Wed, 23 Mar 2016, singh.janmejay wrote: >> >>>>>> >> >>>>>>> >> >>>>>>> What about wrapDynamicObjects="on|off"? That is required >> regardless, >> >>>>>>> >> >>>>>>> right? (if we want to preserve backward compatibility). Unless we >> >>>>>>>> choose to change it anyway (which im fine with). >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>> I think the fact that data from the log could overwrite name or >> >>>>>>> origin >> >>>>>>> is a >> >>>>>>> fatal flaw and the existing format should not be allowed. So I >> think >> >>>>>>> we >> >>>>>>> should break backwards compatibility here (if it was more >> >>>>>>> established, >> >>>>>>> I'd >> >>>>>>> be much more reluctant to do so, but at this point, I think it's >> only >> >>>>>>> early >> >>>>>>> adopters who are using it) >> >>>>>>> >> >>>>>>> So if we are always going to push things down one level, the only >> >>>>>>> question >> >>>>>>> is the format >> >>>>>>> >> >>>>>>> {"name":"h","origin":"dynstats.bucket"},"counters":{"a":80,"b":67}} >> >>>>>>> >> >>>>>>> vs >> >>>>>>> >> >>>>>>> {"name":"h","origin":"dynstats.bucket"},"counters":["a":80,"b":67]} >> >>>>>>> >> >>>>>>> or the horrible >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> {"name":"h","origin":"dynstats.bucket"},"counters":[{"name":"a","value":80},{"name":"b","value":67}]} >> >>>>>>> >> >>>>>>> I really want to avoid this third one. I'd say it's probably better >> >>>>>>> to >> >>>>>>> have >> >>>>>>> to use an external script to process things than make the third >> one a >> >>>>>>> standard :-) >> >>>>>>> >> >>>>>>> David Lang >> >>>>>>> >> >>>>>>> >> >>>>>>> @Rainer: what do you think about foreach handling object? >> >>>>>>> >> >>>>>>> >> >>>>>>>> On Wed, Mar 23, 2016 at 1:01 AM, David Lang <[email protected]> >> wrote: >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> On Wed, 23 Mar 2016, singh.janmejay wrote: >> >>>>>>>>> >> >>>>>>>>> Foreach can only work with arrays as of now. It can't work with >> >>>>>>>>> >> >>>>>>>>> objects (key-value pairs). So [{name: ... value: ...}, {..},...] >> is >> >>>>>>>>>> the only format that will work as of now. >> >>>>>>>>>> >> >>>>>>>>>> We can enhance foreach to work with objects. >> >>>>>>>>>> >> >>>>>>>>>> I can make a flag available at dyn-stats bucket level, which can >> >>>>>>>>>> control serialization format, but that would really be a hack. >> >>>>>>>>>> >> >>>>>>>>>> From single-responsibility pattern pov, impstats should be the >> >>>>>>>>>> only >> >>>>>>>>>> component that decides how to layout data for user to see. >> >>>>>>>>>> >> >>>>>>>>>> How about this: >> >>>>>>>>>> >> >>>>>>>>>> impstats(format="json" wrapDynamicObjects="on"...)? >> >>>>>>>>>> >> >>>>>>>>>> It defaults to off, which keeps backward compatibility. >> >>>>>>>>>> >> >>>>>>>>>> So what do you guys think about: >> >>>>>>>>>> - wrapDynamicObjects="on|off" >> >>>>>>>>>> - generating [{name: a, value: 10}, {...} ...] vs. {a: 10, ...} >> >>>>>>>>>> (foreach will handle the former out of the box, but later is >> >>>>>>>>>> concise, >> >>>>>>>>>> readable and light-weight in addition to being more json-y. >> >>>>>>>>>> - enhancing foreach to work with {a: 10, b: 20...} >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> If we can enhance foreach to work with the concise format, I >> would >> >>>>>>>>> rather >> >>>>>>>>> wait for it instead of introducing the wrapping version. >> >>>>>>>>> >> >>>>>>>>> I'm thinking that foreach walks through arrays, rather than >> mixing >> >>>>>>>>> concepts, >> >>>>>>>>> a foreachobject that gives us a name and contents for a {} list >> of >> >>>>>>>>> objects >> >>>>>>>>> may be the right thing to do? >> >>>>>>>>> >> >>>>>>>>> foreach just returns a single object while foreachobject needs to >> >>>>>>>>> return >> >>>>>>>>> the >> >>>>>>>>> object and name. >> >>>>>>>>> >> >>>>>>>>> although, if we ever get the ability to address arrays directly, >> >>>>>>>>> being >> >>>>>>>>> able >> >>>>>>>>> to look at the array position would be the equivalent of the >> name. >> >>>>>>>>> >> >>>>>>>>> David Lang >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> On Wed, Mar 23, 2016 at 12:26 AM, David Lang <[email protected]> >> >>>>>>>>> wrote: >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>>> On Tue, 22 Mar 2016, singh.janmejay wrote: >> >>>>>>>>>>> >> >>>>>>>>>>> How about this: we support a new flag in impstats which allows >> >>>>>>>>>>> >> >>>>>>>>>>> json-formatted stats-line to optionally use >> >>>>>>>>>>>> encapsulated/wrapped-layout? >> >>>>>>>>>>>> >> >>>>>>>>>>>> impstats(format="json" ...) generates >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> {"name":"msg_per_host","origin":"dynstats.bucket","z-scribe1r-b":80,"SWEB10":67} >> >>>>>>>>>>>> >> >>>>>>>>>>>> however, >> >>>>>>>>>>>> >> >>>>>>>>>>>> impstats(format="json/w" ...) generates {"header": >> >>>>>>>>>>>> {"name":"msg_per_host","origin":"dynstats.bucket"}, >> "counters" : >> >>>>>>>>>>>> {"z-scribe1r-b":80,"SWEB10":67}} >> >>>>>>>>>>>> >> >>>>>>>>>>>> This is relevant, the serilization format we use right now >> mixes >> >>>>>>>>>>>> pre-defined keys with counter-names and it can affect regular >> >>>>>>>>>>>> static >> >>>>>>>>>>>> counters too. >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> with the existing pstats output, there is no ability for >> >>>>>>>>>>> user-defined >> >>>>>>>>>>> data >> >>>>>>>>>>> to become a tag name, so there is no potential for ambiguity. >> but >> >>>>>>>>>>> with >> >>>>>>>>>>> dynastats, this is not a possibility, and the format we use >> >>>>>>>>>>> should >> >>>>>>>>>>> prevent >> >>>>>>>>>>> problems. >> >>>>>>>>>>> >> >>>>>>>>>>> Also, just from a conceptual point of view, why should the >> bucket >> >>>>>>>>>>> contents >> >>>>>>>>>>> be at the same level as the bucket name? >> >>>>>>>>>>> >> >>>>>>>>>>> other than backwards compatibility, what advantage is there of >> >>>>>>>>>>> the >> >>>>>>>>>>> current >> >>>>>>>>>>> version in JSON? the documentation uses the plain text >> >>>>>>>>>>> equivalant, >> >>>>>>>>>>> which >> >>>>>>>>>>> is >> >>>>>>>>>>> perfectly legitimate because there is an order to the line, and >> >>>>>>>>>>> after >> >>>>>>>>>>> you >> >>>>>>>>>>> get past the name and origin, everything else on the line is >> >>>>>>>>>>> name-value >> >>>>>>>>>>> pairs of counters, again, no ambiguity. >> >>>>>>>>>>> >> >>>>>>>>>>> But with JSON, I don't believe that you can depend on tools >> >>>>>>>>>>> maintaining >> >>>>>>>>>>> (or >> >>>>>>>>>>> even identifying) the order of the elements, and if you have >> >>>>>>>>>>> multiple >> >>>>>>>>>>> elements with the same name, it's implementation dependent as >> to >> >>>>>>>>>>> which >> >>>>>>>>>>> one >> >>>>>>>>>>> will be seen. >> >>>>>>>>>>> >> >>>>>>>>>>> So purely from a correctness and defensive programming point of >> >>>>>>>>>>> view, I >> >>>>>>>>>>> think the current JSON serialization should be changed, with >> the >> >>>>>>>>>>> old >> >>>>>>>>>>> format >> >>>>>>>>>>> no longer being an option. >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> As to the details of the new format >> >>>>>>>>>>> >> >>>>>>>>>>> What I'm wanting to do with the counters is something like >> >>>>>>>>>>> >> >>>>>>>>>>> if $!origin == "dynstats.bucket" then { >> >>>>>>>>>>> foreach $.tag $!counters { >> >>>>>>>>>>> /var/log/stats;format >> >>>>>>>>>>> } >> >>>>>>>>>>> } >> >>>>>>>>>>> >> >>>>>>>>>>> to output one line per counter. >> >>>>>>>>>>> >> >>>>>>>>>>> I'm very flexible in how to do this, but I would much rather be >> >>>>>>>>>>> able >> >>>>>>>>>>> to >> >>>>>>>>>>> do >> >>>>>>>>>>> this inside rsyslog than have to serialize things to an >> external >> >>>>>>>>>>> script, >> >>>>>>>>>>> have it parse the json and process it. >> >>>>>>>>>>> >> >>>>>>>>>>> my initial thinking was just do >> >>>>>>>>>>> >> >>>>>>>>>>> counters: [ "z-scribe1r-b":80,"SWEB10":67 ] >> >>>>>>>>>>> >> >>>>>>>>>>> but as I'm typing this, I realize that doesn't work as I don't >> >>>>>>>>>>> have a >> >>>>>>>>>>> way >> >>>>>>>>>>> to >> >>>>>>>>>>> break $.tag down to reference the name and the value. >> >>>>>>>>>>> >> >>>>>>>>>>> I'd hate to have to do something like >> >>>>>>>>>>> >> >>>>>>>>>>> counters: [{"name":"z-scribe1r-b","value":80 >> >>>>>>>>>>> },{"name":"SWEB10","value":67}] >> >>>>>>>>>>> >> >>>>>>>>>>> this mirrors the misuse of XML that gives it such a horrible >> >>>>>>>>>>> reputation. >> >>>>>>>>>>> But >> >>>>>>>>>>> unless we introduce some new function to rsyslog to break >> things >> >>>>>>>>>>> down, >> >>>>>>>>>>> I >> >>>>>>>>>>> don't see a better way. If we do need to do something like >> this, >> >>>>>>>>>>> I >> >>>>>>>>>>> sure >> >>>>>>>>>>> would not want to make it the default JSON, which would result >> in >> >>>>>>>>>>> two >> >>>>>>>>>>> different formats. I hate the idea of starting to have >> different >> >>>>>>>>>>> formats >> >>>>>>>>>>> because of subtypes of data (what is someone wants the cee >> >>>>>>>>>>> version >> >>>>>>>>>>> of >> >>>>>>>>>>> this >> >>>>>>>>>>> for example, you start to have orthoginal format decisions) >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> David Lang >> >>>>>>>>>>> _______________________________________________ >> >>>>>>>>>>> rsyslog mailing list >> >>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>>>>>>>>>> http://www.rsyslog.com/professional-services/ >> >>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED >> by a >> >>>>>>>>>>> myriad >> >>>>>>>>>>> of >> >>>>>>>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> >>>>>>>>>>> you >> >>>>>>>>>>> DON'T >> >>>>>>>>>>> LIKE THAT. >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> _______________________________________________ >> >>>>>>>>>> >> >>>>>>>>>> rsyslog mailing list >> >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>>>>>>>> http://www.rsyslog.com/professional-services/ >> >>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> >>>>>>>>> myriad >> >>>>>>>>> of >> >>>>>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> you >> >>>>>>>>> DON'T >> >>>>>>>>> LIKE THAT. >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> _______________________________________________ >> >>>>>>>> >> >>>>>>>> rsyslog mailing list >> >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>>>>>> http://www.rsyslog.com/professional-services/ >> >>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> >>>>>>> myriad >> >>>>>>> of >> >>>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> >>>>>>> DON'T >> >>>>>>> LIKE THAT. >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>> >> >>>>>> >> >>>>>> _______________________________________________ >> >>>>>> >> >>>>>> rsyslog mailing list >> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>>>> http://www.rsyslog.com/professional-services/ >> >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> >>>>> myriad >> >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> you >> >>>>> DON'T LIKE THAT. >> >>>>> >> >>>>> _______________________________________________ >> >>>>> >> >>>> rsyslog mailing list >> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>>> http://www.rsyslog.com/professional-services/ >> >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> myriad >> >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> >>>> DON'T LIKE THAT. >> >>>> >> >>>> >> >>> _______________________________________________ >> >>> rsyslog mailing list >> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>> http://www.rsyslog.com/professional-services/ >> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> myriad >> >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> >>> DON'T LIKE THAT. >> >>> >> >>> _______________________________________________ >> >> rsyslog mailing list >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> http://www.rsyslog.com/professional-services/ >> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> >> DON'T LIKE THAT. >> >> >> >> _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com/professional-services/ >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> > DON'T LIKE THAT. >> > >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. -- Regards, Janmejay http://codehunk.wordpress.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
