Cool. Will fix both stats representation and foreach in one PR. On Mar 25, 2016 3:10 PM, "Rainer Gerhards" <[email protected]> wrote:
> ACK! Please go, let's set aside my probably too strong concerns. I am > perfectly happy when you say it's a small change and glad you do it! > > Rainer > > Sent from phone, thus brief. > Am 25.03.2016 06:21 schrieb "David Lang" <[email protected]>: > > > As I understand Rainer, go . > > > > David Lang > > > > On Fri, 25 Mar 2016, singh.janmejay wrote: > > > > Makes sense. So final call on foreach? Go or no-go? > >> On Mar 25, 2016 2:38 AM, "David Lang" <[email protected]> wrote: > >> > >> Just a note, the dynastats output line needs this as well as the bucket > >>> line. > >>> > >>> This isn't remote-input, but it needs the same ability to iterate over > >>> things. > >>> > >>> example: > >>> > >>> > >>> > >>> > {"name":"global","origin":"dynstats","message_framing.ops_overflow":0,"message_framing.new_metric_add":5,"message_framing.no_metric":0,"message_framing.metrics_purged":0,"message_framing.ops_ignored":0,"message_framing.purge_triggered":0,"msgs_per_host.ops_overflow":0,"msgs_per_host.new_metric_add":0,"msgs_per_host.no_metric":0,"msgs_per_host.metrics_purged":0,"msgs_per_host.ops_ignored":0,"msgs_per_host.purge_triggered":0,"msgs_per_edge_relay.ops_overflow":0,"msgs_per_edge_relay.new_metric_add":0,"msgs_per_edge_relay.no_metric":0,"msgs_per_edge_relay.metrics_purged":0,"msgs_per_edge_relay.ops_ignored":0,"msgs_per_edge_relay.purge_triggered":0,"msgs_per_core_relay.ops_overflow":0,"msgs_per_core_relay.new_metric_add":0,"msgs_per_core_relay.no_metric":0,"msgs_per_core_relay.metrics_purged":0,"msgs_per_core_relay.ops_ignored":0,"msgs_per_core_relay.purge_triggered":0,"msgs_per_program.ops_overflow":0,"msgs_per_program.new_metric_add":0,"msgs_per_program.no_metric":0,"msgs_pe > ! > >>> > >> r_ > > > >> p! > >> > >>> > >>> > >>> > rogram.metrics_purged":0,"msgs_per_program.ops_ignored":0,"msgs_per_program.purge_triggered":0,"msgs_per_tag.ops_overflow":0,"msgs_per_tag.new_metric_add":0,"msgs_per_tag.no_metric":0,"msgs_per_tag.metrics_purged":0,"msgs_per_tag.ops_ignored":0,"msgs_per_tag.purge_triggered":0} > >>> > >>> David Lang > >>> > >>> On Wed, 23 Mar 2016, singh.janmejay wrote: > >>> > >>> Date: Wed, 23 Mar 2016 09:10:47 +0530 > >>> > >>>> From: singh.janmejay <[email protected]> > >>>> Reply-To: rsyslog-users <[email protected]> > >>>> To: rsyslog-users <[email protected]> > >>>> Subject: Re: [rsyslog] dynastats JSON output need work > >>>> > >>>> Agree, if most of us like it, I can implement foreach for objects. As > of > >>>> now I use omprog to call a script which uses jq and awk to make it > >>>> opentsdb > >>>> and redis friendly. I could have used omredis if foreach worked for > >>>> objects. > >>>> > >>>> Let us conclude. > >>>> > >>>> @Rainer any thoughts on foreach for object? > >>>> On Mar 23, 2016 1:56 AM, "David Lang" <[email protected]> wrote: > >>>> > >>>> On Wed, 23 Mar 2016, singh.janmejay wrote: > >>>> > >>>>> > >>>>> Makes sense. > >>>>> > >>>>> > >>>>>> ..."counters":["a":80,"b":67] won't work, I think you meant > >>>>>> ..."counters": [{"a" : 80}, {"b": 67}]. > >>>>>> > >>>>>> > >>>>>> ahh, showing my lack of knowledge of JSON :-) talking things > through: > >>>>> > >>>>> so our choices are: > >>>>> > >>>>> "counters": [{"a":80},{"b":67}] > >>>>> vs > >>>>> "counters": {"a":80,"b":67} > >>>>> > >>>>> running these through jq gives > >>>>> > >>>>> { > >>>>> "counters": [ > >>>>> { > >>>>> "a": 80 > >>>>> }, > >>>>> { > >>>>> "b": 67 > >>>>> } > >>>>> ] > >>>>> } > >>>>> > >>>>> vs > >>>>> > >>>>> { > >>>>> "counters": { > >>>>> "a": 80, > >>>>> "b": 67 > >>>>> } > >>>>> } > >>>>> > >>>>> or to reference a > >>>>> .counters[0].a > >>>>> vs > >>>>> .counters.a > >>>>> > >>>>> The latter seems to clearly be the cleaner and more logical one to > me. > >>>>> > >>>>> Does this seem sane? > >>>>> > >>>>> But to use it, we need a variation of foreach that gives the object > >>>>> name > >>>>> as well as the object contents. > >>>>> > >>>>> David Lang > >>>>> > >>>>> So it boils down to foreach handling object or not. > >>>>> > >>>>> > >>>>>> Thoughts? > >>>>>> > >>>>>> On Wed, Mar 23, 2016 at 1:18 AM, David Lang <[email protected]> wrote: > >>>>>> > >>>>>> On Wed, 23 Mar 2016, singh.janmejay wrote: > >>>>>> > >>>>>>> > >>>>>>> What about wrapDynamicObjects="on|off"? That is required > regardless, > >>>>>>> > >>>>>>> right? (if we want to preserve backward compatibility). Unless we > >>>>>>>> choose to change it anyway (which im fine with). > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>> I think the fact that data from the log could overwrite name or > >>>>>>> origin > >>>>>>> is a > >>>>>>> fatal flaw and the existing format should not be allowed. So I > think > >>>>>>> we > >>>>>>> should break backwards compatibility here (if it was more > >>>>>>> established, > >>>>>>> I'd > >>>>>>> be much more reluctant to do so, but at this point, I think it's > only > >>>>>>> early > >>>>>>> adopters who are using it) > >>>>>>> > >>>>>>> So if we are always going to push things down one level, the only > >>>>>>> question > >>>>>>> is the format > >>>>>>> > >>>>>>> {"name":"h","origin":"dynstats.bucket"},"counters":{"a":80,"b":67}} > >>>>>>> > >>>>>>> vs > >>>>>>> > >>>>>>> {"name":"h","origin":"dynstats.bucket"},"counters":["a":80,"b":67]} > >>>>>>> > >>>>>>> or the horrible > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > {"name":"h","origin":"dynstats.bucket"},"counters":[{"name":"a","value":80},{"name":"b","value":67}]} > >>>>>>> > >>>>>>> I really want to avoid this third one. I'd say it's probably better > >>>>>>> to > >>>>>>> have > >>>>>>> to use an external script to process things than make the third > one a > >>>>>>> standard :-) > >>>>>>> > >>>>>>> David Lang > >>>>>>> > >>>>>>> > >>>>>>> @Rainer: what do you think about foreach handling object? > >>>>>>> > >>>>>>> > >>>>>>>> On Wed, Mar 23, 2016 at 1:01 AM, David Lang <[email protected]> > wrote: > >>>>>>>> > >>>>>>>> > >>>>>>>> On Wed, 23 Mar 2016, singh.janmejay wrote: > >>>>>>>>> > >>>>>>>>> Foreach can only work with arrays as of now. It can't work with > >>>>>>>>> > >>>>>>>>> objects (key-value pairs). So [{name: ... value: ...}, {..},...] > is > >>>>>>>>>> the only format that will work as of now. > >>>>>>>>>> > >>>>>>>>>> We can enhance foreach to work with objects. > >>>>>>>>>> > >>>>>>>>>> I can make a flag available at dyn-stats bucket level, which can > >>>>>>>>>> control serialization format, but that would really be a hack. > >>>>>>>>>> > >>>>>>>>>> From single-responsibility pattern pov, impstats should be the > >>>>>>>>>> only > >>>>>>>>>> component that decides how to layout data for user to see. > >>>>>>>>>> > >>>>>>>>>> How about this: > >>>>>>>>>> > >>>>>>>>>> impstats(format="json" wrapDynamicObjects="on"...)? > >>>>>>>>>> > >>>>>>>>>> It defaults to off, which keeps backward compatibility. > >>>>>>>>>> > >>>>>>>>>> So what do you guys think about: > >>>>>>>>>> - wrapDynamicObjects="on|off" > >>>>>>>>>> - generating [{name: a, value: 10}, {...} ...] vs. {a: 10, ...} > >>>>>>>>>> (foreach will handle the former out of the box, but later is > >>>>>>>>>> concise, > >>>>>>>>>> readable and light-weight in addition to being more json-y. > >>>>>>>>>> - enhancing foreach to work with {a: 10, b: 20...} > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> If we can enhance foreach to work with the concise format, I > would > >>>>>>>>> rather > >>>>>>>>> wait for it instead of introducing the wrapping version. > >>>>>>>>> > >>>>>>>>> I'm thinking that foreach walks through arrays, rather than > mixing > >>>>>>>>> concepts, > >>>>>>>>> a foreachobject that gives us a name and contents for a {} list > of > >>>>>>>>> objects > >>>>>>>>> may be the right thing to do? > >>>>>>>>> > >>>>>>>>> foreach just returns a single object while foreachobject needs to > >>>>>>>>> return > >>>>>>>>> the > >>>>>>>>> object and name. > >>>>>>>>> > >>>>>>>>> although, if we ever get the ability to address arrays directly, > >>>>>>>>> being > >>>>>>>>> able > >>>>>>>>> to look at the array position would be the equivalent of the > name. > >>>>>>>>> > >>>>>>>>> David Lang > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Wed, Mar 23, 2016 at 12:26 AM, David Lang <[email protected]> > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> On Tue, 22 Mar 2016, singh.janmejay wrote: > >>>>>>>>>>> > >>>>>>>>>>> How about this: we support a new flag in impstats which allows > >>>>>>>>>>> > >>>>>>>>>>> json-formatted stats-line to optionally use > >>>>>>>>>>>> encapsulated/wrapped-layout? > >>>>>>>>>>>> > >>>>>>>>>>>> impstats(format="json" ...) generates > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > {"name":"msg_per_host","origin":"dynstats.bucket","z-scribe1r-b":80,"SWEB10":67} > >>>>>>>>>>>> > >>>>>>>>>>>> however, > >>>>>>>>>>>> > >>>>>>>>>>>> impstats(format="json/w" ...) generates {"header": > >>>>>>>>>>>> {"name":"msg_per_host","origin":"dynstats.bucket"}, > "counters" : > >>>>>>>>>>>> {"z-scribe1r-b":80,"SWEB10":67}} > >>>>>>>>>>>> > >>>>>>>>>>>> This is relevant, the serilization format we use right now > mixes > >>>>>>>>>>>> pre-defined keys with counter-names and it can affect regular > >>>>>>>>>>>> static > >>>>>>>>>>>> counters too. > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> with the existing pstats output, there is no ability for > >>>>>>>>>>> user-defined > >>>>>>>>>>> data > >>>>>>>>>>> to become a tag name, so there is no potential for ambiguity. > but > >>>>>>>>>>> with > >>>>>>>>>>> dynastats, this is not a possibility, and the format we use > >>>>>>>>>>> should > >>>>>>>>>>> prevent > >>>>>>>>>>> problems. > >>>>>>>>>>> > >>>>>>>>>>> Also, just from a conceptual point of view, why should the > bucket > >>>>>>>>>>> contents > >>>>>>>>>>> be at the same level as the bucket name? > >>>>>>>>>>> > >>>>>>>>>>> other than backwards compatibility, what advantage is there of > >>>>>>>>>>> the > >>>>>>>>>>> current > >>>>>>>>>>> version in JSON? the documentation uses the plain text > >>>>>>>>>>> equivalant, > >>>>>>>>>>> which > >>>>>>>>>>> is > >>>>>>>>>>> perfectly legitimate because there is an order to the line, and > >>>>>>>>>>> after > >>>>>>>>>>> you > >>>>>>>>>>> get past the name and origin, everything else on the line is > >>>>>>>>>>> name-value > >>>>>>>>>>> pairs of counters, again, no ambiguity. > >>>>>>>>>>> > >>>>>>>>>>> But with JSON, I don't believe that you can depend on tools > >>>>>>>>>>> maintaining > >>>>>>>>>>> (or > >>>>>>>>>>> even identifying) the order of the elements, and if you have > >>>>>>>>>>> multiple > >>>>>>>>>>> elements with the same name, it's implementation dependent as > to > >>>>>>>>>>> which > >>>>>>>>>>> one > >>>>>>>>>>> will be seen. > >>>>>>>>>>> > >>>>>>>>>>> So purely from a correctness and defensive programming point of > >>>>>>>>>>> view, I > >>>>>>>>>>> think the current JSON serialization should be changed, with > the > >>>>>>>>>>> old > >>>>>>>>>>> format > >>>>>>>>>>> no longer being an option. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> As to the details of the new format > >>>>>>>>>>> > >>>>>>>>>>> What I'm wanting to do with the counters is something like > >>>>>>>>>>> > >>>>>>>>>>> if $!origin == "dynstats.bucket" then { > >>>>>>>>>>> foreach $.tag $!counters { > >>>>>>>>>>> /var/log/stats;format > >>>>>>>>>>> } > >>>>>>>>>>> } > >>>>>>>>>>> > >>>>>>>>>>> to output one line per counter. > >>>>>>>>>>> > >>>>>>>>>>> I'm very flexible in how to do this, but I would much rather be > >>>>>>>>>>> able > >>>>>>>>>>> to > >>>>>>>>>>> do > >>>>>>>>>>> this inside rsyslog than have to serialize things to an > external > >>>>>>>>>>> script, > >>>>>>>>>>> have it parse the json and process it. > >>>>>>>>>>> > >>>>>>>>>>> my initial thinking was just do > >>>>>>>>>>> > >>>>>>>>>>> counters: [ "z-scribe1r-b":80,"SWEB10":67 ] > >>>>>>>>>>> > >>>>>>>>>>> but as I'm typing this, I realize that doesn't work as I don't > >>>>>>>>>>> have a > >>>>>>>>>>> way > >>>>>>>>>>> to > >>>>>>>>>>> break $.tag down to reference the name and the value. > >>>>>>>>>>> > >>>>>>>>>>> I'd hate to have to do something like > >>>>>>>>>>> > >>>>>>>>>>> counters: [{"name":"z-scribe1r-b","value":80 > >>>>>>>>>>> },{"name":"SWEB10","value":67}] > >>>>>>>>>>> > >>>>>>>>>>> this mirrors the misuse of XML that gives it such a horrible > >>>>>>>>>>> reputation. > >>>>>>>>>>> But > >>>>>>>>>>> unless we introduce some new function to rsyslog to break > things > >>>>>>>>>>> down, > >>>>>>>>>>> I > >>>>>>>>>>> don't see a better way. If we do need to do something like > this, > >>>>>>>>>>> I > >>>>>>>>>>> sure > >>>>>>>>>>> would not want to make it the default JSON, which would result > in > >>>>>>>>>>> two > >>>>>>>>>>> different formats. I hate the idea of starting to have > different > >>>>>>>>>>> formats > >>>>>>>>>>> because of subtypes of data (what is someone wants the cee > >>>>>>>>>>> version > >>>>>>>>>>> of > >>>>>>>>>>> this > >>>>>>>>>>> for example, you start to have orthoginal format decisions) > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> David Lang > >>>>>>>>>>> _______________________________________________ > >>>>>>>>>>> rsyslog mailing list > >>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>>>>>> http://www.rsyslog.com/professional-services/ > >>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED > by a > >>>>>>>>>>> myriad > >>>>>>>>>>> of > >>>>>>>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > >>>>>>>>>>> you > >>>>>>>>>>> DON'T > >>>>>>>>>>> LIKE THAT. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> _______________________________________________ > >>>>>>>>>> > >>>>>>>>>> rsyslog mailing list > >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>>>> http://www.rsyslog.com/professional-services/ > >>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>>>>>>>> myriad > >>>>>>>>> of > >>>>>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > >>>>>>>>> DON'T > >>>>>>>>> LIKE THAT. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> _______________________________________________ > >>>>>>>> > >>>>>>>> rsyslog mailing list > >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>> http://www.rsyslog.com/professional-services/ > >>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>>>>>> myriad > >>>>>>> of > >>>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>>>>>> DON'T > >>>>>>> LIKE THAT. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>>> _______________________________________________ > >>>>>> > >>>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com/professional-services/ > >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>>>> myriad > >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > >>>>> DON'T LIKE THAT. > >>>>> > >>>>> _______________________________________________ > >>>>> > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com/professional-services/ > >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>>> DON'T LIKE THAT. > >>>> > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com/professional-services/ > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>> DON'T LIKE THAT. > >>> > >>> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> DON'T LIKE THAT. > >> > >> _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
