Agree, if most of us like it, I can implement foreach for objects. As of now I use omprog to call a script which uses jq and awk to make it opentsdb and redis friendly. I could have used omredis if foreach worked for objects.
Let us conclude. @Rainer any thoughts on foreach for object? On Mar 23, 2016 1:56 AM, "David Lang" <[email protected]> wrote: > On Wed, 23 Mar 2016, singh.janmejay wrote: > > Makes sense. >> >> ..."counters":["a":80,"b":67] won't work, I think you meant >> ..."counters": [{"a" : 80}, {"b": 67}]. >> > > ahh, showing my lack of knowledge of JSON :-) talking things through: > > so our choices are: > > "counters": [{"a":80},{"b":67}] > vs > "counters": {"a":80,"b":67} > > running these through jq gives > > { > "counters": [ > { > "a": 80 > }, > { > "b": 67 > } > ] > } > > vs > > { > "counters": { > "a": 80, > "b": 67 > } > } > > or to reference a > .counters[0].a > vs > .counters.a > > The latter seems to clearly be the cleaner and more logical one to me. > > Does this seem sane? > > But to use it, we need a variation of foreach that gives the object name > as well as the object contents. > > David Lang > > So it boils down to foreach handling object or not. >> >> Thoughts? >> >> On Wed, Mar 23, 2016 at 1:18 AM, David Lang <[email protected]> wrote: >> >>> On Wed, 23 Mar 2016, singh.janmejay wrote: >>> >>> What about wrapDynamicObjects="on|off"? That is required regardless, >>>> right? (if we want to preserve backward compatibility). Unless we >>>> choose to change it anyway (which im fine with). >>>> >>> >>> >>> I think the fact that data from the log could overwrite name or origin >>> is a >>> fatal flaw and the existing format should not be allowed. So I think we >>> should break backwards compatibility here (if it was more established, >>> I'd >>> be much more reluctant to do so, but at this point, I think it's only >>> early >>> adopters who are using it) >>> >>> So if we are always going to push things down one level, the only >>> question >>> is the format >>> >>> {"name":"h","origin":"dynstats.bucket"},"counters":{"a":80,"b":67}} >>> >>> vs >>> >>> {"name":"h","origin":"dynstats.bucket"},"counters":["a":80,"b":67]} >>> >>> or the horrible >>> >>> >>> {"name":"h","origin":"dynstats.bucket"},"counters":[{"name":"a","value":80},{"name":"b","value":67}]} >>> >>> I really want to avoid this third one. I'd say it's probably better to >>> have >>> to use an external script to process things than make the third one a >>> standard :-) >>> >>> David Lang >>> >>> >>> @Rainer: what do you think about foreach handling object? >>>> >>>> On Wed, Mar 23, 2016 at 1:01 AM, David Lang <[email protected]> wrote: >>>> >>>>> >>>>> On Wed, 23 Mar 2016, singh.janmejay wrote: >>>>> >>>>> Foreach can only work with arrays as of now. It can't work with >>>>>> objects (key-value pairs). So [{name: ... value: ...}, {..},...] is >>>>>> the only format that will work as of now. >>>>>> >>>>>> We can enhance foreach to work with objects. >>>>>> >>>>>> I can make a flag available at dyn-stats bucket level, which can >>>>>> control serialization format, but that would really be a hack. >>>>>> >>>>>> From single-responsibility pattern pov, impstats should be the only >>>>>> component that decides how to layout data for user to see. >>>>>> >>>>>> How about this: >>>>>> >>>>>> impstats(format="json" wrapDynamicObjects="on"...)? >>>>>> >>>>>> It defaults to off, which keeps backward compatibility. >>>>>> >>>>>> So what do you guys think about: >>>>>> - wrapDynamicObjects="on|off" >>>>>> - generating [{name: a, value: 10}, {...} ...] vs. {a: 10, ...} >>>>>> (foreach will handle the former out of the box, but later is concise, >>>>>> readable and light-weight in addition to being more json-y. >>>>>> - enhancing foreach to work with {a: 10, b: 20...} >>>>>> >>>>> >>>>> >>>>> >>>>> If we can enhance foreach to work with the concise format, I would >>>>> rather >>>>> wait for it instead of introducing the wrapping version. >>>>> >>>>> I'm thinking that foreach walks through arrays, rather than mixing >>>>> concepts, >>>>> a foreachobject that gives us a name and contents for a {} list of >>>>> objects >>>>> may be the right thing to do? >>>>> >>>>> foreach just returns a single object while foreachobject needs to >>>>> return >>>>> the >>>>> object and name. >>>>> >>>>> although, if we ever get the ability to address arrays directly, being >>>>> able >>>>> to look at the array position would be the equivalent of the name. >>>>> >>>>> David Lang >>>>> >>>>> >>>>> >>>>> On Wed, Mar 23, 2016 at 12:26 AM, David Lang <[email protected]> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, 22 Mar 2016, singh.janmejay wrote: >>>>>>> >>>>>>> How about this: we support a new flag in impstats which allows >>>>>>>> json-formatted stats-line to optionally use >>>>>>>> encapsulated/wrapped-layout? >>>>>>>> >>>>>>>> impstats(format="json" ...) generates >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> {"name":"msg_per_host","origin":"dynstats.bucket","z-scribe1r-b":80,"SWEB10":67} >>>>>>>> >>>>>>>> however, >>>>>>>> >>>>>>>> impstats(format="json/w" ...) generates {"header": >>>>>>>> {"name":"msg_per_host","origin":"dynstats.bucket"}, "counters" : >>>>>>>> {"z-scribe1r-b":80,"SWEB10":67}} >>>>>>>> >>>>>>>> This is relevant, the serilization format we use right now mixes >>>>>>>> pre-defined keys with counter-names and it can affect regular static >>>>>>>> counters too. >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> with the existing pstats output, there is no ability for user-defined >>>>>>> data >>>>>>> to become a tag name, so there is no potential for ambiguity. but >>>>>>> with >>>>>>> dynastats, this is not a possibility, and the format we use should >>>>>>> prevent >>>>>>> problems. >>>>>>> >>>>>>> Also, just from a conceptual point of view, why should the bucket >>>>>>> contents >>>>>>> be at the same level as the bucket name? >>>>>>> >>>>>>> other than backwards compatibility, what advantage is there of the >>>>>>> current >>>>>>> version in JSON? the documentation uses the plain text equivalant, >>>>>>> which >>>>>>> is >>>>>>> perfectly legitimate because there is an order to the line, and after >>>>>>> you >>>>>>> get past the name and origin, everything else on the line is >>>>>>> name-value >>>>>>> pairs of counters, again, no ambiguity. >>>>>>> >>>>>>> But with JSON, I don't believe that you can depend on tools >>>>>>> maintaining >>>>>>> (or >>>>>>> even identifying) the order of the elements, and if you have multiple >>>>>>> elements with the same name, it's implementation dependent as to >>>>>>> which >>>>>>> one >>>>>>> will be seen. >>>>>>> >>>>>>> So purely from a correctness and defensive programming point of >>>>>>> view, I >>>>>>> think the current JSON serialization should be changed, with the old >>>>>>> format >>>>>>> no longer being an option. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> As to the details of the new format >>>>>>> >>>>>>> What I'm wanting to do with the counters is something like >>>>>>> >>>>>>> if $!origin == "dynstats.bucket" then { >>>>>>> foreach $.tag $!counters { >>>>>>> /var/log/stats;format >>>>>>> } >>>>>>> } >>>>>>> >>>>>>> to output one line per counter. >>>>>>> >>>>>>> I'm very flexible in how to do this, but I would much rather be able >>>>>>> to >>>>>>> do >>>>>>> this inside rsyslog than have to serialize things to an external >>>>>>> script, >>>>>>> have it parse the json and process it. >>>>>>> >>>>>>> my initial thinking was just do >>>>>>> >>>>>>> counters: [ "z-scribe1r-b":80,"SWEB10":67 ] >>>>>>> >>>>>>> but as I'm typing this, I realize that doesn't work as I don't have a >>>>>>> way >>>>>>> to >>>>>>> break $.tag down to reference the name and the value. >>>>>>> >>>>>>> I'd hate to have to do something like >>>>>>> >>>>>>> counters: [{"name":"z-scribe1r-b","value":80 >>>>>>> },{"name":"SWEB10","value":67}] >>>>>>> >>>>>>> this mirrors the misuse of XML that gives it such a horrible >>>>>>> reputation. >>>>>>> But >>>>>>> unless we introduce some new function to rsyslog to break things >>>>>>> down, >>>>>>> I >>>>>>> don't see a better way. If we do need to do something like this, I >>>>>>> sure >>>>>>> would not want to make it the default JSON, which would result in two >>>>>>> different formats. I hate the idea of starting to have different >>>>>>> formats >>>>>>> because of subtypes of data (what is someone wants the cee version of >>>>>>> this >>>>>>> for example, you start to have orthoginal format decisions) >>>>>>> >>>>>>> >>>>>>> David Lang >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>> myriad >>>>>>> of >>>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>>>> DON'T >>>>>>> LIKE THAT. >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com/professional-services/ >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>> myriad >>>>> of >>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>> DON'T >>>>> LIKE THAT. >>>>> >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of >>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >>> LIKE THAT. >>> >> >> >> >> >> _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
