>From variable-management pov, loop as a construct will be required as long we have array, associative-array etc.
But logs do have multi-valued fields, they have key-value pairs etc right? So regardless of the change in variable-system, array/associative-array data-type will continue to exist. Which in turn requires ways to work with that data-type. Doesn't it? On Wed, Mar 23, 2016 at 4:58 PM, singh.janmejay <[email protected]> wrote: > In-code footprint of foreach enhancement (to handle object in addition > to array) will be fairly small (I am thinking under 40 lines?). > Majority of addition will be in form of tests (unless we choose to > change the rainerscript signature of that statement). > > Alternatively, from data-traversal functional-completeness pov, the > same thing can be achieved without touching foreach impl, in a > slightly high-overhead way. > > We can create a new rainerscript fn make_list(<object>) => > [kv-tuples...] (or mklist). > > Eg. > {"foo" : "bar", "baz" : "quux"} becomes [{"key": "foo", "value": > "bar"}, {"key": "baz", "value": "quux"}] > > Clearly it will have inferior performance, but it'll make foreach work > for all object use-cases. > > I really feel in terms of complexity though, this (transform approach) > is worse than first-class support (purely because of it using a > round-about way of working). > > > On Wed, Mar 23, 2016 at 3:01 PM, Rainer Gerhards > <[email protected]> wrote: >> 2016-03-22 20:34 GMT+01:00 singh.janmejay <[email protected]>: >>> What about wrapDynamicObjects="on|off"? That is required regardless, >>> right? (if we want to preserve backward compatibility). Unless we >>> choose to change it anyway (which im fine with). >>> >>> @Rainer: what do you think about foreach handling object? >> >> so far I could only manage to have a glimpse at the conversation. I >> admit that I am really concerned about all the extra complexity we >> introduce. And do so for what I think is a border-use case at best. >> There is a lot of work that should really be done in regard to >> variables and performance and I am unsure if this is the right time to >> do large extensions... >> >> I was happy with the dynstats as it looked like a very contained >> solution that did not affect much else. But now it looks we need to >> touch a big deal of code ... code that's not really ready for that... >> >> Maybe I get a different view when I find time to read all this more >> in-depth.... >> >> Rainer >>> >>> On Wed, Mar 23, 2016 at 1:01 AM, David Lang <[email protected]> wrote: >>>> On Wed, 23 Mar 2016, singh.janmejay wrote: >>>> >>>>> Foreach can only work with arrays as of now. It can't work with >>>>> objects (key-value pairs). So [{name: ... value: ...}, {..},...] is >>>>> the only format that will work as of now. >>>>> >>>>> We can enhance foreach to work with objects. >>>>> >>>>> I can make a flag available at dyn-stats bucket level, which can >>>>> control serialization format, but that would really be a hack. >>>>> >>>>> From single-responsibility pattern pov, impstats should be the only >>>>> component that decides how to layout data for user to see. >>>>> >>>>> How about this: >>>>> >>>>> impstats(format="json" wrapDynamicObjects="on"...)? >>>>> >>>>> It defaults to off, which keeps backward compatibility. >>>>> >>>>> So what do you guys think about: >>>>> - wrapDynamicObjects="on|off" >>>>> - generating [{name: a, value: 10}, {...} ...] vs. {a: 10, ...} >>>>> (foreach will handle the former out of the box, but later is concise, >>>>> readable and light-weight in addition to being more json-y. >>>>> - enhancing foreach to work with {a: 10, b: 20...} >>>> >>>> >>>> If we can enhance foreach to work with the concise format, I would rather >>>> wait for it instead of introducing the wrapping version. >>>> >>>> I'm thinking that foreach walks through arrays, rather than mixing >>>> concepts, >>>> a foreachobject that gives us a name and contents for a {} list of objects >>>> may be the right thing to do? >>>> >>>> foreach just returns a single object while foreachobject needs to return >>>> the >>>> object and name. >>>> >>>> although, if we ever get the ability to address arrays directly, being able >>>> to look at the array position would be the equivalent of the name. >>>> >>>> David Lang >>>> >>>> >>>> >>>>> On Wed, Mar 23, 2016 at 12:26 AM, David Lang <[email protected]> wrote: >>>>>> >>>>>> On Tue, 22 Mar 2016, singh.janmejay wrote: >>>>>> >>>>>>> How about this: we support a new flag in impstats which allows >>>>>>> json-formatted stats-line to optionally use >>>>>>> encapsulated/wrapped-layout? >>>>>>> >>>>>>> impstats(format="json" ...) generates >>>>>>> >>>>>>> >>>>>>> {"name":"msg_per_host","origin":"dynstats.bucket","z-scribe1r-b":80,"SWEB10":67} >>>>>>> >>>>>>> however, >>>>>>> >>>>>>> impstats(format="json/w" ...) generates {"header": >>>>>>> {"name":"msg_per_host","origin":"dynstats.bucket"}, "counters" : >>>>>>> {"z-scribe1r-b":80,"SWEB10":67}} >>>>>>> >>>>>>> This is relevant, the serilization format we use right now mixes >>>>>>> pre-defined keys with counter-names and it can affect regular static >>>>>>> counters too. >>>>>> >>>>>> >>>>>> >>>>>> with the existing pstats output, there is no ability for user-defined >>>>>> data >>>>>> to become a tag name, so there is no potential for ambiguity. but with >>>>>> dynastats, this is not a possibility, and the format we use should >>>>>> prevent >>>>>> problems. >>>>>> >>>>>> Also, just from a conceptual point of view, why should the bucket >>>>>> contents >>>>>> be at the same level as the bucket name? >>>>>> >>>>>> other than backwards compatibility, what advantage is there of the >>>>>> current >>>>>> version in JSON? the documentation uses the plain text equivalant, which >>>>>> is >>>>>> perfectly legitimate because there is an order to the line, and after you >>>>>> get past the name and origin, everything else on the line is name-value >>>>>> pairs of counters, again, no ambiguity. >>>>>> >>>>>> But with JSON, I don't believe that you can depend on tools maintaining >>>>>> (or >>>>>> even identifying) the order of the elements, and if you have multiple >>>>>> elements with the same name, it's implementation dependent as to which >>>>>> one >>>>>> will be seen. >>>>>> >>>>>> So purely from a correctness and defensive programming point of view, I >>>>>> think the current JSON serialization should be changed, with the old >>>>>> format >>>>>> no longer being an option. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> As to the details of the new format >>>>>> >>>>>> What I'm wanting to do with the counters is something like >>>>>> >>>>>> if $!origin == "dynstats.bucket" then { >>>>>> foreach $.tag $!counters { >>>>>> /var/log/stats;format >>>>>> } >>>>>> } >>>>>> >>>>>> to output one line per counter. >>>>>> >>>>>> I'm very flexible in how to do this, but I would much rather be able to >>>>>> do >>>>>> this inside rsyslog than have to serialize things to an external script, >>>>>> have it parse the json and process it. >>>>>> >>>>>> my initial thinking was just do >>>>>> >>>>>> counters: [ "z-scribe1r-b":80,"SWEB10":67 ] >>>>>> >>>>>> but as I'm typing this, I realize that doesn't work as I don't have a way >>>>>> to >>>>>> break $.tag down to reference the name and the value. >>>>>> >>>>>> I'd hate to have to do something like >>>>>> >>>>>> counters: [{"name":"z-scribe1r-b","value":80 >>>>>> },{"name":"SWEB10","value":67}] >>>>>> >>>>>> this mirrors the misuse of XML that gives it such a horrible reputation. >>>>>> But >>>>>> unless we introduce some new function to rsyslog to break things down, I >>>>>> don't see a better way. If we do need to do something like this, I sure >>>>>> would not want to make it the default JSON, which would result in two >>>>>> different formats. I hate the idea of starting to have different formats >>>>>> because of subtypes of data (what is someone wants the cee version of >>>>>> this >>>>>> for example, you start to have orthoginal format decisions) >>>>>> >>>>>> >>>>>> David Lang >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com/professional-services/ >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>>>> of >>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >>>>>> LIKE THAT. >>>>> >>>>> >>>>> >>>>> >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >>>> LIKE THAT. >>> >>> >>> >>> -- >>> Regards, >>> Janmejay >>> http://codehunk.wordpress.com >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >>> LIKE THAT. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. > > > > -- > Regards, > Janmejay > http://codehunk.wordpress.com -- Regards, Janmejay http://codehunk.wordpress.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
