To me this is under the same umbrella of features (loop over
collection-style data). That data comes in a few flavors, and we are
doing it in small increments instead of doing it in one-shot and in a
way it does look like feature-creep. But I think some variant (if not
this statement in current form) will be required.
About supporting it in the long term, it makes sense to kill features
when users no longer defend it (by defending I mean someone should
come forward to maintain it, fix it, re-design it etc). Basically, its
a move towards encouraging the cycle of trying ideas and killing them
if they have no takers.
On Wed, Mar 23, 2016 at 5:30 PM, Rainer Gerhards
<rgerha...@hq.adiscon.com> wrote:
> 2016-03-23 12:42 GMT+01:00 singh.janmejay <singh.janme...@gmail.com>:
>> From variable-management pov, loop as a construct will be required as
>> long we have array, associative-array etc.
>>
>> But logs do have multi-valued fields, they have key-value pairs etc right?
>>
>> So regardless of the change in variable-system,
>> array/associative-array data-type will continue to exist. Which in
>> turn requires ways to work with that data-type.
>>
>> Doesn't it?
>
> yeah, but... I have a strong feeling of feature creep here.
> Originally, the config should not support any looping at all. Then we
> had arrays and said "well, it would be nice if we could process them".
> Now we have other types of objects and again "it would be nice if we
> could process them". Next we probably say "well, a for(i = 0 ; i< 10;
> i++) loop would be nice"...
>
> You see what I mean.
>
> Anyhow, maybe we need to go towards full-blown scripting capabilities.
> Not sure the engine is really up to that, but... Given the fact that I
> didn't strongly oppose the current loops system, I think you have a
> good point if the extension can easily be done without adding extra
> complexity. Still I am a bit nervous about the long-term implications,
> but let's see how things evolve.
>
> Rainer
>>
>> On Wed, Mar 23, 2016 at 4:58 PM, singh.janmejay
>> <singh.janme...@gmail.com> wrote:
>>> In-code footprint of foreach enhancement (to handle object in addition
>>> to array) will be fairly small (I am thinking under 40 lines?).
>>> Majority of addition will be in form of tests (unless we choose to
>>> change the rainerscript signature of that statement).
>>>
>>> Alternatively, from data-traversal functional-completeness pov, the
>>> same thing can be achieved without touching foreach impl, in a
>>> slightly high-overhead way.
>>>
>>> We can create a new rainerscript fn make_list(<object>) =>
>>> [kv-tuples...] (or mklist).
>>>
>>> Eg.
>>> {"foo" : "bar", "baz" : "quux"} becomes [{"key": "foo", "value":
>>> "bar"}, {"key": "baz", "value": "quux"}]
>>>
>>> Clearly it will have inferior performance, but it'll make foreach work
>>> for all object use-cases.
>>>
>>> I really feel in terms of complexity though, this (transform approach)
>>> is worse than first-class support (purely because of it using a
>>> round-about way of working).
>>>
>>>
>>> On Wed, Mar 23, 2016 at 3:01 PM, Rainer Gerhards
>>> <rgerha...@hq.adiscon.com> wrote:
>>>> 2016-03-22 20:34 GMT+01:00 singh.janmejay <singh.janme...@gmail.com>:
>>>>> What about wrapDynamicObjects="on|off"? That is required regardless,
>>>>> right? (if we want to preserve backward compatibility). Unless we
>>>>> choose to change it anyway (which im fine with).
>>>>>
>>>>> @Rainer: what do you think about foreach handling object?
>>>>
>>>> so far I could only manage to have a glimpse at the conversation. I
>>>> admit that I am really concerned about all the extra complexity we
>>>> introduce. And do so for what I think is a border-use case at best.
>>>> There is a lot of work that should really be done in regard to
>>>> variables and performance and I am unsure if this is the right time to
>>>> do large extensions...
>>>>
>>>> I was happy with the dynstats as it looked like a very contained
>>>> solution that did not affect much else. But now it looks we need to
>>>> touch a big deal of code ... code that's not really ready for that...
>>>>
>>>> Maybe I get a different view when I find time to read all this more
>>>> in-depth....
>>>>
>>>> Rainer
>>>>>
>>>>> On Wed, Mar 23, 2016 at 1:01 AM, David Lang <da...@lang.hm> wrote:
>>>>>> On Wed, 23 Mar 2016, singh.janmejay wrote:
>>>>>>
>>>>>>> Foreach can only work with arrays as of now. It can't work with
>>>>>>> objects (key-value pairs). So [{name: ... value: ...}, {..},...] is
>>>>>>> the only format that will work as of now.
>>>>>>>
>>>>>>> We can enhance foreach to work with objects.
>>>>>>>
>>>>>>> I can make a flag available at dyn-stats bucket level, which can
>>>>>>> control serialization format, but that would really be a hack.
>>>>>>>
>>>>>>> From single-responsibility pattern pov, impstats should be the only
>>>>>>> component that decides how to layout data for user to see.
>>>>>>>
>>>>>>> How about this:
>>>>>>>
>>>>>>> impstats(format="json" wrapDynamicObjects="on"...)?
>>>>>>>
>>>>>>> It defaults to off, which keeps backward compatibility.
>>>>>>>
>>>>>>> So what do you guys think about:
>>>>>>> - wrapDynamicObjects="on|off"
>>>>>>> - generating [{name: a, value: 10}, {...} ...] vs. {a: 10, ...}
>>>>>>> (foreach will handle the former out of the box, but later is concise,
>>>>>>> readable and light-weight in addition to being more json-y.
>>>>>>> - enhancing foreach to work with {a: 10, b: 20...}
>>>>>>
>>>>>>
>>>>>> If we can enhance foreach to work with the concise format, I would rather
>>>>>> wait for it instead of introducing the wrapping version.
>>>>>>
>>>>>> I'm thinking that foreach walks through arrays, rather than mixing
>>>>>> concepts,
>>>>>> a foreachobject that gives us a name and contents for a {} list of
>>>>>> objects
>>>>>> may be the right thing to do?
>>>>>>
>>>>>> foreach just returns a single object while foreachobject needs to return
>>>>>> the
>>>>>> object and name.
>>>>>>
>>>>>> although, if we ever get the ability to address arrays directly, being
>>>>>> able
>>>>>> to look at the array position would be the equivalent of the name.
>>>>>>
>>>>>> David Lang
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On Wed, Mar 23, 2016 at 12:26 AM, David Lang <da...@lang.hm> wrote:
>>>>>>>>
>>>>>>>> On Tue, 22 Mar 2016, singh.janmejay wrote:
>>>>>>>>
>>>>>>>>> How about this: we support a new flag in impstats which allows
>>>>>>>>> json-formatted stats-line to optionally use
>>>>>>>>> encapsulated/wrapped-layout?
>>>>>>>>>
>>>>>>>>> impstats(format="json" ...) generates
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> {"name":"msg_per_host","origin":"dynstats.bucket","z-scribe1r-b":80,"SWEB10":67}
>>>>>>>>>
>>>>>>>>> however,
>>>>>>>>>
>>>>>>>>> impstats(format="json/w" ...) generates {"header":
>>>>>>>>> {"name":"msg_per_host","origin":"dynstats.bucket"}, "counters" :
>>>>>>>>> {"z-scribe1r-b":80,"SWEB10":67}}
>>>>>>>>>
>>>>>>>>> This is relevant, the serilization format we use right now mixes
>>>>>>>>> pre-defined keys with counter-names and it can affect regular static
>>>>>>>>> counters too.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> with the existing pstats output, there is no ability for user-defined
>>>>>>>> data
>>>>>>>> to become a tag name, so there is no potential for ambiguity. but with
>>>>>>>> dynastats, this is not a possibility, and the format we use should
>>>>>>>> prevent
>>>>>>>> problems.
>>>>>>>>
>>>>>>>> Also, just from a conceptual point of view, why should the bucket
>>>>>>>> contents
>>>>>>>> be at the same level as the bucket name?
>>>>>>>>
>>>>>>>> other than backwards compatibility, what advantage is there of the
>>>>>>>> current
>>>>>>>> version in JSON? the documentation uses the plain text equivalant,
>>>>>>>> which
>>>>>>>> is
>>>>>>>> perfectly legitimate because there is an order to the line, and after
>>>>>>>> you
>>>>>>>> get past the name and origin, everything else on the line is name-value
>>>>>>>> pairs of counters, again, no ambiguity.
>>>>>>>>
>>>>>>>> But with JSON, I don't believe that you can depend on tools maintaining
>>>>>>>> (or
>>>>>>>> even identifying) the order of the elements, and if you have multiple
>>>>>>>> elements with the same name, it's implementation dependent as to which
>>>>>>>> one
>>>>>>>> will be seen.
>>>>>>>>
>>>>>>>> So purely from a correctness and defensive programming point of view, I
>>>>>>>> think the current JSON serialization should be changed, with the old
>>>>>>>> format
>>>>>>>> no longer being an option.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> As to the details of the new format
>>>>>>>>
>>>>>>>> What I'm wanting to do with the counters is something like
>>>>>>>>
>>>>>>>> if $!origin == "dynstats.bucket" then {
>>>>>>>> foreach $.tag $!counters {
>>>>>>>> /var/log/stats;format
>>>>>>>> }
>>>>>>>> }
>>>>>>>>
>>>>>>>> to output one line per counter.
>>>>>>>>
>>>>>>>> I'm very flexible in how to do this, but I would much rather be able to
>>>>>>>> do
>>>>>>>> this inside rsyslog than have to serialize things to an external
>>>>>>>> script,
>>>>>>>> have it parse the json and process it.
>>>>>>>>
>>>>>>>> my initial thinking was just do
>>>>>>>>
>>>>>>>> counters: [ "z-scribe1r-b":80,"SWEB10":67 ]
>>>>>>>>
>>>>>>>> but as I'm typing this, I realize that doesn't work as I don't have a
>>>>>>>> way
>>>>>>>> to
>>>>>>>> break $.tag down to reference the name and the value.
>>>>>>>>
>>>>>>>> I'd hate to have to do something like
>>>>>>>>
>>>>>>>> counters: [{"name":"z-scribe1r-b","value":80
>>>>>>>> },{"name":"SWEB10","value":67}]
>>>>>>>>
>>>>>>>> this mirrors the misuse of XML that gives it such a horrible
>>>>>>>> reputation.
>>>>>>>> But
>>>>>>>> unless we introduce some new function to rsyslog to break things down,
>>>>>>>> I
>>>>>>>> don't see a better way. If we do need to do something like this, I sure
>>>>>>>> would not want to make it the default JSON, which would result in two
>>>>>>>> different formats. I hate the idea of starting to have different
>>>>>>>> formats
>>>>>>>> because of subtypes of data (what is someone wants the cee version of
>>>>>>>> this
>>>>>>>> for example, you start to have orthoginal format decisions)
>>>>>>>>
>>>>>>>>
>>>>>>>> David Lang
>>>>>>>> _______________________________________________
>>>>>>>> rsyslog mailing list
>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>> http://www.rsyslog.com/professional-services/
>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>>>>> myriad
>>>>>>>> of
>>>>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>>>>>> DON'T
>>>>>>>> LIKE THAT.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>> http://www.rsyslog.com/professional-services/
>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>>>>> of
>>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>>>>>> LIKE THAT.
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Janmejay
>>>>> http://codehunk.wordpress.com
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>>> DON'T LIKE THAT.
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>> DON'T LIKE THAT.
>>>
>>>
>>>
>>> --
>>> Regards,
>>> Janmejay
>>> http://codehunk.wordpress.com
>>
>>
>>
>> --
>> Regards,
>> Janmejay
>> http://codehunk.wordpress.com
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>> LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
--
Regards,
Janmejay
http://codehunk.wordpress.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
