Makes sense.
..."counters":["a":80,"b":67] won't work, I think you meant
..."counters": [{"a" : 80}, {"b": 67}].
So it boils down to foreach handling object or not.
Thoughts?
On Wed, Mar 23, 2016 at 1:18 AM, David Lang <[email protected]> wrote:
> On Wed, 23 Mar 2016, singh.janmejay wrote:
>
>> What about wrapDynamicObjects="on|off"? That is required regardless,
>> right? (if we want to preserve backward compatibility). Unless we
>> choose to change it anyway (which im fine with).
>
>
> I think the fact that data from the log could overwrite name or origin is a
> fatal flaw and the existing format should not be allowed. So I think we
> should break backwards compatibility here (if it was more established, I'd
> be much more reluctant to do so, but at this point, I think it's only early
> adopters who are using it)
>
> So if we are always going to push things down one level, the only question
> is the format
>
> {"name":"h","origin":"dynstats.bucket"},"counters":{"a":80,"b":67}}
>
> vs
>
> {"name":"h","origin":"dynstats.bucket"},"counters":["a":80,"b":67]}
>
> or the horrible
>
> {"name":"h","origin":"dynstats.bucket"},"counters":[{"name":"a","value":80},{"name":"b","value":67}]}
>
> I really want to avoid this third one. I'd say it's probably better to have
> to use an external script to process things than make the third one a
> standard :-)
>
> David Lang
>
>
>> @Rainer: what do you think about foreach handling object?
>>
>> On Wed, Mar 23, 2016 at 1:01 AM, David Lang <[email protected]> wrote:
>>>
>>> On Wed, 23 Mar 2016, singh.janmejay wrote:
>>>
>>>> Foreach can only work with arrays as of now. It can't work with
>>>> objects (key-value pairs). So [{name: ... value: ...}, {..},...] is
>>>> the only format that will work as of now.
>>>>
>>>> We can enhance foreach to work with objects.
>>>>
>>>> I can make a flag available at dyn-stats bucket level, which can
>>>> control serialization format, but that would really be a hack.
>>>>
>>>> From single-responsibility pattern pov, impstats should be the only
>>>> component that decides how to layout data for user to see.
>>>>
>>>> How about this:
>>>>
>>>> impstats(format="json" wrapDynamicObjects="on"...)?
>>>>
>>>> It defaults to off, which keeps backward compatibility.
>>>>
>>>> So what do you guys think about:
>>>> - wrapDynamicObjects="on|off"
>>>> - generating [{name: a, value: 10}, {...} ...] vs. {a: 10, ...}
>>>> (foreach will handle the former out of the box, but later is concise,
>>>> readable and light-weight in addition to being more json-y.
>>>> - enhancing foreach to work with {a: 10, b: 20...}
>>>
>>>
>>>
>>> If we can enhance foreach to work with the concise format, I would rather
>>> wait for it instead of introducing the wrapping version.
>>>
>>> I'm thinking that foreach walks through arrays, rather than mixing
>>> concepts,
>>> a foreachobject that gives us a name and contents for a {} list of
>>> objects
>>> may be the right thing to do?
>>>
>>> foreach just returns a single object while foreachobject needs to return
>>> the
>>> object and name.
>>>
>>> although, if we ever get the ability to address arrays directly, being
>>> able
>>> to look at the array position would be the equivalent of the name.
>>>
>>> David Lang
>>>
>>>
>>>
>>>> On Wed, Mar 23, 2016 at 12:26 AM, David Lang <[email protected]> wrote:
>>>>>
>>>>>
>>>>> On Tue, 22 Mar 2016, singh.janmejay wrote:
>>>>>
>>>>>> How about this: we support a new flag in impstats which allows
>>>>>> json-formatted stats-line to optionally use
>>>>>> encapsulated/wrapped-layout?
>>>>>>
>>>>>> impstats(format="json" ...) generates
>>>>>>
>>>>>>
>>>>>>
>>>>>> {"name":"msg_per_host","origin":"dynstats.bucket","z-scribe1r-b":80,"SWEB10":67}
>>>>>>
>>>>>> however,
>>>>>>
>>>>>> impstats(format="json/w" ...) generates {"header":
>>>>>> {"name":"msg_per_host","origin":"dynstats.bucket"}, "counters" :
>>>>>> {"z-scribe1r-b":80,"SWEB10":67}}
>>>>>>
>>>>>> This is relevant, the serilization format we use right now mixes
>>>>>> pre-defined keys with counter-names and it can affect regular static
>>>>>> counters too.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> with the existing pstats output, there is no ability for user-defined
>>>>> data
>>>>> to become a tag name, so there is no potential for ambiguity. but with
>>>>> dynastats, this is not a possibility, and the format we use should
>>>>> prevent
>>>>> problems.
>>>>>
>>>>> Also, just from a conceptual point of view, why should the bucket
>>>>> contents
>>>>> be at the same level as the bucket name?
>>>>>
>>>>> other than backwards compatibility, what advantage is there of the
>>>>> current
>>>>> version in JSON? the documentation uses the plain text equivalant,
>>>>> which
>>>>> is
>>>>> perfectly legitimate because there is an order to the line, and after
>>>>> you
>>>>> get past the name and origin, everything else on the line is name-value
>>>>> pairs of counters, again, no ambiguity.
>>>>>
>>>>> But with JSON, I don't believe that you can depend on tools maintaining
>>>>> (or
>>>>> even identifying) the order of the elements, and if you have multiple
>>>>> elements with the same name, it's implementation dependent as to which
>>>>> one
>>>>> will be seen.
>>>>>
>>>>> So purely from a correctness and defensive programming point of view, I
>>>>> think the current JSON serialization should be changed, with the old
>>>>> format
>>>>> no longer being an option.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> As to the details of the new format
>>>>>
>>>>> What I'm wanting to do with the counters is something like
>>>>>
>>>>> if $!origin == "dynstats.bucket" then {
>>>>> foreach $.tag $!counters {
>>>>> /var/log/stats;format
>>>>> }
>>>>> }
>>>>>
>>>>> to output one line per counter.
>>>>>
>>>>> I'm very flexible in how to do this, but I would much rather be able to
>>>>> do
>>>>> this inside rsyslog than have to serialize things to an external
>>>>> script,
>>>>> have it parse the json and process it.
>>>>>
>>>>> my initial thinking was just do
>>>>>
>>>>> counters: [ "z-scribe1r-b":80,"SWEB10":67 ]
>>>>>
>>>>> but as I'm typing this, I realize that doesn't work as I don't have a
>>>>> way
>>>>> to
>>>>> break $.tag down to reference the name and the value.
>>>>>
>>>>> I'd hate to have to do something like
>>>>>
>>>>> counters: [{"name":"z-scribe1r-b","value":80
>>>>> },{"name":"SWEB10","value":67}]
>>>>>
>>>>> this mirrors the misuse of XML that gives it such a horrible
>>>>> reputation.
>>>>> But
>>>>> unless we introduce some new function to rsyslog to break things down,
>>>>> I
>>>>> don't see a better way. If we do need to do something like this, I sure
>>>>> would not want to make it the default JSON, which would result in two
>>>>> different formats. I hate the idea of starting to have different
>>>>> formats
>>>>> because of subtypes of data (what is someone wants the cee version of
>>>>> this
>>>>> for example, you start to have orthoginal format decisions)
>>>>>
>>>>>
>>>>> David Lang
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>> myriad
>>>>> of
>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>>> DON'T
>>>>> LIKE THAT.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of
>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>>> LIKE THAT.
>>
>>
>>
>>
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
--
Regards,
Janmejay
http://codehunk.wordpress.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
