Makes sense. So final call on foreach? Go or no-go?
On Mar 25, 2016 2:38 AM, "David Lang" <[email protected]> wrote:
> Just a note, the dynastats output line needs this as well as the bucket
> line.
>
> This isn't remote-input, but it needs the same ability to iterate over
> things.
>
> example:
>
>
> {"name":"global","origin":"dynstats","message_framing.ops_overflow":0,"message_framing.new_metric_add":5,"message_framing.no_metric":0,"message_framing.metrics_purged":0,"message_framing.ops_ignored":0,"message_framing.purge_triggered":0,"msgs_per_host.ops_overflow":0,"msgs_per_host.new_metric_add":0,"msgs_per_host.no_metric":0,"msgs_per_host.metrics_purged":0,"msgs_per_host.ops_ignored":0,"msgs_per_host.purge_triggered":0,"msgs_per_edge_relay.ops_overflow":0,"msgs_per_edge_relay.new_metric_add":0,"msgs_per_edge_relay.no_metric":0,"msgs_per_edge_relay.metrics_purged":0,"msgs_per_edge_relay.ops_ignored":0,"msgs_per_edge_relay.purge_triggered":0,"msgs_per_core_relay.ops_overflow":0,"msgs_per_core_relay.new_metric_add":0,"msgs_per_core_relay.no_metric":0,"msgs_per_core_relay.metrics_purged":0,"msgs_per_core_relay.ops_ignored":0,"msgs_per_core_relay.purge_triggered":0,"msgs_per_program.ops_overflow":0,"msgs_per_program.new_metric_add":0,"msgs_per_program.no_metric":0,"msgs_per_
p!
>
> rogram.metrics_purged":0,"msgs_per_program.ops_ignored":0,"msgs_per_program.purge_triggered":0,"msgs_per_tag.ops_overflow":0,"msgs_per_tag.new_metric_add":0,"msgs_per_tag.no_metric":0,"msgs_per_tag.metrics_purged":0,"msgs_per_tag.ops_ignored":0,"msgs_per_tag.purge_triggered":0}
>
> David Lang
>
> On Wed, 23 Mar 2016, singh.janmejay wrote:
>
> Date: Wed, 23 Mar 2016 09:10:47 +0530
>> From: singh.janmejay <[email protected]>
>> Reply-To: rsyslog-users <[email protected]>
>> To: rsyslog-users <[email protected]>
>> Subject: Re: [rsyslog] dynastats JSON output need work
>>
>> Agree, if most of us like it, I can implement foreach for objects. As of
>> now I use omprog to call a script which uses jq and awk to make it
>> opentsdb
>> and redis friendly. I could have used omredis if foreach worked for
>> objects.
>>
>> Let us conclude.
>>
>> @Rainer any thoughts on foreach for object?
>> On Mar 23, 2016 1:56 AM, "David Lang" <[email protected]> wrote:
>>
>> On Wed, 23 Mar 2016, singh.janmejay wrote:
>>>
>>> Makes sense.
>>>
>>>>
>>>> ..."counters":["a":80,"b":67] won't work, I think you meant
>>>> ..."counters": [{"a" : 80}, {"b": 67}].
>>>>
>>>>
>>> ahh, showing my lack of knowledge of JSON :-) talking things through:
>>>
>>> so our choices are:
>>>
>>> "counters": [{"a":80},{"b":67}]
>>> vs
>>> "counters": {"a":80,"b":67}
>>>
>>> running these through jq gives
>>>
>>> {
>>> "counters": [
>>> {
>>> "a": 80
>>> },
>>> {
>>> "b": 67
>>> }
>>> ]
>>> }
>>>
>>> vs
>>>
>>> {
>>> "counters": {
>>> "a": 80,
>>> "b": 67
>>> }
>>> }
>>>
>>> or to reference a
>>> .counters[0].a
>>> vs
>>> .counters.a
>>>
>>> The latter seems to clearly be the cleaner and more logical one to me.
>>>
>>> Does this seem sane?
>>>
>>> But to use it, we need a variation of foreach that gives the object name
>>> as well as the object contents.
>>>
>>> David Lang
>>>
>>> So it boils down to foreach handling object or not.
>>>
>>>>
>>>> Thoughts?
>>>>
>>>> On Wed, Mar 23, 2016 at 1:18 AM, David Lang <[email protected]> wrote:
>>>>
>>>> On Wed, 23 Mar 2016, singh.janmejay wrote:
>>>>>
>>>>> What about wrapDynamicObjects="on|off"? That is required regardless,
>>>>>
>>>>>> right? (if we want to preserve backward compatibility). Unless we
>>>>>> choose to change it anyway (which im fine with).
>>>>>>
>>>>>>
>>>>>
>>>>> I think the fact that data from the log could overwrite name or origin
>>>>> is a
>>>>> fatal flaw and the existing format should not be allowed. So I think we
>>>>> should break backwards compatibility here (if it was more established,
>>>>> I'd
>>>>> be much more reluctant to do so, but at this point, I think it's only
>>>>> early
>>>>> adopters who are using it)
>>>>>
>>>>> So if we are always going to push things down one level, the only
>>>>> question
>>>>> is the format
>>>>>
>>>>> {"name":"h","origin":"dynstats.bucket"},"counters":{"a":80,"b":67}}
>>>>>
>>>>> vs
>>>>>
>>>>> {"name":"h","origin":"dynstats.bucket"},"counters":["a":80,"b":67]}
>>>>>
>>>>> or the horrible
>>>>>
>>>>>
>>>>>
>>>>> {"name":"h","origin":"dynstats.bucket"},"counters":[{"name":"a","value":80},{"name":"b","value":67}]}
>>>>>
>>>>> I really want to avoid this third one. I'd say it's probably better to
>>>>> have
>>>>> to use an external script to process things than make the third one a
>>>>> standard :-)
>>>>>
>>>>> David Lang
>>>>>
>>>>>
>>>>> @Rainer: what do you think about foreach handling object?
>>>>>
>>>>>>
>>>>>> On Wed, Mar 23, 2016 at 1:01 AM, David Lang <[email protected]> wrote:
>>>>>>
>>>>>>
>>>>>>> On Wed, 23 Mar 2016, singh.janmejay wrote:
>>>>>>>
>>>>>>> Foreach can only work with arrays as of now. It can't work with
>>>>>>>
>>>>>>>> objects (key-value pairs). So [{name: ... value: ...}, {..},...] is
>>>>>>>> the only format that will work as of now.
>>>>>>>>
>>>>>>>> We can enhance foreach to work with objects.
>>>>>>>>
>>>>>>>> I can make a flag available at dyn-stats bucket level, which can
>>>>>>>> control serialization format, but that would really be a hack.
>>>>>>>>
>>>>>>>> From single-responsibility pattern pov, impstats should be the only
>>>>>>>> component that decides how to layout data for user to see.
>>>>>>>>
>>>>>>>> How about this:
>>>>>>>>
>>>>>>>> impstats(format="json" wrapDynamicObjects="on"...)?
>>>>>>>>
>>>>>>>> It defaults to off, which keeps backward compatibility.
>>>>>>>>
>>>>>>>> So what do you guys think about:
>>>>>>>> - wrapDynamicObjects="on|off"
>>>>>>>> - generating [{name: a, value: 10}, {...} ...] vs. {a: 10, ...}
>>>>>>>> (foreach will handle the former out of the box, but later is
>>>>>>>> concise,
>>>>>>>> readable and light-weight in addition to being more json-y.
>>>>>>>> - enhancing foreach to work with {a: 10, b: 20...}
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> If we can enhance foreach to work with the concise format, I would
>>>>>>> rather
>>>>>>> wait for it instead of introducing the wrapping version.
>>>>>>>
>>>>>>> I'm thinking that foreach walks through arrays, rather than mixing
>>>>>>> concepts,
>>>>>>> a foreachobject that gives us a name and contents for a {} list of
>>>>>>> objects
>>>>>>> may be the right thing to do?
>>>>>>>
>>>>>>> foreach just returns a single object while foreachobject needs to
>>>>>>> return
>>>>>>> the
>>>>>>> object and name.
>>>>>>>
>>>>>>> although, if we ever get the ability to address arrays directly,
>>>>>>> being
>>>>>>> able
>>>>>>> to look at the array position would be the equivalent of the name.
>>>>>>>
>>>>>>> David Lang
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Mar 23, 2016 at 12:26 AM, David Lang <[email protected]> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, 22 Mar 2016, singh.janmejay wrote:
>>>>>>>>>
>>>>>>>>> How about this: we support a new flag in impstats which allows
>>>>>>>>>
>>>>>>>>>> json-formatted stats-line to optionally use
>>>>>>>>>> encapsulated/wrapped-layout?
>>>>>>>>>>
>>>>>>>>>> impstats(format="json" ...) generates
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> {"name":"msg_per_host","origin":"dynstats.bucket","z-scribe1r-b":80,"SWEB10":67}
>>>>>>>>>>
>>>>>>>>>> however,
>>>>>>>>>>
>>>>>>>>>> impstats(format="json/w" ...) generates {"header":
>>>>>>>>>> {"name":"msg_per_host","origin":"dynstats.bucket"}, "counters" :
>>>>>>>>>> {"z-scribe1r-b":80,"SWEB10":67}}
>>>>>>>>>>
>>>>>>>>>> This is relevant, the serilization format we use right now mixes
>>>>>>>>>> pre-defined keys with counter-names and it can affect regular
>>>>>>>>>> static
>>>>>>>>>> counters too.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> with the existing pstats output, there is no ability for
>>>>>>>>> user-defined
>>>>>>>>> data
>>>>>>>>> to become a tag name, so there is no potential for ambiguity. but
>>>>>>>>> with
>>>>>>>>> dynastats, this is not a possibility, and the format we use should
>>>>>>>>> prevent
>>>>>>>>> problems.
>>>>>>>>>
>>>>>>>>> Also, just from a conceptual point of view, why should the bucket
>>>>>>>>> contents
>>>>>>>>> be at the same level as the bucket name?
>>>>>>>>>
>>>>>>>>> other than backwards compatibility, what advantage is there of the
>>>>>>>>> current
>>>>>>>>> version in JSON? the documentation uses the plain text equivalant,
>>>>>>>>> which
>>>>>>>>> is
>>>>>>>>> perfectly legitimate because there is an order to the line, and
>>>>>>>>> after
>>>>>>>>> you
>>>>>>>>> get past the name and origin, everything else on the line is
>>>>>>>>> name-value
>>>>>>>>> pairs of counters, again, no ambiguity.
>>>>>>>>>
>>>>>>>>> But with JSON, I don't believe that you can depend on tools
>>>>>>>>> maintaining
>>>>>>>>> (or
>>>>>>>>> even identifying) the order of the elements, and if you have
>>>>>>>>> multiple
>>>>>>>>> elements with the same name, it's implementation dependent as to
>>>>>>>>> which
>>>>>>>>> one
>>>>>>>>> will be seen.
>>>>>>>>>
>>>>>>>>> So purely from a correctness and defensive programming point of
>>>>>>>>> view, I
>>>>>>>>> think the current JSON serialization should be changed, with the
>>>>>>>>> old
>>>>>>>>> format
>>>>>>>>> no longer being an option.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> As to the details of the new format
>>>>>>>>>
>>>>>>>>> What I'm wanting to do with the counters is something like
>>>>>>>>>
>>>>>>>>> if $!origin == "dynstats.bucket" then {
>>>>>>>>> foreach $.tag $!counters {
>>>>>>>>> /var/log/stats;format
>>>>>>>>> }
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> to output one line per counter.
>>>>>>>>>
>>>>>>>>> I'm very flexible in how to do this, but I would much rather be
>>>>>>>>> able
>>>>>>>>> to
>>>>>>>>> do
>>>>>>>>> this inside rsyslog than have to serialize things to an external
>>>>>>>>> script,
>>>>>>>>> have it parse the json and process it.
>>>>>>>>>
>>>>>>>>> my initial thinking was just do
>>>>>>>>>
>>>>>>>>> counters: [ "z-scribe1r-b":80,"SWEB10":67 ]
>>>>>>>>>
>>>>>>>>> but as I'm typing this, I realize that doesn't work as I don't
>>>>>>>>> have a
>>>>>>>>> way
>>>>>>>>> to
>>>>>>>>> break $.tag down to reference the name and the value.
>>>>>>>>>
>>>>>>>>> I'd hate to have to do something like
>>>>>>>>>
>>>>>>>>> counters: [{"name":"z-scribe1r-b","value":80
>>>>>>>>> },{"name":"SWEB10","value":67}]
>>>>>>>>>
>>>>>>>>> this mirrors the misuse of XML that gives it such a horrible
>>>>>>>>> reputation.
>>>>>>>>> But
>>>>>>>>> unless we introduce some new function to rsyslog to break things
>>>>>>>>> down,
>>>>>>>>> I
>>>>>>>>> don't see a better way. If we do need to do something like this, I
>>>>>>>>> sure
>>>>>>>>> would not want to make it the default JSON, which would result in
>>>>>>>>> two
>>>>>>>>> different formats. I hate the idea of starting to have different
>>>>>>>>> formats
>>>>>>>>> because of subtypes of data (what is someone wants the cee version
>>>>>>>>> of
>>>>>>>>> this
>>>>>>>>> for example, you start to have orthoginal format decisions)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> David Lang
>>>>>>>>> _______________________________________________
>>>>>>>>> rsyslog mailing list
>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>>> http://www.rsyslog.com/professional-services/
>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>>>>>> myriad
>>>>>>>>> of
>>>>>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>>>>>>> DON'T
>>>>>>>>> LIKE THAT.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>>
>>>>>>> rsyslog mailing list
>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>> http://www.rsyslog.com/professional-services/
>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>>>> myriad
>>>>>>> of
>>>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>>>>> DON'T
>>>>>>> LIKE THAT.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>>
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>> myriad
>>>>> of
>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>>> DON'T
>>>>> LIKE THAT.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>>
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>>
>>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
