As I understand Rainer, go . David Lang
On Fri, 25 Mar 2016, singh.janmejay wrote:
Makes sense. So final call on foreach? Go or no-go? On Mar 25, 2016 2:38 AM, "David Lang" <[email protected]> wrote:Just a note, the dynastats output line needs this as well as the bucket line. This isn't remote-input, but it needs the same ability to iterate over things. example: {"name":"global","origin":"dynstats","message_framing.ops_overflow":0,"message_framing.new_metric_add":5,"message_framing.no_metric":0,"message_framing.metrics_purged":0,"message_framing.ops_ignored":0,"message_framing.purge_triggered":0,"msgs_per_host.ops_overflow":0,"msgs_per_host.new_metric_add":0,"msgs_per_host.no_metric":0,"msgs_per_host.metrics_purged":0,"msgs_per_host.ops_ignored":0,"msgs_per_host.purge_triggered":0,"msgs_per_edge_relay.ops_overflow":0,"msgs_per_edge_relay.new_metric_add":0,"msgs_per_edge_relay.no_metric":0,"msgs_per_edge_relay.metrics_purged":0,"msgs_per_edge_relay.ops_ignored":0,"msgs_per_edge_relay.purge_triggered":0,"msgs_per_core_relay.ops_overflow":0,"msgs_per_core_relay.new_metric_add":0,"msgs_per_core_relay.no_metric":0,"msgs_per_core_relay.metrics_purged":0,"msgs_per_core_relay.ops_ignored":0,"msgs_per_core_relay.purge_triggered":0,"msgs_per_program.ops_overflow":0,"msgs_per_program.new_metric_add":0,"msgs_per_program.no_metric":0,"msgs_pe!
r_
p!rogram.metrics_purged":0,"msgs_per_program.ops_ignored":0,"msgs_per_program.purge_triggered":0,"msgs_per_tag.ops_overflow":0,"msgs_per_tag.new_metric_add":0,"msgs_per_tag.no_metric":0,"msgs_per_tag.metrics_purged":0,"msgs_per_tag.ops_ignored":0,"msgs_per_tag.purge_triggered":0} David Lang On Wed, 23 Mar 2016, singh.janmejay wrote: Date: Wed, 23 Mar 2016 09:10:47 +0530From: singh.janmejay <[email protected]> Reply-To: rsyslog-users <[email protected]> To: rsyslog-users <[email protected]> Subject: Re: [rsyslog] dynastats JSON output need work Agree, if most of us like it, I can implement foreach for objects. As of now I use omprog to call a script which uses jq and awk to make it opentsdb and redis friendly. I could have used omredis if foreach worked for objects. Let us conclude. @Rainer any thoughts on foreach for object? On Mar 23, 2016 1:56 AM, "David Lang" <[email protected]> wrote: On Wed, 23 Mar 2016, singh.janmejay wrote:Makes sense...."counters":["a":80,"b":67] won't work, I think you meant ..."counters": [{"a" : 80}, {"b": 67}].ahh, showing my lack of knowledge of JSON :-) talking things through: so our choices are: "counters": [{"a":80},{"b":67}] vs "counters": {"a":80,"b":67} running these through jq gives { "counters": [ { "a": 80 }, { "b": 67 } ] } vs { "counters": { "a": 80, "b": 67 } } or to reference a .counters[0].a vs .counters.a The latter seems to clearly be the cleaner and more logical one to me. Does this seem sane? But to use it, we need a variation of foreach that gives the object name as well as the object contents. David Lang So it boils down to foreach handling object or not.Thoughts? On Wed, Mar 23, 2016 at 1:18 AM, David Lang <[email protected]> wrote: On Wed, 23 Mar 2016, singh.janmejay wrote:What about wrapDynamicObjects="on|off"? That is required regardless,right? (if we want to preserve backward compatibility). Unless we choose to change it anyway (which im fine with).I think the fact that data from the log could overwrite name or origin is a fatal flaw and the existing format should not be allowed. So I think we should break backwards compatibility here (if it was more established, I'd be much more reluctant to do so, but at this point, I think it's only early adopters who are using it) So if we are always going to push things down one level, the only question is the format {"name":"h","origin":"dynstats.bucket"},"counters":{"a":80,"b":67}} vs {"name":"h","origin":"dynstats.bucket"},"counters":["a":80,"b":67]} or the horrible {"name":"h","origin":"dynstats.bucket"},"counters":[{"name":"a","value":80},{"name":"b","value":67}]} I really want to avoid this third one. I'd say it's probably better to have to use an external script to process things than make the third one a standard :-) David Lang @Rainer: what do you think about foreach handling object?On Wed, Mar 23, 2016 at 1:01 AM, David Lang <[email protected]> wrote:On Wed, 23 Mar 2016, singh.janmejay wrote: Foreach can only work with arrays as of now. It can't work withobjects (key-value pairs). So [{name: ... value: ...}, {..},...] is the only format that will work as of now. We can enhance foreach to work with objects. I can make a flag available at dyn-stats bucket level, which can control serialization format, but that would really be a hack. From single-responsibility pattern pov, impstats should be the only component that decides how to layout data for user to see. How about this: impstats(format="json" wrapDynamicObjects="on"...)? It defaults to off, which keeps backward compatibility. So what do you guys think about: - wrapDynamicObjects="on|off" - generating [{name: a, value: 10}, {...} ...] vs. {a: 10, ...} (foreach will handle the former out of the box, but later is concise, readable and light-weight in addition to being more json-y. - enhancing foreach to work with {a: 10, b: 20...}If we can enhance foreach to work with the concise format, I would rather wait for it instead of introducing the wrapping version. I'm thinking that foreach walks through arrays, rather than mixing concepts, a foreachobject that gives us a name and contents for a {} list of objects may be the right thing to do? foreach just returns a single object while foreachobject needs to return the object and name. although, if we ever get the ability to address arrays directly, being able to look at the array position would be the equivalent of the name. David Lang On Wed, Mar 23, 2016 at 12:26 AM, David Lang <[email protected]> wrote:On Tue, 22 Mar 2016, singh.janmejay wrote: How about this: we support a new flag in impstats which allowsjson-formatted stats-line to optionally use encapsulated/wrapped-layout? impstats(format="json" ...) generates {"name":"msg_per_host","origin":"dynstats.bucket","z-scribe1r-b":80,"SWEB10":67} however, impstats(format="json/w" ...) generates {"header": {"name":"msg_per_host","origin":"dynstats.bucket"}, "counters" : {"z-scribe1r-b":80,"SWEB10":67}} This is relevant, the serilization format we use right now mixes pre-defined keys with counter-names and it can affect regular static counters too.with the existing pstats output, there is no ability for user-defined data to become a tag name, so there is no potential for ambiguity. but with dynastats, this is not a possibility, and the format we use should prevent problems. Also, just from a conceptual point of view, why should the bucket contents be at the same level as the bucket name? other than backwards compatibility, what advantage is there of the current version in JSON? the documentation uses the plain text equivalant, which is perfectly legitimate because there is an order to the line, and after you get past the name and origin, everything else on the line is name-value pairs of counters, again, no ambiguity. But with JSON, I don't believe that you can depend on tools maintaining (or even identifying) the order of the elements, and if you have multiple elements with the same name, it's implementation dependent as to which one will be seen. So purely from a correctness and defensive programming point of view, I think the current JSON serialization should be changed, with the old format no longer being an option. As to the details of the new format What I'm wanting to do with the counters is something like if $!origin == "dynstats.bucket" then { foreach $.tag $!counters { /var/log/stats;format } } to output one line per counter. I'm very flexible in how to do this, but I would much rather be able to do this inside rsyslog than have to serialize things to an external script, have it parse the json and process it. my initial thinking was just do counters: [ "z-scribe1r-b":80,"SWEB10":67 ] but as I'm typing this, I realize that doesn't work as I don't have a way to break $.tag down to reference the name and the value. I'd hate to have to do something like counters: [{"name":"z-scribe1r-b","value":80 },{"name":"SWEB10","value":67}] this mirrors the misuse of XML that gives it such a horrible reputation. But unless we introduce some new function to rsyslog to break things down, I don't see a better way. If we do need to do something like this, I sure would not want to make it the default JSON, which would result in two different formats. I hate the idea of starting to have different formats because of subtypes of data (what is someone wants the cee version of this for example, you start to have orthoginal format decisions) David Lang _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT._______________________________________________rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT._______________________________________________rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT._______________________________________________rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT._______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT._______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
