On Wed, 23 Mar 2016, Rainer Gerhards wrote:
2016-03-22 20:34 GMT+01:00 singh.janmejay <[email protected]>:
What about wrapDynamicObjects="on|off"? That is required regardless,
right? (if we want to preserve backward compatibility). Unless we
choose to change it anyway (which im fine with).
@Rainer: what do you think about foreach handling object?
so far I could only manage to have a glimpse at the conversation. I
admit that I am really concerned about all the extra complexity we
introduce. And do so for what I think is a border-use case at best.
There is a lot of work that should really be done in regard to
variables and performance and I am unsure if this is the right time to
do large extensions...
I was happy with the dynstats as it looked like a very contained
solution that did not affect much else. But now it looks we need to
touch a big deal of code ... code that's not really ready for that...
We are talking about two changes.
1. format change to make the bucket elements be at a different level than the
name/origin parameters to avoid conflicts.
2. an enhancement to foreach to allow for processing multiple items on a list,
not just elements in an array.
I think we need to do #1 right away, even if we don't do #2, just to minimize
the window that rsyslog is generating the format that could be abused.
I think #2 is still going to be contained to a corner of the codebase, the
foreach code. So it shouldn't have any effect on other areas.
David Lang
Maybe I get a different view when I find time to read all this more in-depth....
Rainer
On Wed, Mar 23, 2016 at 1:01 AM, David Lang <[email protected]> wrote:
On Wed, 23 Mar 2016, singh.janmejay wrote:
Foreach can only work with arrays as of now. It can't work with
objects (key-value pairs). So [{name: ... value: ...}, {..},...] is
the only format that will work as of now.
We can enhance foreach to work with objects.
I can make a flag available at dyn-stats bucket level, which can
control serialization format, but that would really be a hack.
From single-responsibility pattern pov, impstats should be the only
component that decides how to layout data for user to see.
How about this:
impstats(format="json" wrapDynamicObjects="on"...)?
It defaults to off, which keeps backward compatibility.
So what do you guys think about:
- wrapDynamicObjects="on|off"
- generating [{name: a, value: 10}, {...} ...] vs. {a: 10, ...}
(foreach will handle the former out of the box, but later is concise,
readable and light-weight in addition to being more json-y.
- enhancing foreach to work with {a: 10, b: 20...}
If we can enhance foreach to work with the concise format, I would rather
wait for it instead of introducing the wrapping version.
I'm thinking that foreach walks through arrays, rather than mixing concepts,
a foreachobject that gives us a name and contents for a {} list of objects
may be the right thing to do?
foreach just returns a single object while foreachobject needs to return the
object and name.
although, if we ever get the ability to address arrays directly, being able
to look at the array position would be the equivalent of the name.
David Lang
On Wed, Mar 23, 2016 at 12:26 AM, David Lang <[email protected]> wrote:
On Tue, 22 Mar 2016, singh.janmejay wrote:
How about this: we support a new flag in impstats which allows
json-formatted stats-line to optionally use
encapsulated/wrapped-layout?
impstats(format="json" ...) generates
{"name":"msg_per_host","origin":"dynstats.bucket","z-scribe1r-b":80,"SWEB10":67}
however,
impstats(format="json/w" ...) generates {"header":
{"name":"msg_per_host","origin":"dynstats.bucket"}, "counters" :
{"z-scribe1r-b":80,"SWEB10":67}}
This is relevant, the serilization format we use right now mixes
pre-defined keys with counter-names and it can affect regular static
counters too.
with the existing pstats output, there is no ability for user-defined
data
to become a tag name, so there is no potential for ambiguity. but with
dynastats, this is not a possibility, and the format we use should
prevent
problems.
Also, just from a conceptual point of view, why should the bucket
contents
be at the same level as the bucket name?
other than backwards compatibility, what advantage is there of the
current
version in JSON? the documentation uses the plain text equivalant, which
is
perfectly legitimate because there is an order to the line, and after you
get past the name and origin, everything else on the line is name-value
pairs of counters, again, no ambiguity.
But with JSON, I don't believe that you can depend on tools maintaining
(or
even identifying) the order of the elements, and if you have multiple
elements with the same name, it's implementation dependent as to which
one
will be seen.
So purely from a correctness and defensive programming point of view, I
think the current JSON serialization should be changed, with the old
format
no longer being an option.
As to the details of the new format
What I'm wanting to do with the counters is something like
if $!origin == "dynstats.bucket" then {
foreach $.tag $!counters {
/var/log/stats;format
}
}
to output one line per counter.
I'm very flexible in how to do this, but I would much rather be able to
do
this inside rsyslog than have to serialize things to an external script,
have it parse the json and process it.
my initial thinking was just do
counters: [ "z-scribe1r-b":80,"SWEB10":67 ]
but as I'm typing this, I realize that doesn't work as I don't have a way
to
break $.tag down to reference the name and the value.
I'd hate to have to do something like
counters: [{"name":"z-scribe1r-b","value":80
},{"name":"SWEB10","value":67}]
this mirrors the misuse of XML that gives it such a horrible reputation.
But
unless we introduce some new function to rsyslog to break things down, I
don't see a better way. If we do need to do something like this, I sure
would not want to make it the default JSON, which would result in two
different formats. I hate the idea of starting to have different formats
because of subtypes of data (what is someone wants the cee version of
this
for example, you start to have orthoginal format decisions)
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
--
Regards,
Janmejay
http://codehunk.wordpress.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
