On Tue, 22 Oct 2013, Rainer Gerhards wrote:
On Tue, Oct 22, 2013 at 2:26 PM, David Lang <[email protected]> wrote:
so, what functions will mmglobal need?
not sure if you saw my message where I proposed a new global() function. I
already followed your earlier advice and searched for a non-action type of
solution (as you said, this makes a lot of sense).
nope, I must have been busy typing this, but glancing at your comments it sounds
like we are mostly on the same page. detailed replies below.
mmcount currently will create a count for each different appname
mmsequence will allow you to create a variable and incriment it at will
(it sounds like these can probably be generalized and unified somehow)
we need the ability to set a value that will be added to the variable set
for future messages
As I see it, there are two parts to this.
1. the part that sets the variables/modifies counters
2. the part that populates the current message variables ($! or $.) with
the current values.
part 1 needs to be thread safe since there may be multiple worker threads
modifying things at once.
how accurate do the numbers in part 2 need to be? if you try to have all
the counters be atomically updated and reported, there will be a lot of
overhead, but since you have one thread populating things while others are
changing them it seems like you may not have a lot of choice?
yup, but this may be somehow part of the operation (can'T think about the
details right now). At least, you get the precision you had with previous
global vars, which was not bad (I talk about races, not the "batch
poblem").
mmcount avoids this problem because it is single-threaded and what it does
is fixed (it maintains counters based on appname), so a single call can
maintain the counters and populate the message variables correctly in all
cases.
but mmsequence sounds like it allows for much more user control, some of
which will need to be conditional, so it may be invoked from within a
condition (and therefor be done within a worker thread)
any string variable setting is going to be done conditionally within a
worker thread.
as I'm trying to type this out and think through it, I'm wondering if the
right answer isn't to do something that very much resembles the 7.4.4
global variables.
mmglobal is going to need to store it's data somewhere, have that
somewhere be the $/ namespace like we had in 7.4.4 instead of having to
modify each message's variable space with all the global variables.
NO! You don't store all global variables in the message space. You just
must make sure that the one(s) your are interested in are stroed in the
message space (or even not, the more I think about it, the more the
embedding of global() into an if-condition should work).
Ok, it had sounded like you were talking about doing
action(type='mmglobal')
and that would populate the current message variables with all the global data.
now, this was my guesswork based on the fact that this is how all the other mm
modules get invoked
I was not suggesting that $/ would be the same implementation that we had
before, simply that it would be the equivalent to your global() function to
return the value of a variable, but looking syntactially like a variable
assignment.
don't allow global variables to be used in anything other than the right
hand side of a set statement (if you care about it, you set a message
variable equal to the global variable, and then refer to the message
variable in your if or template statement). This should avoid all the loop
unrolling problems
yup - but I want this done in a way that's intutitive and cannot
"circumvented" by a user -- global() does that.
agreed, the only difference between my $/ and your global() is syntax.
functionally they are identical. I happen to think that $/ is easier to read
than global(), but I accept that internal implementation details may make
global() much easier to implement.
Then require function calls to change anything in the global namespace
It sounds like we would need something similar to the following set of
functions
globalset(var='$/varname' subvar='$property!or!string' value='string or
number');
globalinc(var='$/varname' subvar='$property!or!string' step='number
default 1');
globaldec(var='$/varname' subvar='$property!or!string' step='number
default 1');
I personally prefer a single functon which gets an operation argument, like
I described, but I think the idea is mostly the same".
I'm ok with that when the different things the function does are very similar, I
really hate it when the names get too generic and you have to start reading
parameters to figure out what's really happening. in the v6+ syntax the fact
that everything is an action() is annoying, but there is enough commonality for
me to accept it.
the worst case is what I see people tending to do with XML where everything is
<object> and you have to look at all the parameters to figure out what is
actually happening :-)
in this case, I see there being functions being different enough to be worth
different names.
I could easily live with something like:
globalmodify(var='varname' subvar='$property!and!or!string' setvalue='value'
step='number')
where you should not use setvalue and step at the same time (setvalue sets it to
a particular value, step modifies the current value by number, positive or
negative)
having it return the new value makes a lot of sense (allowing it to be used in a
single statement if desired)
given your earlier example of loop unrolling, I'm not sure that this could be
used in an if statement, but if it can, great!
you could make it return the old value of the variable if you use setvalue, but
having it return the new value in some cases and the old value in others exceeds
by 'combine things into one function' tolorance :-) (not to mention the question
of what to return if someone uses both setvalue and step? ;-p ) so we should
still need a separate function to return the old value and set a new one
atomically.
also, a note, in my example above it would create $/hostcount!host1,
$/hostcount!host2, etc and when I did the call to return the current value to
the $.myhostcount variable, I would get $.myhostcount!host1, etc
I don't know if it makes more sense to have varname that can contain variable
portions, or have those variable portions be in subvar. I think there is a _lot_
of value in teaching rsyslog to support the idea of $!var!{$property}!var
everywhere, but that's probably a large change to the engine.
this would let you set a string variable, set and use a counter, including
counting based on the contents of a message
so if you wanted counts per hostname you would do something like:
globalinc(var='$/hostcounts' subvar='$hostname')
which would create $/hostcounts!<hostname> variables as needed.
when you want to use these counts, you can
set $.myhostcounts = $/hostcounts;
NO! Because you once again use global space between statements. Problem
reapears. The operation MUST happen and result returned immediately. This
would work:
set $.myhostcounts = globalinc(var='$/hostcounts' subvar='$hostname')
because it is in a single statement. Note the difference (two vs. one
stmts).
you aren't always going to be able to combine things into a single statement,
sometimes you will be dealing with multiple counters you want to manage (say
hostname based and application based).
or what if you want to do an if condition based on a global variable, and then
modify it based on the result? (think reset a counter for example)
my idea is that any function that modifies a global variable must do so
instantly and atomically, and any read of the global variable must also be
instant.
If you have multiple statements that use a global variable, it may be changed
between those statements by other threads, which is why you should make a local
copy and refer to that.
and then use them in the message
Then you would need something like
globalReportReset(var='$/**varname')
that you would use instead of $/hostcounts in the set above that would
atomically set $.myhostcounts and then destroy all the variables (the reset
portion)
The one other thing that I think would end up being needed, is some way to
do something to access a variable based on a property, something like:
$.myhostcounts!{$hostname}
returning the value for the hostnmae of the current message
I think I'll make a test implementation of the global() function as I
described it. HOWEVER, I am still in the middle of that refactoring for
variable names, and for some reason I have not made much progress with that
;) So before coming to the interesting thing (global()). there is a lot
more of important boring work in front of me.
this was intended to get ideas out there and have people (including me) thinking
about options for a bit before we get to the implementation stage.
I think that we have very similar functions in mind, we differ on some syntactic
things, and I think I've identified a couple corner cases that I don't see in
your version, but I'll wait until you have time to look at things more (and get
the current cleanup done)
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
