On Tue, Oct 22, 2013 at 2:26 PM, David Lang <[email protected]> wrote:
> so, what functions will mmglobal need?
>
>
not sure if you saw my message where I proposed a new global() function. I
already followed your earlier advice and searched for a non-action type of
solution (as you said, this makes a lot of sense).
> mmcount currently will create a count for each different appname
>
> mmsequence will allow you to create a variable and incriment it at will
>
> (it sounds like these can probably be generalized and unified somehow)
>
> we need the ability to set a value that will be added to the variable set
> for future messages
>
>
> As I see it, there are two parts to this.
>
> 1. the part that sets the variables/modifies counters
>
> 2. the part that populates the current message variables ($! or $.) with
> the current values.
>
> part 1 needs to be thread safe since there may be multiple worker threads
> modifying things at once.
>
> how accurate do the numbers in part 2 need to be? if you try to have all
> the counters be atomically updated and reported, there will be a lot of
> overhead, but since you have one thread populating things while others are
> changing them it seems like you may not have a lot of choice?
>
yup, but this may be somehow part of the operation (can'T think about the
details right now). At least, you get the precision you had with previous
global vars, which was not bad (I talk about races, not the "batch
poblem").
>
>
>
> mmcount avoids this problem because it is single-threaded and what it does
> is fixed (it maintains counters based on appname), so a single call can
> maintain the counters and populate the message variables correctly in all
> cases.
>
> but mmsequence sounds like it allows for much more user control, some of
> which will need to be conditional, so it may be invoked from within a
> condition (and therefor be done within a worker thread)
>
> any string variable setting is going to be done conditionally within a
> worker thread.
>
>
> as I'm trying to type this out and think through it, I'm wondering if the
> right answer isn't to do something that very much resembles the 7.4.4
> global variables.
>
> mmglobal is going to need to store it's data somewhere, have that
> somewhere be the $/ namespace like we had in 7.4.4 instead of having to
> modify each message's variable space with all the global variables.
NO! You don't store all global variables in the message space. You just
must make sure that the one(s) your are interested in are stroed in the
message space (or even not, the more I think about it, the more the
embedding of global() into an if-condition should work).
> don't allow global variables to be used in anything other than the right
> hand side of a set statement (if you care about it, you set a message
> variable equal to the global variable, and then refer to the message
> variable in your if or template statement). This should avoid all the loop
> unrolling problems
>
>
yup - but I want this done in a way that's intutitive and cannot
"circumvented" by a user -- global() does that.
> Then require function calls to change anything in the global namespace
>
> It sounds like we would need something similar to the following set of
> functions
>
> globalset(var='$/varname' subvar='$property!or!string' value='string or
> number');
>
> globalinc(var='$/varname' subvar='$property!or!string' step='number
> default 1');
>
> globaldec(var='$/varname' subvar='$property!or!string' step='number
> default 1');
>
>
>
I personally prefer a single functon which gets an operation argument, like
I described, but I think the idea is mostly the same".
> this would let you set a string variable, set and use a counter, including
> counting based on the contents of a message
>
> so if you wanted counts per hostname you would do something like:
>
> globalinc(var='$/hostcounts' subvar='$hostname')
>
> which would create $/hostcounts!<hostname> variables as needed.
>
> when you want to use these counts, you can
>
> set $.myhostcounts = $/hostcounts;
>
>
NO! Because you once again use global space between statements. Problem
reapears. The operation MUST happen and result returned immediately. This
would work:
set $.myhostcounts = globalinc(var='$/hostcounts' subvar='$hostname')
because it is in a single statement. Note the difference (two vs. one
stmts).
> and then use them in the message
>
>
> Then you would need something like
>
> globalReportReset(var='$/**varname')
>
> that you would use instead of $/hostcounts in the set above that would
> atomically set $.myhostcounts and then destroy all the variables (the reset
> portion)
>
> The one other thing that I think would end up being needed, is some way to
> do something to access a variable based on a property, something like:
>
> $.myhostcounts!{$hostname}
>
> returning the value for the hostnmae of the current message
>
>
I think I'll make a test implementation of the global() function as I
described it. HOWEVER, I am still in the middle of that refactoring for
variable names, and for some reason I have not made much progress with that
;) So before coming to the interesting thing (global()). there is a lot
more of important boring work in front of me.
Rainer
> David Lang
>
>
>
> On Tue, 22 Oct 2013, David Lang wrote:
>
> Date: Tue, 22 Oct 2013 04:30:44 -0700 (PDT)
>> From: David Lang <[email protected]>
>> Reply-To: rsyslog-users <[email protected]>
>> To: rsyslog-users <[email protected]>
>> Subject: Re: [rsyslog] global variable use cases
>>
>> On Tue, 22 Oct 2013, Rainer Gerhards wrote:
>>
>> On Tue, Oct 22, 2013 at 1:09 PM, David Lang <[email protected]> wrote:
>>>
>>> well, I wouldn't want to have the module store a counter into the $!
>>>> namespace, too much chance of conflicting with something from a message,
>>>> and if I've got several things stored (for different purposes), I don't
>>>> want them to all show up when I output $!
>>>>
>>>>
>>> let the user decided. For this, we have local vars.
>>>
>>
>> Ok, I thought you were saying that they would go into $!, not $.
>>
>> as you show below, doing this for an explicit save is very
>> stratightforward (although why do you need both key and store_to? I would
>> think either would be enough to specify the variable name to use)
>>
>> but for counting and sequences, especially for ones that count separately
>> based on a message property (such as appname), this seems harder.
>>
>> although I guess when you are setting up the counter, you can specify a
>> base variable, and allow the module to create subvariables from there.
>>
>> store it in $. or in some different namespace (say $/ ;-p )
>>>>
>>>> for counting and sequence numbers this sounds reasonable.
>>>>
>>>> but what are you thinking about for storing other values? I'm expecthing
>>>> that most of these other values will be strings.
>>>>
>>>>
>>> What's the difference?
>>>
>>> e.g. (just psedudocode)
>>>
>>> action(type="globals" key="mykey" value="some string"
>>> store_to=".localvar")
>>>
>>> it may be useful to implement this as function rather than action or
>>> even a
>>> totally new object -- makes integrating with expressions much easier (for
>>> outputs, you must always go through a template).
>>>
>>
>> I think a function would be better, doing it as an action brings along a
>> lot of baggage to confuse people (separate queues for example)
>>
>> But the 7.5.4 globals vars cannot do this, as you cannot attach semantics
>>> to them that *force* to write to a local var -- this is the root problem.
>>>
>>
>> well, since they were in a different namespace entirely, they didn't need
>> to write to a local var
>>
>>
>> One other issue that this does create is that there are now two classes
>> of message modification modules
>>
>> 1. modules that could have multiple copies running, needing only to
>> coordinate enough to not be modifying the same message
>>
>> 2. modules that are doing things across messages, so multiple copies get
>> really messy
>>
>> I'm not sure that this is really a whole lot better, but I guess it's
>> taking it away from directly being exposed to the users, so less potential
>> for confusion.
>>
>>
>> David Lang
>> ______________________________**_________________
>> rsyslog mailing list
>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>> ______________________________**_________________
> rsyslog mailing list
> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
