Very nice. Wasn't sure how much math functionality is available, and I'm on my
phone.
Any requirements around the actual variable name?
--Robert
________________________________
From: David Lang<mailto:[email protected]>
Sent: 10/20/2013 7:45 AM
To: rsyslog-users<mailto:[email protected]>
Subject: Re: [rsyslog] A solution to action load balancing
something more along the lines of
set $\globalvar = globalvar +1
if $\globalvar % 3 = 0 then action1
if $\globalvar % 3 = 1 then action2
if $\globalvar % 3 = 2 then action3
just incrament globalvar for each message, then use the modulas function
(remainder after integer division) to pick a round-robin action.
David Lang
On Sun, 20 Oct 2013, Robert McIntyre wrote:
> Date: Sun, 20 Oct 2013 07:23:34 -0700
> From: Robert McIntyre <[email protected]>
> Reply-To: rsyslog-users <[email protected]>
> To: rsyslog-users <[email protected]>
> Subject: Re: [rsyslog] A solution to action load balancing
>
> Wow, this email thread is very timely! I am interested in using the local
> variables to do some load balancing in my system, but am having a bit of
> trouble wrapping my head around the variable naming requirements. I looked
> at some of the CEE/Lumberjack docs, and they seem to define a rigid set of
> possible variable names.
>
> My goal would be something simple like:
>
> If globalvar==0
> set globalvar = 1
> execute action 0
> stop
> If globalvar==1
> set globalvar=2
> execute action 1
> stop
> If globalvar==2
> set globalvar=0
> execute action 2
> stop
>
> This would give me a simple round-robin load balance for actions. I
> understand that the updates may not be atomic and precise, but it would
> probably just be noise at the event volumes I expect.
>
> Can someone get me started with some references/example code (other than
> Rainer's below)?
>
> Thanks!
> Robert
> ________________________________
> From: Rainer Gerhards<mailto:[email protected]>
> Sent: 10/20/2013 6:26 AM
> To: rsyslog-users<mailto:[email protected]>
> Subject: Re: [rsyslog] A solution to action load balancing
>
> Thanks for the interesting approach.
>
> Please note that you can do everything (except random mode) with global
> variables. Doc here:
>
> http://www.rsyslog.com/doc/rainerscript.html
> and some more here:
> http://blog.gerhards.net/2012/09/setting-variables-in-rsyslog-v7.html
> (the latter handles message variables, but obviously it works for all name
> spaces)
>
> Random mode could be fairly simple implemented by introducing a new
> function call. As I want to limit duplicate new functionality (which I need
> to maintain in the long run), I will most probably not merge the module.
>
> But the approach you have taken is interesting. Actually, what you want to
> define is a set of action instances with equal configuration, and you don't
> really care which instance is used to execute a request. This makes a lot
> of sense, as very often it is extremely hard (even undoable) to write a
> truely thread-safe single instance action (e.g. you have no chance for
> MySQL and other databases, as by definition they do not support multiple
> concurrent calls on the same handle).
>
> Having multiple instances relaxes this very much. As I said, I have looked
> at the core engine code in question. Supporting thread-safe actions is
> probably not too hard to do (but not a half hour job!), but it is
> questionable how much can be gained from it.
>
> However, I think adding such a multi-instance facility that you describe
> makes much more sense for a bunch of actions and also doesn't sound like
> being very hard to do (actually, I need to create n instances instead of a
> single one and need to pthread_trylock() until I find a free one - or all
> busy, in which case we need to wait). This is also very intersting because
> of its broad aplicability and offers potential great benefit. So I will try
> to look into this within the next days to weeks. If there is no hidden
> problem, I think this could materialize relatively quickly.
>
> Thanks for the suggestion!
> Rainer
>
>
> On Sun, Oct 20, 2013 at 11:18 AM, Pavel Levshin <[email protected]>wrote:
>
>> Hello.
>>
>>
>> So, now I know that actions are not reentrant in rsyslog. Therefore, any
>> single action cannot consume more than one core of CPU. Nowadays, there
>> are common servers having 24 cores, and this limits our ability to handle
>> high load. Making all modules thread-safe would be great, but takes huge
>> amount of effort. And there is much simpler solution.
>>
>> We can define multiple identical or similar actions and divide the load
>> between them. Rsyslog has conditional statements to accomplish this.
>> But, unfortunately, rsyslog does not provide a variable or a function which
>> could be checked in such statement. Or, possibly, I am just unable to find
>> it. Basically, all we need is a variable, which values are evenly
>> distributed in a known range.
>>
>> Additional use of this method would be to balance load between two or more
>> different output actions. Say, to use rsyslog as a network load balancer.
>>
>> Let me introduce mmsequence. It is a message modification module, heavily
>> based on mmcount. It's purpose is to generate some numbers and
>> store them in message properties. I'm not a rsyslog guru or a professional
>> programmer, so please review my code. But, at least, it seemes to work
>> here.
>>
>> The patch is based on HEAD.
>>
>> Description:
>>
>> This module generates numeric sequences of different kinds. It can be used
>> to count messages up to a limit and to number them. It can generate random
>> numbers in a given range.
>>
>> This module is implemented via the output module interface, so it is
>> called just as an action. The number generated is stored in
>> "CEE/Lumberjack"
>> property of the message.
>>
>> Action Parameters:
>>
>> - mode "random" or "instance" or "key"
>> Specifies mode of the action. Default mode is "random", which
>> generates uniformly distributed integer numbers in a range defined
>> by "from" and "to".
>>
>> In "instance" mode, the action produces a counter in range [from,
>> to).
>> This counter is specific to this action instance.
>>
>> In "key" mode, the counter can be shared between multiple
>> instances.
>> This counter is identified by a name, which is defined with "key"
>> parameter.
>>
>> - from [non-negative integer], default "0"
>> Starting value for counters and lower margin for random generator.
>>
>> - to [positive integer], default "2"
>> Upper margin for all sequences. Note that this margin is not
>> inclusive. When next value for a counter is equal or greater than
>> this parameter, the counter resets to the starting value.
>>
>> - step [non-negative integer], default "1"
>> Increment for counters. If step is "0", it can be used to fetch
>> current value without modification. The latter not applies to
>> "random" mode. This is useful in "key" mode or to get constant
>> values in "instance" mode.
>>
>> - key [word], default ""
>> Name of the global counter which is used in this action.
>>
>> - var [word], default "!mmsequence"
>> Name of the message property where the number will be stored.
>> Should start with "!".
>>
>> Sample:
>>
>> # load balance
>> Ruleset(
>> name="logd"
>> queue.workerthreads="5"
>> ){
>>
>> Action(
>> type="mmsequence"
>> mode="instance"
>> from="0"
>> to="2"
>> var="!seq"
>> )
>>
>> if $!seq == "0" then {
>> Action(
>> type="mmnormalize"
>> userawmsg="on"
>> rulebase="/etc/rsyslog.d/**rules.rb"
>> )
>> } else {
>> Action(
>> type="mmnormalize"
>> userawmsg="on"
>> rulebase="/etc/rsyslog.d/**rules.rb"
>> )
>> }
>>
>> # output logic here
>> }
>> # generate random numbers
>> action(
>> type="mmsequence"
>> mode="random"
>> to="100"
>> var="!rndz"
>> )
>> # count from 0 to 99
>> action(
>> type="mmsequence"
>> mode="instance"
>> to="100"
>> var="!cnt1"
>> )
>> # the same as before but the counter is global
>> action(
>> type="mmsequence"
>> mode="key"
>> key="key1"
>> to="100"
>> var="!cnt2"
>> )
>> # count specific messages but place the counter in every message
>> if $msg contains "txt" then
>> action(
>> type="mmsequence"
>> mode="key"
>> to="100"
>> var="!cnt3"
>> )
>> else
>> action(
>> type="mmsequence"
>> mode="key"
>> to="100"
>> step="0"
>> var="!cnt3"
>> key=""
>> )
>>
>>
>>
>> Legacy Configuration Directives:
>>
>> Not supported.
>>
>>
>> --
>> Pavel Levshin
>>
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
