Thanks for the interesting approach. Please note that you can do everything (except random mode) with global variables. Doc here:
http://www.rsyslog.com/doc/rainerscript.html and some more here: http://blog.gerhards.net/2012/09/setting-variables-in-rsyslog-v7.html (the latter handles message variables, but obviously it works for all name spaces) Random mode could be fairly simple implemented by introducing a new function call. As I want to limit duplicate new functionality (which I need to maintain in the long run), I will most probably not merge the module. But the approach you have taken is interesting. Actually, what you want to define is a set of action instances with equal configuration, and you don't really care which instance is used to execute a request. This makes a lot of sense, as very often it is extremely hard (even undoable) to write a truely thread-safe single instance action (e.g. you have no chance for MySQL and other databases, as by definition they do not support multiple concurrent calls on the same handle). Having multiple instances relaxes this very much. As I said, I have looked at the core engine code in question. Supporting thread-safe actions is probably not too hard to do (but not a half hour job!), but it is questionable how much can be gained from it. However, I think adding such a multi-instance facility that you describe makes much more sense for a bunch of actions and also doesn't sound like being very hard to do (actually, I need to create n instances instead of a single one and need to pthread_trylock() until I find a free one - or all busy, in which case we need to wait). This is also very intersting because of its broad aplicability and offers potential great benefit. So I will try to look into this within the next days to weeks. If there is no hidden problem, I think this could materialize relatively quickly. Thanks for the suggestion! Rainer On Sun, Oct 20, 2013 at 11:18 AM, Pavel Levshin <[email protected]>wrote: > Hello. > > > So, now I know that actions are not reentrant in rsyslog. Therefore, any > single action cannot consume more than one core of CPU. Nowadays, there > are common servers having 24 cores, and this limits our ability to handle > high load. Making all modules thread-safe would be great, but takes huge > amount of effort. And there is much simpler solution. > > We can define multiple identical or similar actions and divide the load > between them. Rsyslog has conditional statements to accomplish this. > But, unfortunately, rsyslog does not provide a variable or a function which > could be checked in such statement. Or, possibly, I am just unable to find > it. Basically, all we need is a variable, which values are evenly > distributed in a known range. > > Additional use of this method would be to balance load between two or more > different output actions. Say, to use rsyslog as a network load balancer. > > Let me introduce mmsequence. It is a message modification module, heavily > based on mmcount. It's purpose is to generate some numbers and > store them in message properties. I'm not a rsyslog guru or a professional > programmer, so please review my code. But, at least, it seemes to work > here. > > The patch is based on HEAD. > > Description: > > This module generates numeric sequences of different kinds. It can be used > to count messages up to a limit and to number them. It can generate random > numbers in a given range. > > This module is implemented via the output module interface, so it is > called just as an action. The number generated is stored in > "CEE/Lumberjack" > property of the message. > > Action Parameters: > > - mode "random" or "instance" or "key" > Specifies mode of the action. Default mode is "random", which > generates uniformly distributed integer numbers in a range defined > by "from" and "to". > > In "instance" mode, the action produces a counter in range [from, > to). > This counter is specific to this action instance. > > In "key" mode, the counter can be shared between multiple > instances. > This counter is identified by a name, which is defined with "key" > parameter. > > - from [non-negative integer], default "0" > Starting value for counters and lower margin for random generator. > > - to [positive integer], default "2" > Upper margin for all sequences. Note that this margin is not > inclusive. When next value for a counter is equal or greater than > this parameter, the counter resets to the starting value. > > - step [non-negative integer], default "1" > Increment for counters. If step is "0", it can be used to fetch > current value without modification. The latter not applies to > "random" mode. This is useful in "key" mode or to get constant > values in "instance" mode. > > - key [word], default "" > Name of the global counter which is used in this action. > > - var [word], default "!mmsequence" > Name of the message property where the number will be stored. > Should start with "!". > > Sample: > > # load balance > Ruleset( > name="logd" > queue.workerthreads="5" > ){ > > Action( > type="mmsequence" > mode="instance" > from="0" > to="2" > var="!seq" > ) > > if $!seq == "0" then { > Action( > type="mmnormalize" > userawmsg="on" > rulebase="/etc/rsyslog.d/**rules.rb" > ) > } else { > Action( > type="mmnormalize" > userawmsg="on" > rulebase="/etc/rsyslog.d/**rules.rb" > ) > } > > # output logic here > } > # generate random numbers > action( > type="mmsequence" > mode="random" > to="100" > var="!rndz" > ) > # count from 0 to 99 > action( > type="mmsequence" > mode="instance" > to="100" > var="!cnt1" > ) > # the same as before but the counter is global > action( > type="mmsequence" > mode="key" > key="key1" > to="100" > var="!cnt2" > ) > # count specific messages but place the counter in every message > if $msg contains "txt" then > action( > type="mmsequence" > mode="key" > to="100" > var="!cnt3" > ) > else > action( > type="mmsequence" > mode="key" > to="100" > step="0" > var="!cnt3" > key="" > ) > > > > Legacy Configuration Directives: > > Not supported. > > > -- > Pavel Levshin > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
