Hi Pavel, This looks awesome! Judging from the example, I could use this to work around the fact that mmnormalize could be a bottleneck, as Rainer pointed out in another thread you started?
I was about to start using mmnormalize to parse common logging formats, but I was put off by the fact that it would impact performance so much, because I'm dealing with high traffic as well. Plus the fact that I don't know how to match strings until the end of the line, but that's another thread :) I'm curious what others say about it before I put my "testing bunny" hat on :) Best regards, Radu 2013/10/20 Pavel Levshin <[email protected]> > Hello. > > > So, now I know that actions are not reentrant in rsyslog. Therefore, any > single action cannot consume more than one core of CPU. Nowadays, there > are common servers having 24 cores, and this limits our ability to handle > high load. Making all modules thread-safe would be great, but takes huge > amount of effort. And there is much simpler solution. > > We can define multiple identical or similar actions and divide the load > between them. Rsyslog has conditional statements to accomplish this. > But, unfortunately, rsyslog does not provide a variable or a function which > could be checked in such statement. Or, possibly, I am just unable to find > it. Basically, all we need is a variable, which values are evenly > distributed in a known range. > > Additional use of this method would be to balance load between two or more > different output actions. Say, to use rsyslog as a network load balancer. > > Let me introduce mmsequence. It is a message modification module, heavily > based on mmcount. It's purpose is to generate some numbers and > store them in message properties. I'm not a rsyslog guru or a professional > programmer, so please review my code. But, at least, it seemes to work > here. > > The patch is based on HEAD. > > Description: > > This module generates numeric sequences of different kinds. It can be used > to count messages up to a limit and to number them. It can generate random > numbers in a given range. > > This module is implemented via the output module interface, so it is > called just as an action. The number generated is stored in > "CEE/Lumberjack" > property of the message. > > Action Parameters: > > - mode "random" or "instance" or "key" > Specifies mode of the action. Default mode is "random", which > generates uniformly distributed integer numbers in a range defined > by "from" and "to". > > In "instance" mode, the action produces a counter in range [from, > to). > This counter is specific to this action instance. > > In "key" mode, the counter can be shared between multiple > instances. > This counter is identified by a name, which is defined with "key" > parameter. > > - from [non-negative integer], default "0" > Starting value for counters and lower margin for random generator. > > - to [positive integer], default "2" > Upper margin for all sequences. Note that this margin is not > inclusive. When next value for a counter is equal or greater than > this parameter, the counter resets to the starting value. > > - step [non-negative integer], default "1" > Increment for counters. If step is "0", it can be used to fetch > current value without modification. The latter not applies to > "random" mode. This is useful in "key" mode or to get constant > values in "instance" mode. > > - key [word], default "" > Name of the global counter which is used in this action. > > - var [word], default "!mmsequence" > Name of the message property where the number will be stored. > Should start with "!". > > Sample: > > # load balance > Ruleset( > name="logd" > queue.workerthreads="5" > ){ > > Action( > type="mmsequence" > mode="instance" > from="0" > to="2" > var="!seq" > ) > > if $!seq == "0" then { > Action( > type="mmnormalize" > userawmsg="on" > rulebase="/etc/rsyslog.d/**rules.rb" > ) > } else { > Action( > type="mmnormalize" > userawmsg="on" > rulebase="/etc/rsyslog.d/**rules.rb" > ) > } > > # output logic here > } > # generate random numbers > action( > type="mmsequence" > mode="random" > to="100" > var="!rndz" > ) > # count from 0 to 99 > action( > type="mmsequence" > mode="instance" > to="100" > var="!cnt1" > ) > # the same as before but the counter is global > action( > type="mmsequence" > mode="key" > key="key1" > to="100" > var="!cnt2" > ) > # count specific messages but place the counter in every message > if $msg contains "txt" then > action( > type="mmsequence" > mode="key" > to="100" > var="!cnt3" > ) > else > action( > type="mmsequence" > mode="key" > to="100" > step="0" > var="!cnt3" > key="" > ) > > > > Legacy Configuration Directives: > > Not supported. > > > -- > Pavel Levshin > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
