Sorry, I am swamped with fixing an important segfault issue we see in one environment, so I do not have time for an more in-depth answer (other list members may have). But I suggest to look into multiple ruleset support, which is in its infancy, but may help.
Rainer > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of Aaron Nichols > Sent: Tuesday, September 22, 2009 3:51 AM > To: rsyslog-users > Subject: [rsyslog] Improving filter performance & general performance > > Hi Everyone, > > I have rsyslog 4.4.1 chugging along reasonably well but am > looking for ways > to improve performance and optimize the filter ruleset. > Unfortunately I have > to create fairly extensive rulesets to filter on hostname, > programname, > facility, priority, etc. Some log sources generate a high > volume of logs (a > few Mbytes/sec) across multiple machines and others generate a fairly > routine amount of log data - maybe 5 meg per day. Many filters have > duplicate conditions for some values but there is always > variance. I have > tried to order the rules so that the highest volume logs > match first and > then are discarded. I've included a sample of the rules used > for my highest > volume logs (names changed to protect the innocent). > > If there are ways to chain or nest rules so that I can take > advantage of > matches already made against a log entry to filter it > minimally that would > be great. For example, most of the below rules filter on the > same facility & > list of hostnames but look for different values in the > 'rawmsg'. If I could > filter on the facility & hostname once and then rawmsg to > sort to different > destinations I'm guessing it would be lower overhead but I > don't really know > how the processing logic works. > > Also - if a condition is not met, are other parts of the > filter evaluated? > For example, if a message was received on local0, would any conditions > beyond "if $syslogfacility-text == 'local1'" be evaluated? Is it more > efficient to filter on the undecoded value syslogfacility vs. > syslogfacility-text? > > I'm looking for suggestions or general techniques for optimizing rule > performance under these circumstances. > > $template XMLFormat, "%syslogtag%%msg%\n" > > if $syslogfacility-text == 'local1' and ( \ > $fromhost startswith 'hosta' or \ > $fromhost startswith 'hostb' or \ > $fromhost startswith 'hostc' or \ > $fromhost startswith 'hostd' \ > ) and $rawmsg contains 'protocolLogRecord' then > -/log/syslog/collated/server/protocol.log;XMLFormat > & ~ # discard after match > if $syslogfacility-text == 'local1' and ( \ > $fromhost startswith 'hosta' or \ > $fromhost startswith 'hostb' or \ > $fromhost startswith 'hostc' or \ > $fromhost startswith 'hostd' \ > ) and $rawmsg contains 'messageLogRecord' then > -/log/syslog/collated/server/message.log;XMLFormat > & ~ # discard after match > if $syslogfacility-text == 'local1' and ( \ > $fromhost startswith 'hosta' or \ > $fromhost startswith 'hostb' or \ > $fromhost startswith 'hostc' or \ > $fromhost startswith 'hostd' \ > ) and $rawmsg contains 'clientLogRecord' then > -/log/syslog/collated/server/client.log;XMLFormat > & ~ # discard after match > if $syslogfacility-text == 'local2' and ( \ > $fromhost startswith 'hosta' or \ > $fromhost startswith 'hostb' or \ > $fromhost startswith 'hostc' or \ > $fromhost startswith 'hostd' \ > ) then -/log/syslog/collated/server/usage.log;XMLFormat > & ~ # discard after match > if $syslogfacility-text == 'local1' and ( \ > $fromhost startswith 'hosta' or \ > $fromhost startswith 'hostb' or \ > $fromhost startswith 'hostc' or \ > $fromhost startswith 'hostd' \ > ) and $rawmsg contains 'WAP Page Service ID' then > -/log/syslog/collated/server/customer-service;XMLFormat > & ~ # discard after match > if $syslogfacility-text == 'local1' and ( \ > $fromhost startswith 'hosta' or \ > $fromhost startswith 'hostb' or \ > $fromhost startswith 'hostc' or \ > $fromhost startswith 'hostd' \ > ) and $rawmsg contains 'locationlogrecord' then > -/log/syslog/collated/server/lbs.log;XMLFormat > & ~ # discard after match > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
