Thanks Rainer, I will change it from "contains" to maybe "startswith"? I am looking for a "stop directive on the documentation" but I am not seeing it? ----- Original Message ----- From: Rainer Gerhards Sent: 09/25/13 11:22 AM To: rsyslog-users Subject: Re: [rsyslog] perfomance tweaking
On Wed, Sep 25, 2013 at 5:09 PM, Robert <[email protected]> wrote: > sure sorry I begin a new thread, but I should just reply to it: here is > the config: > > Thx! Some comments inline below: > # Run this in debugger mode to troubleshoot > module(load="impstats" interval="300" severity="7") > (important) nit-picking: this is not "debug mode", but tells you about the internal counters. This is a huge difference, because stats have a very light overhead, whereas in debug mode things are slowed down to one tenth of the usual performance (or even lower). > # # to actually gather the data: > syslog.=debug /var/log/rsyslog-stats > #/var/log/debugformat;RSYSLOG_DebugFormat > > # rsyslog v7 configuration file > > # For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html > # If you experience problems, see > http://www.rsyslog.com/doc/troubleshoot.html > > #### MODULES #### > > module(load="imuxsock") # needs to be done just once Roberto 8-9-13 > #SysSock.FlowControl=(:"on") # enable flow control (use if needed) Roberto > 8-9-13 > #$ModLoad imuxsock # provides support for local system logging (e.g. via > logger command) > > module(load="imklog") > #$ModLoad imklog # provides kernel logging support (previously done by > rklogd) > #$ModLoad immark # provides --MARK-- message capability > > # Provides UDP syslog reception > module(load="imudp" > TimeRequery="10" > SchedulingPolicy="fifo" > SchedulingPriority="10") # needs to be done just once Roberto 8-9-13 > input(type="imudp" port="514") # Roberto 8-9-13 > > # Use default timestamp format > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > > # Include all config files in /etc/rsyslog.d/ > $IncludeConfig /etc/rsyslog.d/*.conf > > # Set Main Message Queue Size - default is 10000 > $MainMsgQueueType FixedArray #LinkedList > $MainMsgQueueSize 20000000 > $MainMsgQueueWorkerThreads 2 > $MainMsgQueueDequeueBatchSize 1000 > I would even try to set this higher, maybe 10000 - but that's probably n ot a big difference (but worth a try). > $InputUDPMaxSessions 40000000 > > :hostname, contains, "fdfw" action(type="omfile" > DirCreateMode="0700" > FileCreateMode="0644" > File="/logs/f_fw" > AsyncWriting="on" > FlushOnTXEnd="on" > IOBufferSize="8k" > queue.type="FixedArray" queue.dequeuebatchsize="100" queue.size="10000" > ) > > :hostname, contains, "mail" action(type="omfile" > DirCreateMode="0700" > FileCreateMode="0644" > File="/logs/f_mail" > AsyncWriting="on" > FlushOnTXEnd="on" > IOBufferSize="8k" > queue.type="FixedArray" queue.dequeuebatchsize="100" queue.size="10000" > ) > > :hostname, contains, "shib" action(type="omfile" > DirCreateMode="0700" > FileCreateMode="0644" > File="/logs/f_shib" > AsyncWriting="on" > FlushOnTXEnd="on" > IOBufferSize="8k" > queue.type="FixedArray" queue.dequeuebatchsize="100" queue.size="10000" > ) > > :hostname, contains, "pdc" action(type="omfile" > DirCreateMode="0700" > FileCreateMode="0644" > File="/logs/f_ad" > AsyncWriting="on" > FlushOnTXEnd="on" > IOBufferSize="8k" > queue.type="FixedArray" queue.dequeuebatchsize="100" queue.size="10000" > ) > > :hostname, contains, "networks" action(type="omfile" > DirCreateMode="0700" > FileCreateMode="0644" > File="/logs/f_networks" > AsyncWriting="on" > FlushOnTXEnd="on" > IOBufferSize="8k" > queue.type="FixedArray" queue.dequeuebatchsize="100" queue.size="10000" > ) > > :hostname, contains, "rout" action(type="omfile" > DirCreateMode="0700" > FileCreateMode="0644" > File="/logs/f_router" > AsyncWriting="on" > FlushOnTXEnd="on" > IOBufferSize="8k" > queue.type="FixedArray" queue.dequeuebatchsize="100" queue.size="10000" > ) > > :hostname, contains, "vm" action(type="omfile" > DirCreateMode="0700" > FileCreateMode="0644" > File="/logs/vm" > AsyncWriting="on" > FlushOnTXEnd="on" > IOBufferSize="8k" > queue.type="FixedArray" queue.dequeuebatchsize="100" queue.size="10000" > ) > > :hostname, contains, "pix" action(type="omfile" > DirCreateMode="0700" > FileCreateM ode="0644" > File="/logs/f_pix" > AsyncWriting="on" > FlushOnTXEnd="on" > IOBufferSize="8k" > queue.type="FixedArray" queue.dequeuebatchsize="100" queue.size="10000" > ) > looking at it, I need to mention that "contains" type of filters are very expensive, because we need to search multiple times inside the string. If you can avoid them, that is a huge win. From what I read so far, this is probably the culprit here. But one more thing: is it possible that one message falls in different categories? If not, you can stop processing (via the "stop" directive) as soon as the message is processed. This will save you evaluation of all the other filters... Again, this can be a huge difference (especially if you filter for the common case first). Rainer _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WEL L: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. Robert. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
