sure sorry I begin a new thread, but I should just reply to it: here is the config:
# Run this in debugger mode to troubleshoot module(load="impstats" interval="300" severity="7") # # to actually gather the data: syslog.=debug /var/log/rsyslog-stats #/var/log/debugformat;RSYSLOG_DebugFormat # rsyslog v7 configuration file # For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html # If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html #### MODULES #### module(load="imuxsock") # needs to be done just once Roberto 8-9-13 #SysSock.FlowControl=(:"on") # enable flow control (use if needed) Roberto 8-9-13 #$ModLoad imuxsock # provides support for local system logging (e.g. via logger command) module(load="imklog") #$ModLoad imklog # provides kernel logging support (previously done by rklogd) #$ModLoad immark # provides --MARK-- message capability # Provides UDP syslog reception module(load="imudp" TimeRequery="10" SchedulingPolicy="fifo" SchedulingPriority="10") # needs to be done just once Roberto 8-9-13 input(type="imudp" port="514") # Roberto 8-9-13 # Use default timestamp format $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # Include all config files in /etc/rsyslog.d/ $IncludeConfig /etc/rsyslog.d/*.conf # Set Main Message Queue Size - default is 10000 $MainMsgQueueType FixedArray #LinkedList $MainMsgQueueSize 20000000 $MainMsgQueueWorkerThreads 2 $MainMsgQueueDequeueBatchSize 1000 $InputUDPMaxSessions 40000000 :hostname, contains, "fdfw" action(type="omfile" DirCreateMode="0700" FileCreateMode="0644" File="/logs/f_fw" AsyncWriting="on" FlushOnTXEnd="on" IOBufferSize="8k" queue.type="FixedArray" queue.dequeuebatchsize="100" queue.size="10000" ) :hostname, contains, "mail" action(type="omfile" DirCreateMode="0700" FileCreateMode="0644" File="/logs/f_mail" AsyncWriting="on" FlushOnTXEnd="on" IOBufferSize="8k" queue.type="FixedArray" queue.dequeuebatchsize="100" queue.size="10000" ) :hostname, contains, "shib" action(type="omfile" DirCreateMode="0700" FileCreateMode="0644" File="/logs/f_shib" AsyncWriting="on" FlushOnTXEnd="on" IOBufferSize="8k" queue.type="FixedArray" queue.dequeuebatchsize="100" queue.size="10000" ) :hostname, contains, "pdc" action(type="omfile" DirCreateMode="0700" FileCreateMode="0644" File="/logs/f_ad" AsyncWriting="on" FlushOnTXEnd="on" IOBufferSize="8k" queue.type="FixedArray" queue.dequeuebatchsize="100" queue.size="10000" ) :hostname, contains, "networks" action(type="omfile" DirCreateMode="0700" FileCreateMode="0644" File="/logs/f_networks" AsyncWriting="on" FlushOnTXEnd="on" IOBufferSize="8k" queue.type="FixedArray" queue.dequeuebatchsize="100" queue.size="10000" ) :hostname, contains, "rout" action(type="omfile" DirCreateMode="0700" FileCreateMode="0644" File="/logs/f_router" AsyncWriting="on" FlushOnTXEnd="on" IOBufferSize="8k" queue.type="FixedArray" queue.dequeuebatchsize="100" queue.size="10000" ) :hostname, contains, "vm" action(type="omfile" DirCreateMode="0700" FileCreateMode="0644" File="/logs/vm" AsyncWriting="on" FlushOnTXEnd="on" IOBufferSize="8k" queue.type="FixedArray" queue.dequeuebatchsize="100" queue.size="10000" ) :hostname, contains, "pix" action(type="omfile" DirCreateMode="0700" FileCreateMode="0644" File="/logs/f_pix" AsyncWriting="on" FlushOnTXEnd="on" IOBufferSize="8k" queue.type="FixedArray" queue.dequeuebatchsize="100" queue.size="10000" ) ----- Original Message ----- From: Rainer Gerhards Sent: 09/25/13 11:03 AM To: rsyslog-users Subject: Re: [rsyslog] perfomance tweaking Hi Robert, On Wed, Sep 25, 2013 at 5:00 PM, Robert <[email protected]> wrote: > Thanks for the help David, I changed the worker threads to 2 from 8, > improved the cpu stats significantly. yeah, so we had too much concurrency... > I also modified the syntax for each rule like you suggested, It ran great > with 250k fps, not dropping a single packet. I deciced to up the spirent to > 300k and it did well but not as I hoped: > > I think we should again review the config to see what may causing cpu-boundness. Could you provide another copy (it's a bit hard following this thread as you often break it with non-replies ;)). thx, Rainer > [rcortiz@simon logs]$ ls -alh > total 31G > drwxr-xr-x 2 root wheel 125 Sep 25 10:27 . > dr-xr-xr-x. 27 root root 4.0K Sep 17 15:14 .. > -rwxr-xr-x 1 root nobody 1.1G Sep 25 10:39 f_ad > -rwxr-xr-x 1 root nobody 8.3G Sep 25 10:39 f_fw > -rwxr-xr-x 1 root nobody 6.4G Sep 25 10:39 f_mail > -rwxr-xr-x 1 root nobody 1.1G Sep 25 10:39 f_networks > -rwxr-xr- x 1 root nobody 1.1G Sep 25 10:39 f_pix > -rwxr-xr-x 1 root nobody 1.1G Sep 25 10:39 f_router > -rwxr-xr-x 1 root nobody 6.3G Sep 25 10:39 f_shib > -rwxr-xr-x 1 root nobody 1.1G Sep 25 10:39 vm > [rcortiz@simon logs]$ sudo sudo grep 10:29:00 * | wc -l > 279050 > [rcortiz@simon logs]$ sudo sudo grep 10:29:01 * | wc -l > 275160 > [rcortiz@simon logs]$ sudo sudo grep 10:29:02 * | wc -l > 258010 > [rcortiz@simon logs]$ sudo sudo grep 10:29:03 * | wc -l > 255300 > [rcortiz@simon logs]$ sudo sudo grep 10:29:04 * | wc -l > 258730 > [rcortiz@simon logs]$ sudo sudo grep 10:29:05 * | wc -l > 269470 > [rcortiz@simon logs]$ > > is there anything I can modify to improve the stats?: > > # Provides UDP syslog reception > module(load="imudp" > TimeRequery="10" > SchedulingPolicy="fifo" > SchedulingPriority="10") > input(type="imudp" port="514") > > # Set Main Message Queue Size - default is 10000 > $MainMsgQueueType FixedArray > $MainMsgQueueSize 20000000 > $MainMsgQueueWorkerThreads 2 > $M ainMsgQueueDequeueBatchSize 1000 > $InputUDPMaxSessions 40000000 > > Robert. > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. Robert. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
