Im using a spirent emulator ----- Original Message ----- From: Robert McIntyre Sent: 09/24/13 09:17 AM To: rsyslog-users Subject: Re: [rsyslog] performance tweaking
Side question: what are you using to generate the messages? Thanks! Robert ________________________________ From: Robert<mailto:[email protected]> Sent: 9/24/2013 6:05 AM To: rsyslog-users<mailto:[email protected]> Subject: [rsyslog] performance tweaking Hello Everyone, So I went ahead and removed the action queues as well as the OMFile from all the rules, and it didnt seem to help. Currently I am emulating traffic at 250k mps and I am dropping alot of messages. [rcortiz@simon logs]$ ls f_ad f_fw f_mail f_networks f_pix f_router f_shib f_vm [rcortiz@simon logs]$ sudo grep 14:16:00 * | wc -l 206838 [rcortiz@simon logs]$ sudo grep 14:16:01 * | wc -l 213102 [rcortiz@simon logs]$ sudo grep 14:16:02 * | wc -l 205855 [rcortiz@simon logs]$ sudo grep 14:16:03 * | wc -l 205438 [rcortiz@simon logs]$ sudo grep 14:16:04 * | wc -l 212700 [rcortiz@simon logs]$ sudo grep 14:16:05 * | wc -l 215814 [rcortiz@simon logs]$ sudo grep 14:16:06 * | wc -l 215401 [rcortiz@simon logs]$ s udo grep 14:16:07 * | wc -l 215484 [rcortiz@simon logs]$ sudo grep 14:16:10 * | wc -l 212617 [rcortiz@simon logs]$ sudo grep 14:16:20 * | wc -l 218299 [rcortiz@simon logs]$ sudo grep 14:16:30 * | wc -l 248083 [rcortiz@simon logs]$ sudo grep 14:16:40 * | wc -l 199360 [rcortiz@simon logs]$ sudo grep 14:16:50 * | wc -l 216817 [rcortiz@simon logs]$ sudo grep 14:17:00 * | wc -l 250006 [rcortiz@simon logs]$ sudo grep 14:17:01 * | wc -l 250000 [rcortiz@simon logs]$ sudo grep 14:17:02 * | wc -l 249999 [rcortiz@simon logs]$ sudo grep 14:17:03 * | wc -l 218583 [rcortiz@simon logs]$ sudo grep 14:17:04 * | wc -l 210238 [rcortiz@simon logs]$ sudo grep 14:17:05 * | wc -l 220693 [rcortiz@simon logs]$ sudo grep 14:17:06 * | wc -l 217096 [rcortiz@simon logs]$ sudo grep 14:17:07 * | wc -l 217989 [rcortiz@simon logs]$ sudo grep 14:17:08 * | wc -l 212015 [rcortiz@simon logs]$ sudo grep 14:17:010 * | wc -l ^C [rcortiz@simon logs]$ sudo grep 14:17:10 * | wc -l 234442 [rcortiz@simon logs]$ sudo gr ep 14:17:11 * | wc -l 227256 [rcortiz@simon logs]$ sudo grep 14:17:20 * | wc -l 235159 [rcortiz@simon logs]$ sudo grep 14:17:30 * | wc -l 249996 [rcortiz@simon logs]$ sudo grep 14:17:31 * | wc -l 223702 [rcortiz@simon logs]$ sudo grep 14:17:40 * | wc -l 194462 [rcortiz@simon logs]$ sudo grep 14:17:50 * | wc -l 31053 Whenever I add the OMFile setinngs back it gives me better results, but still drops a significant amount [rcortiz@simon logs]$ sudo grep 14:49:00 * | wc -l 242390 [rcortiz@simon logs]$ sudo grep 14:49:10 * | wc -l 242167 [rcortiz@simon logs]$ sudo grep 14:49:11 * | wc -l 245970 [rcortiz@simon logs]$ sudo grep 14:49:12 * | wc -l 245712 [rcortiz@simon logs]$ sudo grep 14:49:13 * | wc -l 243364 [rcortiz@simon logs]$ sudo grep 14:49:14 * | wc -l ^[[A242561 [rcortiz@simon logs]$ sudo grep 14:49:20 * | wc -l 242729 [rcortiz@simon logs]$ sudo grep 14:49:21 * | wc -l 243464 [rcortiz@simon logs]$ sudo grep 14:49:22 * | wc -l 243144 [rcortiz@simon logs]$ sudo grep 14:49:25 * | wc -l 246727 [rcortiz@simon logs]$ sudo grep 14:49:26 * | wc -l 243934 [rcortiz@simon logs]$ sudo grep 14:49:27 * | wc -l 246269 [rcortiz@simon logs]$ sudo grep 14:49:28 * | wc -l 245634 [rcortiz@simon logs]$ sudo grep 14:49:29 * | wc -l 238512 [rcortiz@simon logs]$ sudo grep 14:49:30 * | wc -l 240527 [rcortiz@simon logs]$ sudo grep 14:49:31 * | wc -l 242542 [rcortiz@simon logs]$ sudo grep 14:50:00 * | wc -l 243648 [rcortiz@simon logs]$ sudo grep 14:50:01 * | wc -l 241878 [rcortiz@simon logs]$ sudo grep 14:50:02 * | wc -l 238950 [rcortiz@simon logs]$ sudo grep 14:50:03 * | wc -l 247795 [rcortiz@simon logs]$ I have attached my mpstats, my overall cpu stats, and the rsyslog-stats from the impstats module. Another question I had was: when I write these Action and OMfile settings in the legacy syntax I think this is right, but what is the right way of doing so with the new syntax?...not sure how to add the queue sizes? ************************************** #$ActionQueueWor kerThreads 8 #$ActionQueueSize 10000000 #$ActionQueueType FixedArray #LinkedList - use asynchronous processing $OMFileAsyncWriting on $OMFileFlushOnTXEnd on $OMFileIOBufferSize 1000k :hostname, contains, "fdfw"/logs/f_fw ************************************** #:hostname, contains, "fdfw" action(type="omfile" DirCreateMode="0700" FileCreateMode="0644" File="/logs/f_fw" AsyncWriting="on" FlushOnTXEnd="on" IOBufferSize="1000k") **************************************** thanks for the help everyone Robert. Robert. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
