Side question: what are you using to generate the messages?
Thanks!
Robert
________________________________
From: Robert<mailto:[email protected]>
Sent: 9/24/2013 6:05 AM
To: rsyslog-users<mailto:[email protected]>
Subject: [rsyslog] performance tweaking
Hello Everyone,
So I went ahead and removed the action queues as well as the OMFile from all
the rules, and it didnt seem to help. Currently I am emulating traffic at 250k
mps and I am dropping alot of messages.
[rcortiz@simon logs]$ ls
f_ad f_fw f_mail f_networks f_pix f_router f_shib f_vm
[rcortiz@simon logs]$ sudo grep 14:16:00 * | wc -l
206838
[rcortiz@simon logs]$ sudo grep 14:16:01 * | wc -l
213102
[rcortiz@simon logs]$ sudo grep 14:16:02 * | wc -l
205855
[rcortiz@simon logs]$ sudo grep 14:16:03 * | wc -l
205438
[rcortiz@simon logs]$ sudo grep 14:16:04 * | wc -l
212700
[rcortiz@simon logs]$ sudo grep 14:16:05 * | wc -l
215814
[rcortiz@simon logs]$ sudo grep 14:16:06 * | wc -l
215401
[rcortiz@simon logs]$ sudo grep 14:16:07 * | wc -l
215484
[rcortiz@simon logs]$ sudo grep 14:16:10 * | wc -l
212617
[rcortiz@simon logs]$ sudo grep 14:16:20 * | wc -l
218299
[rcortiz@simon logs]$ sudo grep 14:16:30 * | wc -l
248083
[rcortiz@simon logs]$ sudo grep 14:16:40 * | wc -l
199360
[rcortiz@simon logs]$ sudo grep 14:16:50 * | wc -l
216817
[rcortiz@simon logs]$ sudo grep 14:17:00 * | wc -l
250006
[rcortiz@simon logs]$ sudo grep 14:17:01 * | wc -l
250000
[rcortiz@simon logs]$ sudo grep 14:17:02 * | wc -l
249999
[rcortiz@simon logs]$ sudo grep 14:17:03 * | wc -l
218583
[rcortiz@simon logs]$ sudo grep 14:17:04 * | wc -l
210238
[rcortiz@simon logs]$ sudo grep 14:17:05 * | wc -l
220693
[rcortiz@simon logs]$ sudo grep 14:17:06 * | wc -l
217096
[rcortiz@simon logs]$ sudo grep 14:17:07 * | wc -l
217989
[rcortiz@simon logs]$ sudo grep 14:17:08 * | wc -l
212015
[rcortiz@simon logs]$ sudo grep 14:17:010 * | wc -l
^C
[rcortiz@simon logs]$ sudo grep 14:17:10 * | wc -l
234442
[rcortiz@simon logs]$ sudo grep 14:17:11 * | wc -l
227256
[rcortiz@simon logs]$ sudo grep 14:17:20 * | wc -l
235159
[rcortiz@simon logs]$ sudo grep 14:17:30 * | wc -l
249996
[rcortiz@simon logs]$ sudo grep 14:17:31 * | wc -l
223702
[rcortiz@simon logs]$ sudo grep 14:17:40 * | wc -l
194462
[rcortiz@simon logs]$ sudo grep 14:17:50 * | wc -l
31053
Whenever I add the OMFile setinngs back it gives me better results, but still
drops a significant amount
[rcortiz@simon logs]$ sudo grep 14:49:00 * | wc -l
242390
[rcortiz@simon logs]$ sudo grep 14:49:10 * | wc -l
242167
[rcortiz@simon logs]$ sudo grep 14:49:11 * | wc -l
245970
[rcortiz@simon logs]$ sudo grep 14:49:12 * | wc -l
245712
[rcortiz@simon logs]$ sudo grep 14:49:13 * | wc -l
243364
[rcortiz@simon logs]$ sudo grep 14:49:14 * | wc -l
^[[A242561
[rcortiz@simon logs]$ sudo grep 14:49:20 * | wc -l
242729
[rcortiz@simon logs]$ sudo grep 14:49:21 * | wc -l
243464
[rcortiz@simon logs]$ sudo grep 14:49:22 * | wc -l
243144
[rcortiz@simon logs]$ sudo grep 14:49:25 * | wc -l
246727
[rcortiz@simon logs]$ sudo grep 14:49:26 * | wc -l
243934
[rcortiz@simon logs]$ sudo grep 14:49:27 * | wc -l
246269
[rcortiz@simon logs]$ sudo grep 14:49:28 * | wc -l
245634
[rcortiz@simon logs]$ sudo grep 14:49:29 * | wc -l
238512
[rcortiz@simon logs]$ sudo grep 14:49:30 * | wc -l
240527
[rcortiz@simon logs]$ sudo grep 14:49:31 * | wc -l
242542
[rcortiz@simon logs]$ sudo grep 14:50:00 * | wc -l
243648
[rcortiz@simon logs]$ sudo grep 14:50:01 * | wc -l
241878
[rcortiz@simon logs]$ sudo grep 14:50:02 * | wc -l
238950
[rcortiz@simon logs]$ sudo grep 14:50:03 * | wc -l
247795
[rcortiz@simon logs]$
I have attached my mpstats, my overall cpu stats, and the rsyslog-stats from
the impstats module.
Another question I had was: when I write these Action and OMfile settings in
the legacy syntax I think this is right, but what is the right way of doing so
with the new syntax?...not sure how to add the queue sizes?
**************************************
#$ActionQueueWorkerThreads 8
#$ActionQueueSize 10000000
#$ActionQueueType FixedArray #LinkedList - use asynchronous processing
$OMFileAsyncWriting on
$OMFileFlushOnTXEnd on
$OMFileIOBufferSize 1000k
:hostname, contains, "fdfw"/logs/f_fw
**************************************
#:hostname, contains, "fdfw" action(type="omfile"
DirCreateMode="0700"
FileCreateMode="0644"
File="/logs/f_fw"
AsyncWriting="on"
FlushOnTXEnd="on"
IOBufferSize="1000k")
****************************************
thanks for the help everyone
Robert.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
