Ok, looking at the stats, actions 2 and 3 are filling their queues, it may be
that the others aren't, only because with 2 threads, they are getting stuck
waiting behind these first two.
since rsyslog is keeping up on the main queue with only 2 threads, and the main
queue is now not doing much other than move the logs to the action queue, I
would experiment with the main queue worker thread count, try dropping it to 1
and see if that helps or hurts. if it hurts, try changing it to 3
one more data gathering thing for you to do
can you run
iostat -xk 10
during your load test? I want to just double check that you aren't running into
problems with a disk that can't keep up. If any of the drives show 100%
utilization (or close to it) then you may be having problems from that.
more comments inline:
On Wed, 25 Sep 2013, Robert wrote:
sure sorry I begin a new thread, but I should just reply to it: here is the
config:
# Run this in debugger mode to troubleshoot
module(load="impstats" interval="300" severity="7")
# # to actually gather the data:
syslog.=debug /var/log/rsyslog-stats
#/var/log/debugformat;RSYSLOG_DebugFormat
# rsyslog v7 configuration file
# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
#### MODULES ####
module(load="imuxsock") # needs to be done just once Roberto 8-9-13
#SysSock.FlowControl=(:"on") # enable flow control (use if needed) Roberto
8-9-13
#$ModLoad imuxsock # provides support for local system logging (e.g. via logger
command)
module(load="imklog")
#$ModLoad imklog # provides kernel logging support (previously done by rklogd)
#$ModLoad immark # provides --MARK-- message capability
# Provides UDP syslog reception
module(load="imudp"
TimeRequery="10"
SchedulingPolicy="fifo"
SchedulingPriority="10") # needs to be done just once Roberto 8-9-13
input(type="imudp" port="514") # Roberto 8-9-13
try increasing timerequery from 10 to 100
add batchsize="100"
these should reduce the cpu needed for imudp
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
# Set Main Message Queue Size - default is 10000
$MainMsgQueueType FixedArray #LinkedList
$MainMsgQueueSize 20000000
$MainMsgQueueWorkerThreads 2
$MainMsgQueueDequeueBatchSize 1000
$InputUDPMaxSessions 40000000
:hostname, contains, "fdfw" action(type="omfile"
DirCreateMode="0700"
FileCreateMode="0644"
File="/logs/f_fw"
AsyncWriting="on"
FlushOnTXEnd="on"
IOBufferSize="8k"
queue.type="FixedArray" queue.dequeuebatchsize="100" queue.size="10000"
)
try removing the AsyncWriting and FlushOnTXEnd parameters, see if that helps or
hurts. when just writing to a file with one of the built-in templates, I would
not expect it to help much, and the overhead of communicating with the separate
thread to do the write is probably higher than just writing directly.
If we find that disk I/O is a limiting factor, we will want to look at this
again and enable compression (ziplevel="9")
it would also be worth experimenting with slightly larger buffer sizes, does it
help or hurt if you go from 8k to 16k for example?
as rainer says, if you can avoid using 'contains' tests, it should speed up the
tests.
if your names are fdfw-1 or similar, changing the test to
:hostname, startswith, "fdfw" action(type="omfile"
would speed things up significantly. If the string you are searching for is not
at the beginning, then contains is what you need.
the best tests to use are (in order)
isequal / isempty
startswith
contains
regex
ereregex
As rainer says, if you know that a log line will only match one rule, tell
rsyslog this so it doesn't waste time checking other rules (and then you can
re-order rules so the most commonly matched rules are first). now, it may be
that putting each output in it's own queue defeats this.
:hostname, contains, "fdfw" {
action(type="omfile"
DirCreateMode="0700"
FileCreateMode="0644"
File="/logs/f_fw"
AsyncWriting="on"
FlushOnTXEnd="on"
IOBufferSize="8k"
queue.type="FixedArray" queue.dequeuebatchsize="100" queue.size="10000"
)
stop
}
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
