On Wed, 17 Dec 2014, Marc Fournier wrote:
Excerpts from David Lang's message of 2014-12-16 20:41:49 +0100:
On Tue, 16 Dec 2014, Marc Fournier wrote:
Hello,
openldap writes its logs using syslog(3). Three log messages are emitted
for each ldap search operation.
On a given openldap setup, I reach about 2500 operations per second. The
simple fact of shutting rsyslog down icreases this number up to 5000
(logs are lost obviously).
Writing to a local file, to /dev/null, or to a remote server over UDP
doesn't significantly change this number. I tried tweaking various
settings related to the main queue but didn't see any huge difference.
ipmstats shows exactly the same numbers for "imuxsock: submitted" than
for "main Q: enqueued". All the discarded/full/failed/etc counters have
a value of 0.
This is rsyslog-8.6.0-2.el6.x86_64 from the adiscon RPM repo (but had
similar problems with version 5.8.10-9.el6_6 which comes with el6).
Here's the relevant bits of my config:
module(load="impstats" log.syslog="off" log.file="/var/log/impstats.log"
resetCounters="off" interval="5")
module(load="imuxsock" SysSock.RateLimit.Interval="0" SysSock.Annotate="off"
SysSock.FlowControl="off" SysSock.IgnoreTimestamp="off")
$MaxMessageSize 2k
$MainMsgQueueDequeueBatchSize 1024
$MainMsgQueueType LinkedList
$MainMsgQueueSize 100000
$MainMsgQueueWorkerThreads 1
$RepeatedMsgReduction off
$SystemLogRateLimitInterval 0
$ActionQueueType LinkedList
$ActionQueueSize 100000
$ActionQueueDiscardMark 95000
$ActionQueueDiscardSeverity 0
$ActionQueueTimeoutEnqueue 0
$ActionQueueDequeueSlowdown 1000
$ActionQueueWorkerThreads 2
$ActionQueueDequeueBatchSize 128
$ActionResumeRetryCount -1
#local4.* -/var/log/ldap/ldap.log
#local4.* ~
local4.* -/dev/null
#local4.* @localhost
So I'm wondering which one of rsyslog or /dev/log is actually the
bottleneck here and what can I do about it ?
/dev/log is just a socket to talk to rsyslog
One major problem here, why do you have ActionQueueDequeueSlowdown configured?
that deliberatly pauses before trying to process the next messages.
you really shouldn't need to have an action queue for writing to a local file,
you should remove all of the actionqueue parameters (and if you do decide that
you want a queue, use the new action() syntax, it makes it much clearer what's
happening)
Ok, right my config is a mess I agree ! Been iterating over it since
back when I used 5.8.10.
So I started over with a brand new config. The best results I got so far
was with these settings:
module(load="impstats" log.syslog="off" log.file="/var/log/impstats.log"
resetCounters="off" interval="5")
module(load="imuxsock" SysSock.RateLimit.Interval="0" SysSock.Annotate="off"
SysSock.FlowControl="off" SysSock.IgnoreTimestamp="off")
$MaxMessageSize 2k
$MainMsgQueueDequeueBatchSize 4096
$MainMsgQueueType LinkedList
$MainMsgQueueSize 250000
$MainMsgQueueWorkerThreads 4
$MainMsgQueueWorkerThreadMinimumMessages 100000
$RepeatedMsgReduction off
$SystemLogRateLimitInterval 0
*.* ~
The numbers are slightly better now: ~3500 ops/s vs 2500 yesterday. But
still a significant difference with the max I reach when running with
rsyslogd stopped (~5000 ops/s).
Is this an expected compromise ? Or are there possible improvements I
can do ?
I'm not seeing anything obviously wrong here, does impstats show anything
interesting (errors, etc)
I would actually reduce the worker threads down to 1 (it's counter-intuitive,
but having too many threads in rsyslog can actually slow things down because of
the overhead involved with locking and unlocking the queues)
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
