what version of rsyslog are you running?
rsyslog should be able to keep up with a very high volume of logs being
delivered to /dev/log, but there are significant improvements between the
version of rsyslog that was shipped in 12.05 (which I think was 5.x) and the
current 8.x versions
David Lang
On Fri, 28 Nov 2014, Mehrdad Pazooki wrote:
Date: Fri, 28 Nov 2014 10:14:30 -0500
From: Mehrdad Pazooki <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] Calling syslog() is slow - it takes +100usec
There is no systemd on this platform.
More details about the platform:
Description: Ubuntu 12.04.1 LTS
Release: 12.04
Codename: precise
Linux 3.13.0-39-generic #66-Ubuntu SMP
On Fri, Nov 28, 2014 at 7:14 AM, Mehrdad Pazooki <[email protected]>
wrote:
It's running in Ubuntu (precise). I'll check it for systemd. I think
syslog call is blocking because of /dev/log . if I use UDP locally, will
that bypass /Dev/log?
On Fri, Nov 28, 2014, 04:08 Rainer Gerhards <[email protected]>
wrote:
which platform is this running on? Is systemd journal active?
Rainer
2014-11-27 22:37 GMT+01:00 Mehrdad Pazooki <[email protected]>:
Hi,
I am calling syslog in c++ using syslog.h . the time it takes on each
call
is too long. I am comparing it with IPC Queue and there is almost
100usec
overhead when I call syslog() vs ipc.
I did some benchmarking, when I use a single process for hitting syslog
the
performance is great it stays at avg 11usec/record for 1m records on my
system. but if I run 10 of these processes the time will increase to
about
100usec/record.
Another problem is that memory usage remains really low ~0% so that
tells
me syslog() is somehow blocking and using cpu rather than do a set an
forget to the queue and process it later on. another issue is that all
workers are not engage. I see 5 workers but only 3 are working.
The way I'm using it in prod is about 500 threads hit syslog(). I need
to
keep the time it takes for a syslog() call to a constant of 10usec
ideal to
max 40usec.
Also, I am not doing openlog() before calling syslog()
This is my configuration:
# /etc/rsyslog.conf Configuration file for rsyslog.
#
# For more information see
# /usr/share/doc/rsyslog-doc/
html/rsyslog_conf.html
#
# Default logging rules can be found in /etc/rsyslog.d/50-default.conf
#################
#### MODULES ####
#################
$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog # provides kernel logging support (previously done by
rklogd)
#$ModLoad immark # provides --MARK-- message capability
# provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514
# provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514
# For debug only
# module(load="impstats" interval="1" severity="8")
###########################
#### GLOBAL DIRECTIVES ####
###########################
#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# Filter duplicated messages
$RepeatedMsgReduction off
#
# Set the default permissions for all log files.
#
$FileOwner root
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser root
$PrivDropToGroup root
#
# Where to place spool files
#
$WorkDirectory /var/spool/rsyslog
$ActionResumeRetryCount -1 # infinite
retries if
host is down
$SystemLogRateLimitInterval 0
$SystemLogRateLimitBurst 0
$MaxMessageSize 4k
main_queue(
queue.Type="LinkedList"
queue.Size="1000000"
queue.workerThreads="8"
queue.workerThreadMinimumMessages="10000"
queue.SaveOnShutdown="on"
)
template(name="localone" type="string" string="%msg:2:$%\n") # ":2:$"
-->
tells rsyslog to ignore the leading char, b/c rsyslog adds a leading
space
to every message
if $syslogfacility-text == 'local1' then {
action(
type="omfile"
template="localone"
file="/var/log/local1.log"
flushOnTXEnd="off"
asyncWriting="on"
ioBufferSize="64k"
queue.type="LinkedList"
queue.size="1000000"
queue.dequeueBatchSize="4096"
queue.workerThreads="8"
queue.workerThreadMinimumMessages="10000"
queue.SaveOnShutdown="on"
action.ResumeRetryCount="-1"
)
stop
}
#if $syslogtag contains 'rsyslogd-pstats' then {
# action(
# type="omfile"
# queue.type="LinkedList"
# queue.discardmark="1000"
# name="pstats"
# file="/var/log/pstats.log"
# )
# stop
#}
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
