I am using:
rsyslogd 8.4.2.ad1, compiled with:
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: No
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
Number of Bits in RainerScript integers: 32 (due to too-old json-c lib)
It is handling a very high volume but syslog() blocks and the blocking time
is not constant, so as I add more processes that are making a syslog() call
the blocking time increases by 10usec per process on my machine.
On Fri, Nov 28, 2014 at 4:50 PM, David Lang <[email protected]> wrote:
> what version of rsyslog are you running?
>
> rsyslog should be able to keep up with a very high volume of logs being
> delivered to /dev/log, but there are significant improvements between the
> version of rsyslog that was shipped in 12.05 (which I think was 5.x) and
> the current 8.x versions
>
> David Lang
>
>
> On Fri, 28 Nov 2014, Mehrdad Pazooki wrote:
>
> Date: Fri, 28 Nov 2014 10:14:30 -0500
>> From: Mehrdad Pazooki <[email protected]>
>> Reply-To: rsyslog-users <[email protected]>
>> To: rsyslog-users <[email protected]>
>> Subject: Re: [rsyslog] Calling syslog() is slow - it takes +100usec
>>
>>
>> There is no systemd on this platform.
>>
>> More details about the platform:
>>
>> Description: Ubuntu 12.04.1 LTS
>> Release: 12.04
>> Codename: precise
>>
>> Linux 3.13.0-39-generic #66-Ubuntu SMP
>>
>>
>> On Fri, Nov 28, 2014 at 7:14 AM, Mehrdad Pazooki <[email protected]>
>> wrote:
>>
>> It's running in Ubuntu (precise). I'll check it for systemd. I think
>>> syslog call is blocking because of /dev/log . if I use UDP locally, will
>>> that bypass /Dev/log?
>>>
>>>
>>> On Fri, Nov 28, 2014, 04:08 Rainer Gerhards <[email protected]>
>>> wrote:
>>>
>>> which platform is this running on? Is systemd journal active?
>>>>
>>>> Rainer
>>>>
>>>> 2014-11-27 22:37 GMT+01:00 Mehrdad Pazooki <[email protected]>:
>>>>
>>>> Hi,
>>>>>
>>>>> I am calling syslog in c++ using syslog.h . the time it takes on each
>>>>>
>>>> call
>>>>
>>>>> is too long. I am comparing it with IPC Queue and there is almost
>>>>>
>>>> 100usec
>>>>
>>>>> overhead when I call syslog() vs ipc.
>>>>> I did some benchmarking, when I use a single process for hitting syslog
>>>>>
>>>> the
>>>>
>>>>> performance is great it stays at avg 11usec/record for 1m records on my
>>>>> system. but if I run 10 of these processes the time will increase to
>>>>>
>>>> about
>>>>
>>>>> 100usec/record.
>>>>>
>>>>> Another problem is that memory usage remains really low ~0% so that
>>>>>
>>>> tells
>>>>
>>>>> me syslog() is somehow blocking and using cpu rather than do a set an
>>>>> forget to the queue and process it later on. another issue is that all
>>>>> workers are not engage. I see 5 workers but only 3 are working.
>>>>> The way I'm using it in prod is about 500 threads hit syslog(). I need
>>>>>
>>>> to
>>>>
>>>>> keep the time it takes for a syslog() call to a constant of 10usec
>>>>>
>>>> ideal to
>>>>
>>>>> max 40usec.
>>>>> Also, I am not doing openlog() before calling syslog()
>>>>>
>>>>> This is my configuration:
>>>>>
>>>>> # /etc/rsyslog.conf Configuration file for rsyslog.
>>>>> #
>>>>> # For more information see
>>>>> # /usr/share/doc/rsyslog-doc/
>>>>> html/rsyslog_conf.html
>>>>> #
>>>>> # Default logging rules can be found in /etc/rsyslog.d/50-default.conf
>>>>>
>>>>>
>>>>> #################
>>>>> #### MODULES ####
>>>>> #################
>>>>> $ModLoad imuxsock # provides support for local system logging
>>>>> $ModLoad imklog # provides kernel logging support (previously done by
>>>>> rklogd)
>>>>> #$ModLoad immark # provides --MARK-- message capability
>>>>>
>>>>> # provides UDP syslog reception
>>>>> #$ModLoad imudp
>>>>> #$UDPServerRun 514
>>>>>
>>>>> # provides TCP syslog reception
>>>>> #$ModLoad imtcp
>>>>> #$InputTCPServerRun 514
>>>>>
>>>>>
>>>>> # For debug only
>>>>> # module(load="impstats" interval="1" severity="8")
>>>>>
>>>>> ###########################
>>>>> #### GLOBAL DIRECTIVES ####
>>>>> ###########################
>>>>>
>>>>> #
>>>>> # Use traditional timestamp format.
>>>>> # To enable high precision timestamps, comment out the following line.
>>>>> #
>>>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>>>>
>>>>> # Filter duplicated messages
>>>>> $RepeatedMsgReduction off
>>>>>
>>>>> #
>>>>> # Set the default permissions for all log files.
>>>>> #
>>>>> $FileOwner root
>>>>> $FileGroup adm
>>>>> $FileCreateMode 0640
>>>>> $DirCreateMode 0755
>>>>> $Umask 0022
>>>>> $PrivDropToUser root
>>>>> $PrivDropToGroup root
>>>>>
>>>>> #
>>>>> # Where to place spool files
>>>>> #
>>>>> $WorkDirectory /var/spool/rsyslog
>>>>>
>>>>> $ActionResumeRetryCount -1 # infinite
>>>>>
>>>> retries if
>>>>
>>>>> host is down
>>>>> $SystemLogRateLimitInterval 0
>>>>> $SystemLogRateLimitBurst 0
>>>>> $MaxMessageSize 4k
>>>>>
>>>>> main_queue(
>>>>> queue.Type="LinkedList"
>>>>> queue.Size="1000000"
>>>>> queue.workerThreads="8"
>>>>> queue.workerThreadMinimumMessages="10000"
>>>>> queue.SaveOnShutdown="on"
>>>>> )
>>>>>
>>>>>
>>>>>
>>>>> template(name="localone" type="string" string="%msg:2:$%\n") # ":2:$"
>>>>>
>>>> -->
>>>>
>>>>> tells rsyslog to ignore the leading char, b/c rsyslog adds a leading
>>>>>
>>>> space
>>>>
>>>>> to every message
>>>>>
>>>>> if $syslogfacility-text == 'local1' then {
>>>>> action(
>>>>> type="omfile"
>>>>> template="localone"
>>>>> file="/var/log/local1.log"
>>>>> flushOnTXEnd="off"
>>>>> asyncWriting="on"
>>>>> ioBufferSize="64k"
>>>>> queue.type="LinkedList"
>>>>> queue.size="1000000"
>>>>> queue.dequeueBatchSize="4096"
>>>>> queue.workerThreads="8"
>>>>> queue.workerThreadMinimumMessages="10000"
>>>>> queue.SaveOnShutdown="on"
>>>>> action.ResumeRetryCount="-1"
>>>>> )
>>>>> stop
>>>>> }
>>>>>
>>>>> #if $syslogtag contains 'rsyslogd-pstats' then {
>>>>> # action(
>>>>> # type="omfile"
>>>>> # queue.type="LinkedList"
>>>>> # queue.discardmark="1000"
>>>>> # name="pstats"
>>>>> # file="/var/log/pstats.log"
>>>>> # )
>>>>> # stop
>>>>> #}
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>> myriad
>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>>> DON'T LIKE THAT.
>>>>>
>>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>> DON'T LIKE THAT.
>>>>
>>>>
>>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
