Hi Robert, That will make rsyslog do system calls to check the time once every 10 messages, instead of once every message. Here's a thread where David and Rainer explained the thing to me: http://comments.gmane.org/gmane.comp.sysutils.rsyslog/9924
Regarding threads, I think 2-8 should be the number for your action queue, but Rainer's suggestion would be to set that sort of number for the main queue as well, which does most of the work. Best regards, Radu 2013/9/5 <[email protected]> > thanks Radu, I will modify the setting, what will that do exactly? > > > ----- Original Message ----- > > From: Radu Gheorghe > > Sent: 09/05/13 12:39 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] perfomance tweaking > > > > Also, I would change the imudp initialization with this: > > > > module(load="imudp" > > TimeRequery="10") > > > > I see you have the TimeRequery option in old config format and commented. > > > > Best regards, > > Radu > > > > > > 2013/9/5 Rainer Gerhards <[email protected]> > > > > > IF I read correctly, you have not set the number of *main message* > queue > > > worker threads. Depending on the # of cores you have, a number between > 2 > > > and 8 would probably be a good fit (but don't set it extremely large, > that > > > would hurt much if it is really used...). > > > > > > For better info, we would need to see where the time is actually > spent, but > > > I guess the worker change should already do what you want. > > > > > > Rainer > > > > > > > > > On Thu, Sep 5, 2013 at 5:35 PM, Robert Ortiz <[email protected]> wrote: > > > > > > > Hello everyone, > > > > > > > > I upgraded to 7.4.3, and as everyone mentioned (David) it improved > > > > significantly, I am at 110k mps, which is great, but I think it can > do > > > > better, currently rsyslog nice level is at its max, and I am curious > if > > > > there was anything that I could do to bring it up to 200k mps? this > is my > > > > current config file, I've tweaked it around and removed unnecessary > > > things, > > > > but I am wondering if anyone can see if there is something that I am > > > > missing or something I need to unmark to improve the performance, > > > > > > > > Thanks in advance > > > > > > > > > > > > # For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html > > > > # If you experience problems, see > > > > http://www.rsyslog.com/doc/troubleshoot.html > > > > > > > > #### MODULES #### > > > > > > > > module(load="imuxsock") # needs to be done just once Roberto 8-9-13 > > > > SysSock.FlowControl=(:"on") # enable flow control (use if needed) > Roberto > > > > 8-9-13 > > > > #$ModLoad imuxsock # provides support for local system logging (e.g. > via > > > > logger command) > > > > > > > > module(load="imklog") > > > > #$ModLoad imklog # provides kernel logging support (previously done > by > > > > rklogd) > > > > $ModLoad immark # provides --MARK-- message capability > > > > > > > > # Provides UDP syslog reception > > > > module(load="imudp") # needs to be done just once > > > > input(type="imudp" port="514") > > > > #$ModLoad imudp > > > > #$UDPServerRun 514 > > > > #$UDPServerTimeRequery 1000000 > > > > #$UDPServerTimeRequery 10 - Gil 06/06/13 > > > > > > > > # Provides TCP syslog reception > > > > #$ModLoad imptcp > > > > #$InputTCPServerRun 514 > > > > > > > > > > > > #### GLOBAL DIRECTIVES #### > > > > > > > > # Use default timestamp format > > > > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > > > > > > > > # File syncing capability is disabled by default. This feature is > usually > > > > not required, > > > > # not useful and an extreme performance hit > > > > #$ActionFileEnableSync on > > > > > > > > # Include all config files in /etc/rsyslog.d/ > > > > $IncludeConfig /etc/rsyslog.d/*.conf > > > > > > > > # Set Buffer Size - default is 4k > > > > #$OMFileIOBufferSize 128k # - Gil 06/06/13 > > > > $OMFileAsyncWriting on > > > > $OMFileFlushOnTXEnd on > > > > $OMFileFlushInterval 1 > > > > #$OMFileZipLevel 9 > > > > $OMFileIOBufferSize 1000k > > > > > > > > #Turn on Main Ruleset Roberto 8-20-13 > > > > #$RulesetCreateMainQueue on > > > > > > > > > > > > # Set Main Message Queue Size - default is 10000 > > > > $MainMsgQueueSize 20000000 # Roberto 8-9-13 > > > > $InputUDPMaxSessions 40000000 > > > > $MainMsgQueueDequeueBatchSize 500000 > > > > > > > > > > > > #### RULES #### > > > > > > > > # Log all kernel messages to the console. > > > > # Logging much else clutters up the screen. > > > > #kern.* > > > > # /dev/console > > > > > > > > #Specific ruleset for remote messages > > > > #$Ruleset <name> > > > > > > > > #*.* /var/log/test/f_all #Roberto 8-21-13 > > > > #Module (load="builtin:omfile") > > > > #*.* action(type="omfile" > > > > # DirCreateMode="0700" > > > > # FileCreateMode="0644" > > > > # File="/var/log/test/alllogs") > > > > > > > > > > > > #switch back to default ruleset > > > > #$Ruleset RSYSLOG DefaultRuleset > > > > > > > > # Begin action Roberto 8-20-13 > > > > # $ActionOmrulesetRulesetName somename > > > > $ActionQueueWorkerThreads 5000000 > > > > $ActionQueueSize 10000000 > > > > #$ActionQueueType LinkedList # use asynchronous processing > > > > #$ActionQueueFileName aaaaafwd # set file name, also enables disk > mode > > > > #$ActionQueueMaxFileSize 400m # default: 1m, should be 1% of > MaxDiskSpace > > > > #$ActionQueueMaxDiskSpace 40g # space limit (use as much as possible) > > > > #$ActionQueueTimeoutEnqueue 0 # throtteling, 0 disables throttling > and > > > > discard immediately if queue is full > > > > #$ActionQueueDequeueBatchSize 500000 > > > > #$ActionResumeRetryCount -1 # infinite retries on insert failure > > > > #$ActionResumeInterval 1 # faster than default 30 second delay > > > > #$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts > > > down > > > > > > > > > > > > > > > > :hostname, contains, "fdfw" /var/log/test/f_fw > > > > :hostname, contains, "mail" /var/log/test/f_mail > > > > :hostname, contains, "shib" /var/log/test/f_shib > > > > :hostname, contains, "pdc" /var/log/test/f_ad > > > > :hostname, contains, "networks" /var/log/test/f_networks > > > > :hostname, contains, "rout" /var/log/test/f_router > > > > :hostname, contains, "vm" /var/log/test/f_vm > > > > :hostname, contains, "pix" /var/log/test/f_pix > > > > > > > > > > > > > > > > # Log anything (except mail) of level info or higher. > > > > # Don't log private authentication messages! > > > > #*.info;mail.none;authpriv.none;cron.none /var/log/messages > > > > #*.debug /var/log/messages > > > > > > > > # The authpriv file has restricted access. > > > > #authpriv.* /var/log/secure > > > > > > > > # Log all the mail messages in one place. > > > > #mail.* -/var/log/maillog > > > > > > > > > > > > # Log cron stuff > > > > #cron.* /var/log/cron > > > > > > > > # Everybody gets emergency messages > > > > #*.emerg * > > > > > > > > # Save news errors of level crit and higher in a special file. > > > > #uucp,news.crit /var/log/spooler > > > > > > > > > > > > # ### begin forwarding rule ### > > > > # The statement between the begin ... end define a SINGLE forwarding > > > > # rule. They belong together, do NOT split them. If you create > multiple > > > > # forwarding rules, duplicate the whole block! > > > > # Remote Logging (we use TCP for reliable delivery) > > > > # > > > > # An on-disk queue is created for this action. If the remote host is > > > > # down, messages are spooled to disk and sent when it is up again. > > > > #$WorkDirectory /var/lib/rsyslog # where to place spool files > > > > #$ActionQueueFileName fwdRule1 # unique name prefix for spool files > > > > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as > possible) > > > > #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown > > > > #$ActionQueueType LinkedList # run asynchronously > > > > #$ActionResumeRetryCount -1 # infinite retries if host is down > > > > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional > > > > #*.* @@remote-host:514 > > > > # ### end of the forwarding rule ### > > > > (END) > > > > > > > > > > > > > > > > > > > > Robert. > > > > _______________________________________________ > > > > rsyslog mailing list > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > http://www.rsyslog.com/professional-services/ > > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > > > > DON'T LIKE THAT. > > > > > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com/professional-services/ > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > > DON'T LIKE THAT. > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
