Thanks Rainer, David, and Radu for the input. After reading Rainer's blog about turning lanes and understand rsyslogs queues, it makes more sense, thanks for the articles,
I changed the following queues in the conf file to this: $MainMsgQueueType FixedArray $MainMsgQueueSize 20000000 ***Regarding this entry I was curious what is the appropiate way to define the queue size, like it is here, or in MB, or Gs? $MainMsgQueueWorkerThreads 8 # $MainMsgQueueWorkerTimeoutThreadShutdown -1 $MainMsgQueueDequeueBatchSize 1000 # $MainMsgQueueSaveOnShutdown on $InputUDPMaxSessions 40000000 >From the articles and the feedback I was not sure whether I should defien the >Action queue size and worker theads or not? # $ActionOmrulesetRulesetName somename $ActionQueueWorkerThreads 8 $ActionQueueSize 10000000 $ActionQueueType FixedArray #LinkedList - use asynchronous processing # $ActionQueueDequeueBatchSize 500000 # $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down Should the MainMsg and Action dequeue batch sizes be set to the same entry? And lastly I actually do not want to compress while filtering, and I thought that the $OMFile settings where there to write to a flat file? How should I modify my conf to not actually compress any logs? $OMFileAsyncWriting on $OMFileFlushOnTXEnd on $OMFileFlushInterval 1 # $OMFileZipLevel 9 $OMFileIOBufferSize 1000k Thanks Robert > ----- Original Message ----- > From: Rainer Gerhards > Sent: 09/06/13 05:35 AM > To: rsyslog-users > Subject: Re: [rsyslog] perfomance tweaking > > On Thu, Sep 5, 2013 at 11:02 PM, David Lang <[email protected]> wrote: > > > On Thu, 5 Sep 2013, Robert Ortiz wrote: > > > > Hello everyone, > >> > >> I upgraded to 7.4.3, and as everyone mentioned (David) it improved > >> significantly, I am at 110k mps, which is great, but I think it can do > >> better, currently rsyslog nice level is at its max, and I am curious if > >> there was anything that I could do to bring it up to 200k mps? this is my > >> current config file, I've tweaked it around and removed unnecessary things, > >> but I am wondering if anyone can see if there is something that I am > >> missing or something I need to unmark to improve the performance, > >> > > > > As Rainer says, we need to find out where the bottleneck is, if you can > > run top, hit 'H' so that it shows threads, and then load it down and tell > > us which thread is hitting max CPU it would tell us where to look > > > > everything below is specualtion and places that can possibly be improved, > > but if that is not the bottleneck, it won't make any difference unless the > > system as a whole is running out of CPU. > > > > setting the timerequery for UDP will help if the bottleneck is there. > > > > are you disabling DNS lookups at startup (the -x flag on rsyslogd), this > > is a very significant win if you can do it. If you have not done this, you > > need to check and see if your DNS server is the bottleneck. > > > > With v7, this should not affect performance very much. Do you have other > experience? > > > > > > I suggest using queue type FixedArray instead of LinkedList. It will use > > the max amount of memory all the time, but my testing showed that this was > > faster than LinkedList (which makes sense, since rsyslog doesn't need to > > manipulate the list and bounce all over memory to access the queue) > > > > you have the zip compression set to 9, which is the most CPU intensive > > setting, if the output ends up being your bottleneck, try reducing this and > > see if things improve. > > > > not knowing your system names, if you can change any of the tests from > > 'contains' to 'startswith', that could improve things. > > > > if your output is the bottleneck, you can split the outputs to use > > multiple queues instead of all being part of one queue (add another set of > > queue parameters about midway through the tests) > > > > The action queue parameter that Rainier suggested may help with this as > > well, if you cna use multiple threads to read from the queue and output the > > messages this can be as good or better than using multiple queues. > > > > > Be careful: I was talking about the main message queue. If I don't overlook > anything, the action queue settings are not very relevant here. > > > > Rainer, am I correct in reading this tht the actionqueue will be used for > > all of the output? I know that using the v7 config you cannot set an action > > queue to be used for more than one output (without resorting to rulesets), > > but I think that in the earlier config language the queue settings remain > > until they are set to something else. > > > > I think these are all auto-reset parameters, so they only apply to the > first output. I have not checked the code, mostly because I really think > the action queue parameters make a big difference here. > > I have written an in-depth blog about those tuning parameters some time > ago. It can be found here: > > http://blog.gerhards.net/2013/06/rsyslog-performance-main-and-action.html > > Also, I think the $OMfile parameters also apply only to the first file - > but I may be wrong. My strong recommendation is to use new-style config in > complex cases, as it is then much easier to see what is actually going on > (after all, that's why the new style was implemented ;)). > > > > > > > > looking through the config, I strongly suspect that your bottleneck is in > > writing the files. You have all of the writing being done with a single > > thread, and that thread is compressing the logs, so it has a lot of work to > > do. > > > > yup, I guess it is the filters and running the main queue on multiple > threads will take care of that. > > > > > > one thing that may help is if you turn on asyncwriting, that will decouple > > the filtering and creation of the string to be written from the process of > > writing it. Since you are doing compression as you are writing it, this > > should move the compression out to a separate thread (or possibly a > > separate thread per output??? Rainer? > > > the background writer thread is per action instance. Compression is done on > the background writer threads - in other words: "yes". But again, I doubt > the parameters apply to all actions. > > Rainer > > ) instead of it all being done in the worker thread. > > > > David Lang > > > > > > > > Thanks in advance > >> > >> > >> # For more information see /usr/share/doc/rsyslog-*/**rsyslog_conf.html > >> # If you experience problems, see http://www.rsyslog.com/doc/** > >> troubleshoot.html <http://www.rsyslog.com/doc/troubleshoot.html> > >> > >> #### MODULES #### > >> > >> module(load="imuxsock") # needs to be done just once Roberto 8-9-13 > >> SysSock.FlowControl=(:"on") # enable flow control (use if needed) Roberto > >> 8-9-13 > >> #$ModLoad imuxsock # provides support for local system logging (e.g. via > >> logger command) > >> > >> module(load="imklog") > >> #$ModLoad imklog # provides kernel logging support (previously done by > >> rklogd) > >> $ModLoad immark # provides --MARK-- message capability > >> > >> # Provides UDP syslog reception > >> module(load="imudp") # needs to be done just once > >> input(type="imudp" port="514") > >> #$ModLoad imudp > >> #$UDPServerRun 514 > >> #$UDPServerTimeRequery 1000000 > >> #$UDPServerTimeRequery 10 - Gil 06/06/13 > >> > >> # Provides TCP syslog reception > >> #$ModLoad imptcp > >> #$InputTCPServerRun 514 > >> > >> > >> #### GLOBAL DIRECTIVES #### > >> > >> # Use default timestamp format > >> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > >> > >> # File syncing capability is disabled by default. This feature is usually > >> not required, > >> # not useful and an extreme performance hit > >> #$ActionFileEnableSync on > >> > >> # Include all config files in /etc/rsyslog.d/ > >> $IncludeConfig /etc/rsyslog.d/*.conf > >> > >> # Set Buffer Size - default is 4k > >> #$OMFileIOBufferSize 128k # - Gil 06/06/13 > >> $OMFileAsyncWriting on > >> $OMFileFlushOnTXEnd on > >> $OMFileFlushInterval 1 > >> #$OMFileZipLevel 9 > >> $OMFileIOBufferSize 1000k > >> > >> #Turn on Main Ruleset Roberto 8-20-13 > >> #$RulesetCreateMainQueue on > >> > >> > >> # Set Main Message Queue Size - default is 10000 > >> $MainMsgQueueSize 20000000 # Roberto 8-9-13 > >> $InputUDPMaxSessions 40000000 > >> $MainMsgQueueDequeueBatchSize 500000 > >> > >> > >> #### RULES #### > >> > >> # Log all kernel messages to the console. > >> # Logging much else clutters up the screen. > >> #kern.* > >> # /dev/console > >> > >> #Specific ruleset for remote messages > >> #$Ruleset <name> > >> > >> #*.* /var/log/test/f_all #Roberto 8-21-13 > >> #Module (load="builtin:omfile") > >> #*.* action(type="omfile" > >> # DirCreateMode="0700" > >> # FileCreateMode="0644" > >> # File="/var/log/test/alllogs") > >> > >> > >> #switch back to default ruleset > >> #$Ruleset RSYSLOG DefaultRuleset > >> > >> # Begin action Roberto 8-20-13 > >> # $ActionOmrulesetRulesetName somename > >> $ActionQueueWorkerThreads 5000000 > >> $ActionQueueSize 10000000 > >> #$ActionQueueType LinkedList # use asynchronous processing > >> #$ActionQueueFileName aaaaafwd # set file name, also enables disk mode > >> #$ActionQueueMaxFileSize 400m # default: 1m, should be 1% of MaxDiskSpace > >> #$ActionQueueMaxDiskSpace 40g # space limit (use as much as possible) > >> #$ActionQueueTimeoutEnqueue 0 # throtteling, 0 disables throttling and > >> discard immediately if queue is full > >> #$ActionQueueDequeueBatchSize 500000 > >> #$ActionResumeRetryCount -1 # infinite retries on insert failure > >> #$ActionResumeInterval 1 # faster than default 30 second delay > >> #$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down > >> > >> > >> > >> :hostname, contains, "fdfw" /var/log/test/f_fw > >> :hostname, contains, "mail" /var/log/test/f_mail > >> :hostname, contains, "shib" /var/log/test/f_shib > >> :hostname, contains, "pdc" /var/log/test/f_ad > >> :hostname, contains, "networks" /var/log/test/f_networks > >> :hostname, contains, "rout" /var/log/test/f_router > >> :hostname, contains, "vm" /var/log/test/f_vm > >> :hostname, contains, "pix" /var/log/test/f_pix > >> > >> > >> > >> # Log anything (except mail) of level info or higher. > >> # Don't log private authentication messages! > >> #*.info;mail.none;authpriv.**none;cron.none /var/log/messages > >> #*.debug /var/log/messages > >> > >> # The authpriv file has restricted access. > >> #authpriv.* /var/log/secure > >> > >> # Log all the mail messages in one place. > >> #mail.* -/var/log/maillog > >> > >> > >> # Log cron stuff > >> #cron.* /var/log/cron > >> > >> # Everybody gets emergency messages > >> #*.emerg * > >> > >> # Save news errors of level crit and higher in a special file. > >> #uucp,news.crit /var/log/spooler > >> > >> > >> # ### begin forwarding rule ### > >> # The statement between the begin ... end define a SINGLE forwarding > >> # rule. They belong together, do NOT split them. If you create multiple > >> # forwarding rules, duplicate the whole block! > >> # Remote Logging (we use TCP for reliable delivery) > >> # > >> # An on-disk queue is created for this action. If the remote host is > >> # down, messages are spooled to disk and sent when it is up again. > >> #$WorkDirectory /var/lib/rsyslog # where to place spool files > >> #$ActionQueueFileName fwdRule1 # unique name prefix for spool files > >> #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) > >> #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown > >> #$ActionQueueType LinkedList # run asynchronously > >> #$ActionResumeRetryCount -1 # infinite retries if host is down > >> # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional > >> #*.* @@remote-host:514 > >> # ### end of the forwarding rule ### > >> (END) > >> > >> > >> > >> > >> Robert. > >> ______________________________**_________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> > >> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> DON'T LIKE THAT. > >> > >> ______________________________**_________________ > > rsyslog mailing list > > http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> > > http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
