On Thu, Sep 5, 2013 at 11:02 PM, David Lang <[email protected]> wrote: > On Thu, 5 Sep 2013, Robert Ortiz wrote: > > Hello everyone, >> >> I upgraded to 7.4.3, and as everyone mentioned (David) it improved >> significantly, I am at 110k mps, which is great, but I think it can do >> better, currently rsyslog nice level is at its max, and I am curious if >> there was anything that I could do to bring it up to 200k mps? this is my >> current config file, I've tweaked it around and removed unnecessary things, >> but I am wondering if anyone can see if there is something that I am >> missing or something I need to unmark to improve the performance, >> > > As Rainer says, we need to find out where the bottleneck is, if you can > run top, hit 'H' so that it shows threads, and then load it down and tell > us which thread is hitting max CPU it would tell us where to look > > everything below is specualtion and places that can possibly be improved, > but if that is not the bottleneck, it won't make any difference unless the > system as a whole is running out of CPU. > > setting the timerequery for UDP will help if the bottleneck is there. > > are you disabling DNS lookups at startup (the -x flag on rsyslogd), this > is a very significant win if you can do it. If you have not done this, you > need to check and see if your DNS server is the bottleneck. >
With v7, this should not affect performance very much. Do you have other experience? > > I suggest using queue type FixedArray instead of LinkedList. It will use > the max amount of memory all the time, but my testing showed that this was > faster than LinkedList (which makes sense, since rsyslog doesn't need to > manipulate the list and bounce all over memory to access the queue) > > you have the zip compression set to 9, which is the most CPU intensive > setting, if the output ends up being your bottleneck, try reducing this and > see if things improve. > > not knowing your system names, if you can change any of the tests from > 'contains' to 'startswith', that could improve things. > > if your output is the bottleneck, you can split the outputs to use > multiple queues instead of all being part of one queue (add another set of > queue parameters about midway through the tests) > > The action queue parameter that Rainier suggested may help with this as > well, if you cna use multiple threads to read from the queue and output the > messages this can be as good or better than using multiple queues. > > Be careful: I was talking about the main message queue. If I don't overlook anything, the action queue settings are not very relevant here. > Rainer, am I correct in reading this tht the actionqueue will be used for > all of the output? I know that using the v7 config you cannot set an action > queue to be used for more than one output (without resorting to rulesets), > but I think that in the earlier config language the queue settings remain > until they are set to something else. > I think these are all auto-reset parameters, so they only apply to the first output. I have not checked the code, mostly because I really think the action queue parameters make a big difference here. I have written an in-depth blog about those tuning parameters some time ago. It can be found here: http://blog.gerhards.net/2013/06/rsyslog-performance-main-and-action.html Also, I think the $OMfile parameters also apply only to the first file - but I may be wrong. My strong recommendation is to use new-style config in complex cases, as it is then much easier to see what is actually going on (after all, that's why the new style was implemented ;)). > > > looking through the config, I strongly suspect that your bottleneck is in > writing the files. You have all of the writing being done with a single > thread, and that thread is compressing the logs, so it has a lot of work to > do. > yup, I guess it is the filters and running the main queue on multiple threads will take care of that. > > one thing that may help is if you turn on asyncwriting, that will decouple > the filtering and creation of the string to be written from the process of > writing it. Since you are doing compression as you are writing it, this > should move the compression out to a separate thread (or possibly a > separate thread per output??? Rainer? the background writer thread is per action instance. Compression is done on the background writer threads - in other words: "yes". But again, I doubt the parameters apply to all actions. Rainer ) instead of it all being done in the worker thread. > > David Lang > > > > Thanks in advance >> >> >> # For more information see /usr/share/doc/rsyslog-*/**rsyslog_conf.html >> # If you experience problems, see http://www.rsyslog.com/doc/** >> troubleshoot.html <http://www.rsyslog.com/doc/troubleshoot.html> >> >> #### MODULES #### >> >> module(load="imuxsock") # needs to be done just once Roberto 8-9-13 >> SysSock.FlowControl=(:"on") # enable flow control (use if needed) Roberto >> 8-9-13 >> #$ModLoad imuxsock # provides support for local system logging (e.g. via >> logger command) >> >> module(load="imklog") >> #$ModLoad imklog # provides kernel logging support (previously done by >> rklogd) >> $ModLoad immark # provides --MARK-- message capability >> >> # Provides UDP syslog reception >> module(load="imudp") # needs to be done just once >> input(type="imudp" port="514") >> #$ModLoad imudp >> #$UDPServerRun 514 >> #$UDPServerTimeRequery 1000000 >> #$UDPServerTimeRequery 10 - Gil 06/06/13 >> >> # Provides TCP syslog reception >> #$ModLoad imptcp >> #$InputTCPServerRun 514 >> >> >> #### GLOBAL DIRECTIVES #### >> >> # Use default timestamp format >> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat >> >> # File syncing capability is disabled by default. This feature is usually >> not required, >> # not useful and an extreme performance hit >> #$ActionFileEnableSync on >> >> # Include all config files in /etc/rsyslog.d/ >> $IncludeConfig /etc/rsyslog.d/*.conf >> >> # Set Buffer Size - default is 4k >> #$OMFileIOBufferSize 128k # - Gil 06/06/13 >> $OMFileAsyncWriting on >> $OMFileFlushOnTXEnd on >> $OMFileFlushInterval 1 >> #$OMFileZipLevel 9 >> $OMFileIOBufferSize 1000k >> >> #Turn on Main Ruleset Roberto 8-20-13 >> #$RulesetCreateMainQueue on >> >> >> # Set Main Message Queue Size - default is 10000 >> $MainMsgQueueSize 20000000 # Roberto 8-9-13 >> $InputUDPMaxSessions 40000000 >> $MainMsgQueueDequeueBatchSize 500000 >> >> >> #### RULES #### >> >> # Log all kernel messages to the console. >> # Logging much else clutters up the screen. >> #kern.* >> # /dev/console >> >> #Specific ruleset for remote messages >> #$Ruleset <name> >> >> #*.* /var/log/test/f_all #Roberto 8-21-13 >> #Module (load="builtin:omfile") >> #*.* action(type="omfile" >> # DirCreateMode="0700" >> # FileCreateMode="0644" >> # File="/var/log/test/alllogs") >> >> >> #switch back to default ruleset >> #$Ruleset RSYSLOG DefaultRuleset >> >> # Begin action Roberto 8-20-13 >> # $ActionOmrulesetRulesetName somename >> $ActionQueueWorkerThreads 5000000 >> $ActionQueueSize 10000000 >> #$ActionQueueType LinkedList # use asynchronous processing >> #$ActionQueueFileName aaaaafwd # set file name, also enables disk mode >> #$ActionQueueMaxFileSize 400m # default: 1m, should be 1% of MaxDiskSpace >> #$ActionQueueMaxDiskSpace 40g # space limit (use as much as possible) >> #$ActionQueueTimeoutEnqueue 0 # throtteling, 0 disables throttling and >> discard immediately if queue is full >> #$ActionQueueDequeueBatchSize 500000 >> #$ActionResumeRetryCount -1 # infinite retries on insert failure >> #$ActionResumeInterval 1 # faster than default 30 second delay >> #$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down >> >> >> >> :hostname, contains, "fdfw" /var/log/test/f_fw >> :hostname, contains, "mail" /var/log/test/f_mail >> :hostname, contains, "shib" /var/log/test/f_shib >> :hostname, contains, "pdc" /var/log/test/f_ad >> :hostname, contains, "networks" /var/log/test/f_networks >> :hostname, contains, "rout" /var/log/test/f_router >> :hostname, contains, "vm" /var/log/test/f_vm >> :hostname, contains, "pix" /var/log/test/f_pix >> >> >> >> # Log anything (except mail) of level info or higher. >> # Don't log private authentication messages! >> #*.info;mail.none;authpriv.**none;cron.none /var/log/messages >> #*.debug /var/log/messages >> >> # The authpriv file has restricted access. >> #authpriv.* /var/log/secure >> >> # Log all the mail messages in one place. >> #mail.* -/var/log/maillog >> >> >> # Log cron stuff >> #cron.* /var/log/cron >> >> # Everybody gets emergency messages >> #*.emerg * >> >> # Save news errors of level crit and higher in a special file. >> #uucp,news.crit /var/log/spooler >> >> >> # ### begin forwarding rule ### >> # The statement between the begin ... end define a SINGLE forwarding >> # rule. They belong together, do NOT split them. If you create multiple >> # forwarding rules, duplicate the whole block! >> # Remote Logging (we use TCP for reliable delivery) >> # >> # An on-disk queue is created for this action. If the remote host is >> # down, messages are spooled to disk and sent when it is up again. >> #$WorkDirectory /var/lib/rsyslog # where to place spool files >> #$ActionQueueFileName fwdRule1 # unique name prefix for spool files >> #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) >> #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown >> #$ActionQueueType LinkedList # run asynchronously >> #$ActionResumeRetryCount -1 # infinite retries if host is down >> # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional >> #*.* @@remote-host:514 >> # ### end of the forwarding rule ### >> (END) >> >> >> >> >> Robert. >> ______________________________**_________________ >> rsyslog mailing list >> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> >> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >> ______________________________**_________________ > rsyslog mailing list > http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> > http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
