On Thu, Sep 26, 2013 at 12:31 AM, David Lang <[email protected]> wrote: > Ok, looking at the stats, actions 2 and 3 are filling their queues, it may > be that the others aren't, only because with 2 threads, they are getting > stuck waiting behind these first two. > > sometimes it seems to be hard to actually read what was written. I had overlooked that the actions still run on their own queues. I would strongly suggest to remove that (reasons see in my last posting). This should not be done with any other change, as that would make it hard to find what actually did a change.
> since rsyslog is keeping up on the main queue with only 2 threads, and the > main queue is now not doing much other than move the logs to the action > queue, Just to make sure this is understood: the heavy plumbing is still done on the main queue worker threads, as this is where the filters are evaluated. Rainer > I would experiment with the main queue worker thread count, try dropping > it to 1 and see if that helps or hurts. if it hurts, try changing it to 3 > > one more data gathering thing for you to do > > can you run > iostat -xk 10 > during your load test? I want to just double check that you aren't running > into problems with a disk that can't keep up. If any of the drives show > 100% utilization (or close to it) then you may be having problems from that. > > more comments inline: > > > On Wed, 25 Sep 2013, Robert wrote: > > sure sorry I begin a new thread, but I should just reply to it: here is >> the config: >> >> # Run this in debugger mode to troubleshoot >> module(load="impstats" interval="300" severity="7") >> # # to actually gather the data: >> syslog.=debug /var/log/rsyslog-stats >> #/var/log/debugformat;RSYSLOG_**DebugFormat >> >> # rsyslog v7 configuration file >> >> # For more information see /usr/share/doc/rsyslog-*/**rsyslog_conf.html >> # If you experience problems, see http://www.rsyslog.com/doc/** >> troubleshoot.html <http://www.rsyslog.com/doc/troubleshoot.html> >> >> #### MODULES #### >> >> module(load="imuxsock") # needs to be done just once Roberto 8-9-13 >> #SysSock.FlowControl=(:"on") # enable flow control (use if needed) >> Roberto 8-9-13 >> #$ModLoad imuxsock # provides support for local system logging (e.g. via >> logger command) >> >> module(load="imklog") >> #$ModLoad imklog # provides kernel logging support (previously done by >> rklogd) >> #$ModLoad immark # provides --MARK-- message capability >> >> # Provides UDP syslog reception >> module(load="imudp" >> TimeRequery="10" >> SchedulingPolicy="fifo" >> SchedulingPriority="10") # needs to be done just once Roberto 8-9-13 >> input(type="imudp" port="514") # Roberto 8-9-13 >> > > try increasing timerequery from 10 to 100 > add batchsize="100" > > these should reduce the cpu needed for imudp > > > > # Use default timestamp format >> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat >> >> # Include all config files in /etc/rsyslog.d/ >> $IncludeConfig /etc/rsyslog.d/*.conf >> >> # Set Main Message Queue Size - default is 10000 >> $MainMsgQueueType FixedArray #LinkedList >> $MainMsgQueueSize 20000000 >> $MainMsgQueueWorkerThreads 2 >> $MainMsgQueueDequeueBatchSize 1000 >> $InputUDPMaxSessions 40000000 >> >> :hostname, contains, "fdfw" action(type="omfile" >> DirCreateMode="0700" >> FileCreateMode="0644" >> File="/logs/f_fw" >> AsyncWriting="on" >> FlushOnTXEnd="on" >> IOBufferSize="8k" >> queue.type="FixedArray" queue.dequeuebatchsize="100" queue.size="10000" >> ) >> > > try removing the AsyncWriting and FlushOnTXEnd parameters, see if that > helps or hurts. when just writing to a file with one of the built-in > templates, I would not expect it to help much, and the overhead of > communicating with the separate thread to do the write is probably higher > than just writing directly. > > If we find that disk I/O is a limiting factor, we will want to look at > this again and enable compression (ziplevel="9") > > it would also be worth experimenting with slightly larger buffer sizes, > does it help or hurt if you go from 8k to 16k for example? > > as rainer says, if you can avoid using 'contains' tests, it should speed > up the tests. > > if your names are fdfw-1 or similar, changing the test to > > :hostname, startswith, "fdfw" action(type="omfile" > > would speed things up significantly. If the string you are searching for > is not at the beginning, then contains is what you need. > > the best tests to use are (in order) > isequal / isempty > startswith > contains > regex > ereregex > > As rainer says, if you know that a log line will only match one rule, tell > rsyslog this so it doesn't waste time checking other rules (and then you > can re-order rules so the most commonly matched rules are first). now, it > may be that putting each output in it's own queue defeats this. > > > :hostname, contains, "fdfw" { > action(type="omfile" > DirCreateMode="0700" > FileCreateMode="0644" > File="/logs/f_fw" > AsyncWriting="on" > FlushOnTXEnd="on" > IOBufferSize="8k" > queue.type="FixedArray" queue.dequeuebatchsize="100" > queue.size="10000" > ) > stop > > } > ______________________________**_________________ > rsyslog mailing list > http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> > http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
