Thanks for reply David - I've enjoyed reading your notes/prezos on rsyslog recently!
I feel it's somewhere downstream as well. I've made these changes to try and improve throughput. I added haproxy between rsyslog and a logstash farm. I have two logstash instances accepting connections from the haproxy frontend. I'm still trying to figure out how to get balanced traffic across each, but I see traffic hitting both at some point. A spike came in again this AM and there were a few thousand DA files created so I'm not sure what else to try here. Is there a way to increase the rsyslog output processing? Can multiple tcp output connections be established to the haproxy input to increase output? Can the processing of DA files be increased/accelerated somehow? Thanks for any pointers! Doug On Fri, Jun 27, 2014 at 3:07 PM, David Lang <[email protected]> wrote: > On Fri, 27 Jun 2014, Doug McClure wrote: > > I'm in the tuning & optimization phase for a large rsyslog - logstash >> deployment and working through very close monitoring of system >> performance, >> logging load and imstats. >> >> This morning, a very large incoming log spike occurred on the rsyslog >> imptcp input and was processed. That spike was routed to my omfwd action >> to >> be shipped to a logstash tcp input. >> >> What it appears is happening is that very large input spike went to disk >> assisted queues and has created over 37K files on disk. They are being >> processed and shipped and I have a steady 60Kb/s output headed towards >> logstash tcp input. >> >> My question - where should I begin to look to optimize the processing of >> that output action queue or changing configuration to avoid queuing up so >> much in the first place? How would I determine if this is due to rsyslog >> setup or on the logstash input side? >> > > This is probably logstash not being able to keep up with the flood of > traffic from rsyslog. Then once rsyslog spills to disk, it gets > significantly slower (one of the things that can use a revision/upgrade is > the disk queue code) > > you can specify a larger queue to hold more im memory (also, FixedArray > seems to be a little more efficient thatn LinkedList, but that's not your > problem now) > > If logstash can handle more inputs effectively, you can try adding more > workers, but that can acutally slow things down (the locking between > threads can sometimes slow things down more than the added cores working > speed it up) > > But start by looking on the logstash side. > > David Lang > > Looking at imstats in the analyzer for logstash output (omfwd) I see >> significant processed/failed numbers and then in the logstash ouput -DA- >> it >> looks like things are as described above - large queues, with no failures >> being processed gradually. >> >> >> rsyslog 7.4.4 output action: >> >> action(type="omfwd" >> Target="10.x.x.x" >> Port="10515" >> Protocol="tcp" >> Template="LogFormatDSV" >> queue.filename="logstashqueue" # set file name, also enables >> disk mode >> queue.size="1000000" >> queue.type="LinkedList" # use asynchronous processing >> queue.workerthreads="5" >> name="logstashforwarder" >> ) >> >> >> Thanks for any pointers! >> >> Doug >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >> _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
