I just found the magical setting it appears - rebindinterval! I set this to 10 and have balanced traffic across each logstash instances behind haproxy.
Any guidance on setting rebindinterval? On Sat, Jun 28, 2014 at 11:00 AM, Doug McClure <[email protected]> wrote: > Thanks for reply David - I've enjoyed reading your notes/prezos on rsyslog > recently! > > I feel it's somewhere downstream as well. I've made these changes to try > and improve throughput. > > I added haproxy between rsyslog and a logstash farm. I have two logstash > instances accepting connections from the haproxy frontend. I'm still > trying to figure out how to get balanced traffic across each, but I see > traffic hitting both at some point. > > A spike came in again this AM and there were a few thousand DA files > created so I'm not sure what else to try here. Is there a way to increase > the rsyslog output processing? Can multiple tcp output connections be > established to the haproxy input to increase output? Can the processing of > DA files be increased/accelerated somehow? > > > Thanks for any pointers! > > Doug > > > On Fri, Jun 27, 2014 at 3:07 PM, David Lang <[email protected]> wrote: > >> On Fri, 27 Jun 2014, Doug McClure wrote: >> >> I'm in the tuning & optimization phase for a large rsyslog - logstash >>> deployment and working through very close monitoring of system >>> performance, >>> logging load and imstats. >>> >>> This morning, a very large incoming log spike occurred on the rsyslog >>> imptcp input and was processed. That spike was routed to my omfwd action >>> to >>> be shipped to a logstash tcp input. >>> >>> What it appears is happening is that very large input spike went to disk >>> assisted queues and has created over 37K files on disk. They are being >>> processed and shipped and I have a steady 60Kb/s output headed towards >>> logstash tcp input. >>> >>> My question - where should I begin to look to optimize the processing of >>> that output action queue or changing configuration to avoid queuing up so >>> much in the first place? How would I determine if this is due to rsyslog >>> setup or on the logstash input side? >>> >> >> This is probably logstash not being able to keep up with the flood of >> traffic from rsyslog. Then once rsyslog spills to disk, it gets >> significantly slower (one of the things that can use a revision/upgrade is >> the disk queue code) >> >> you can specify a larger queue to hold more im memory (also, FixedArray >> seems to be a little more efficient thatn LinkedList, but that's not your >> problem now) >> >> If logstash can handle more inputs effectively, you can try adding more >> workers, but that can acutally slow things down (the locking between >> threads can sometimes slow things down more than the added cores working >> speed it up) >> >> But start by looking on the logstash side. >> >> David Lang >> >> Looking at imstats in the analyzer for logstash output (omfwd) I see >>> significant processed/failed numbers and then in the logstash ouput -DA- >>> it >>> looks like things are as described above - large queues, with no failures >>> being processed gradually. >>> >>> >>> rsyslog 7.4.4 output action: >>> >>> action(type="omfwd" >>> Target="10.x.x.x" >>> Port="10515" >>> Protocol="tcp" >>> Template="LogFormatDSV" >>> queue.filename="logstashqueue" # set file name, also enables >>> disk mode >>> queue.size="1000000" >>> queue.type="LinkedList" # use asynchronous processing >>> queue.workerthreads="5" >>> name="logstashforwarder" >>> ) >>> >>> >>> Thanks for any pointers! >>> >>> Doug >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >>> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
