Rainer - I'll try and see if I can schedule a time to get those. Unfortunately these are the production servers, so it is difficult to test this. However I've just decided to schedule a time to take a traffic snapshot of traffic so I can replay it through our staging environment and see about simulating the issues I'm seeing.
I'll definitely setup the log.file settings for impstats. What you described sounds exactly like the issue I'm experiencing and makes complete sense now that you point it out. Thanks! Radu - I'll definitely make some of those changes to see how it impacts. I was able to confirm the ES cluster is able to handle far more traffic than is being sent to it. So I know it is the configuration I have in place on the Rsyslog side that I need to resolve. The queue.workerthreads should help in it's own. I've also increased the dequeuebatchsize to 1000 and we'll see how that impacts things. I'll ad more information as I get it. (Rainer, I'll be sending the traffic to the Elasticsearch cluster again today. When I do I'll take a snapshot of the data being output impstats and post it for you to see.) -- James -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Radu Gheorghe Sent: Monday, October 14, 2013 2:37 AM To: rsyslog-users Subject: Re: [rsyslog] Queues And Max Sizes 2013/10/14 Rainer Gerhards <[email protected]> > [...] As you deal with a slow output, I don't think this will have much effect. > > > Right! It should help if you: a) increase the batch size (20 is very little, especially if you have some fast servers). Somewhere between 100 and 1000 is a good starting point, although you might want to go higher if you have big boxes for ES. b) increase the number of queue.workerthreads. Because, after sending a bulk rsyslog has to wait for the response from ES and parse it. So you'll get better throughput if you index logs on multiple threads (even more than you have CPUs, because of the waiting involved) If you want to look at some more optimizations on the ES side, there are lots of them in my presentation Monitorama EU: http://blog.sematext.com/2013/09/24/presentation-on-centralizing-logs/ _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
