> -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Rainer Gerhards > Sent: Monday, September 14, 2015 3:20 PM > To: rsyslog-users > Subject: Re: [rsyslog] recommendations for omelasticsearch queue sizes > > 2015-09-14 13:05 GMT+02:00 Risto Vaarandi <[email protected]>: > > Thanks to all who provided suggestions and comments -- I've managed to > > create a much better configuration :) I'd like to clarify one detail, > > though. Do the following directives > > > > Action.resumeretrycount="5" > > Action.resumeinterval="60" > > > > mean that after message writing failure, rsyslog will have 4 additional > > tries > with 60 second intervals? In other words, if the message is not written within > 240 seconds, it's lost? > > close ;) > > The interval is extended after a given number of retries. If you need it > totally > right, I would probably need to check the code/doc, but out of my head I > think it is doubled every 10 retries until a max is reached. So in the > concrete > case, it's lost after 240 seconds, just as you say. With count="15", I think > it > would take 1200 seconds. (10*60, 5*120). Again, if it needs to be totally > correct, I would need to check in detail.
Thanks! I noticed this heuristic in the docs, and it's a nice way to handle action.resumeretrycount="-1" when the destination disappears for a longer period of time. Regards, risto > > HTH > Rainer > > > > > Kind regards, > > risto > > > >> -----Original Message----- > >> From: [email protected] [mailto:rsyslog- > >> [email protected]] On Behalf Of Ciprian Hacman > >> Sent: Wednesday, September 09, 2015 5:04 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] recommendations for omelasticsearch queue > >> sizes > >> > >> Hi Risto, > >> > >> I think your queue sizes are quite small for the amount of logs you > >> receive. It should be able to hold data for at least N seconds, so N x > >> 5000. > >> dequeuebatchsize should also be higher, maybe 5000. > >> resumeretrycount="-1" is nice in theory, but a reasonable value would > >> work better in practice. A value of "5" works ok for me for example. > >> > >> You may want to use disk assisted queues > >> http://www.rsyslog.com/doc/v8- stable/concepts/queues.html#disk- > queues. > >> Here is an example config: > >> > >> action( > >> > >> name="plain" > >> > >> type="omelasticsearch" > >> > >> server="server" > >> > >> serverport="80" > >> > >> template="plain" > >> > >> dynSearchIndex="on" > >> > >> searchIndex="index" > >> > >> searchType="syslog" > >> > >> bulkmode="on" > >> > >> action.resumeRetryCount="5" > >> > >> action.resumeInterval="60" > >> > >> queue.dequeuebatchsize="5000" > >> > >> queue.workerthreads="5" > >> > >> queue.type="FixedArray" > >> > >> queue.size="1000000" > >> > >> queue.spoolDirectory="/mnt/rsyslog/queues" > >> > >> queue.filename="plain" > >> > >> queue.maxfilesize="100m" > >> > >> queue.maxdiskspace="1g" > >> > >> queue.highwatermark="50000" > >> > >> queue.lowwatermark="20000" > >> > >> queue.saveonshutdown="on" > >> > >> ) > >> > >> Regards, > >> Ciprian > >> -- > >> Performance Monitoring * Log Analytics * Search Analytics Solr & > >> Elasticsearch Support * http://sematext.com/ > >> > >> > >> On Wed, Sep 9, 2015 at 3:30 PM, Risto Vaarandi > >> <[email protected]> > >> wrote: > >> > >> > Hi all, > >> > I am currently tuning one of my rsyslog+elasticsearch installations > >> > and questions about optimal settings have emerged. In the web, > >> > there is a nice guide with several recommendations > >> > http://blog.sematext.com/2014/01/20/rsyslog-8-1-elasticsearch-outpu > >> > t-p erformance/, but it has one elasticsearch action, while my > >> > configuration has many. In a nutshell, my current setup looks like > >> > this: > >> > > >> > ruleset(name="network" queue.size="100000" > >> queue.dequeuebatchsize="100" > >> > queue.workerthreads="2") { > >> > > >> > action(type="mmutf8fix" replacementChar="_") > >> > > >> > if $programname contains 'app1' then { > >> > action(type="omelasticsearch" template="App1" > dynSearchIndex="on" > >> > searchIndex="SyslogIndex" server="localhost" bulkmode="on" > >> > queue.type="linkedlist" queue.size="10000" > >> queue.dequeuebatchsize="500" > >> > action.resumeretrycount="-1") > >> > stop > >> > } > >> > > >> > if $programname contains 'app2' then { > >> > action(type="omelasticsearch" template="App2" > dynSearchIndex="on" > >> > searchIndex="SyslogIndex" server="localhost" bulkmode="on" > >> > queue.type="linkedlist" queue.size="10000" > >> queue.dequeuebatchsize="500" > >> > action.resumeretrycount="-1") > >> > stop > >> > } > >> > > >> > ... > >> > > >> > action(type="omelasticsearch" template="Generic" > dynSearchIndex="on" > >> > searchIndex="SyslogIndex" server="localhost" bulkmode="on" > >> > queue.type="linkedlist" queue.size="10000" > >> queue.dequeuebatchsize="500" > >> > action.resumeretrycount="-1") > >> > stop > >> > > >> > } > >> > > >> > input(type="imtcp" port="514" ruleset="network") > >> > > >> > > >> > Altogether, I have about 20 omelasticsearch actions in the above > >> > block of statements. My questions is -- should I use larger values > >> > for queue and batch size than just 10000 and 500? The guide > >> > http://blog.sematext.com/2014/01/20/rsyslog-8-1-elasticsearch-outpu > >> > t-p erformance/ recommends much larger values, but these are used > >> > for only one action statement which handles all writes to > >> > Elasticsearch. In contrast, my setup has many actions, and although > >> > some actions are less busy, the most active > >> > 7-8 actions see roughly the same amount of traffic. This > >> > installations receives 4-5 thousand messages per second, but the > >> > workload will increase gradually. Also, what about the queue sizes > >> > for the entire ruleset, do the current settings look reasonable? > >> > (As I have understood, each ruleset uses its own queue, and > >> > changing the size of the main queue does not influence the > >> > ruleset.) > >> > > >> > Are there any other settings I should consider, in order to > >> > increase performance? > >> > > >> > Kind regards, > >> > risto > >> > _______________________________________________ > >> > rsyslog mailing list > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > http://www.rsyslog.com/professional-services/ > >> > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > >> > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > >> > you DON'T LIKE THAT. > >> > > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: > >> This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites > beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: > This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond > our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
