Hi Risto, I think your queue sizes are quite small for the amount of logs you receive. It should be able to hold data for at least N seconds, so N x 5000. dequeuebatchsize should also be higher, maybe 5000. resumeretrycount="-1" is nice in theory, but a reasonable value would work better in practice. A value of "5" works ok for me for example.
You may want to use disk assisted queues http://www.rsyslog.com/doc/v8-stable/concepts/queues.html#disk-queues. Here is an example config: action( name="plain" type="omelasticsearch" server="server" serverport="80" template="plain" dynSearchIndex="on" searchIndex="index" searchType="syslog" bulkmode="on" action.resumeRetryCount="5" action.resumeInterval="60" queue.dequeuebatchsize="5000" queue.workerthreads="5" queue.type="FixedArray" queue.size="1000000" queue.spoolDirectory="/mnt/rsyslog/queues" queue.filename="plain" queue.maxfilesize="100m" queue.maxdiskspace="1g" queue.highwatermark="50000" queue.lowwatermark="20000" queue.saveonshutdown="on" ) Regards, Ciprian -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Wed, Sep 9, 2015 at 3:30 PM, Risto Vaarandi <[email protected]> wrote: > Hi all, > I am currently tuning one of my rsyslog+elasticsearch installations and > questions about optimal settings have emerged. In the web, there is a nice > guide with several recommendations > http://blog.sematext.com/2014/01/20/rsyslog-8-1-elasticsearch-output-performance/, > but it has one elasticsearch action, while my configuration has many. In a > nutshell, my current setup looks like this: > > ruleset(name="network" queue.size="100000" queue.dequeuebatchsize="100" > queue.workerthreads="2") { > > action(type="mmutf8fix" replacementChar="_") > > if $programname contains 'app1' then { > action(type="omelasticsearch" template="App1" dynSearchIndex="on" > searchIndex="SyslogIndex" server="localhost" bulkmode="on" > queue.type="linkedlist" queue.size="10000" queue.dequeuebatchsize="500" > action.resumeretrycount="-1") > stop > } > > if $programname contains 'app2' then { > action(type="omelasticsearch" template="App2" dynSearchIndex="on" > searchIndex="SyslogIndex" server="localhost" bulkmode="on" > queue.type="linkedlist" queue.size="10000" queue.dequeuebatchsize="500" > action.resumeretrycount="-1") > stop > } > > ... > > action(type="omelasticsearch" template="Generic" dynSearchIndex="on" > searchIndex="SyslogIndex" server="localhost" bulkmode="on" > queue.type="linkedlist" queue.size="10000" queue.dequeuebatchsize="500" > action.resumeretrycount="-1") > stop > > } > > input(type="imtcp" port="514" ruleset="network") > > > Altogether, I have about 20 omelasticsearch actions in the above block of > statements. My questions is -- should I use larger values for queue and > batch size than just 10000 and 500? The guide > http://blog.sematext.com/2014/01/20/rsyslog-8-1-elasticsearch-output-performance/ > recommends much larger values, but these are used for only one action > statement which handles all writes to Elasticsearch. In contrast, my setup > has many actions, and although some actions are less busy, the most active > 7-8 actions see roughly the same amount of traffic. This installations > receives 4-5 thousand messages per second, but the workload will increase > gradually. Also, what about the queue sizes for the entire ruleset, do the > current settings look reasonable? (As I have understood, each ruleset uses > its own queue, and changing the size of the main queue does not influence > the ruleset.) > > Are there any other settings I should consider, in order to increase > performance? > > Kind regards, > risto > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
