dequeuebatchsize don't has the same meaning as logstash flush_size. You can check http://www.gossamer-threads.com/lists/rsyslog/users/17550 and http://www.gossamer-threads.com/lists/rsyslog/users/17825 for more informations. btw: use too many actions may got 429 error from elasticsearch. I use temp variables to help different logs to use the same template, then I had one action with larger batch and less workers. That's good for elasticsearch performance.
2015-09-09 20:30 GMT+08:00 Risto Vaarandi <[email protected]>: > Hi all, > I am currently tuning one of my rsyslog+elasticsearch installations and > questions about optimal settings have emerged. In the web, there is a nice > guide with several recommendations > http://blog.sematext.com/2014/01/20/rsyslog-8-1-elasticsearch-output-performance/, > but it has one elasticsearch action, while my configuration has many. In a > nutshell, my current setup looks like this: > > ruleset(name="network" queue.size="100000" queue.dequeuebatchsize="100" > queue.workerthreads="2") { > > action(type="mmutf8fix" replacementChar="_") > > if $programname contains 'app1' then { > action(type="omelasticsearch" template="App1" dynSearchIndex="on" > searchIndex="SyslogIndex" server="localhost" bulkmode="on" > queue.type="linkedlist" queue.size="10000" queue.dequeuebatchsize="500" > action.resumeretrycount="-1") > stop > } > > if $programname contains 'app2' then { > action(type="omelasticsearch" template="App2" dynSearchIndex="on" > searchIndex="SyslogIndex" server="localhost" bulkmode="on" > queue.type="linkedlist" queue.size="10000" queue.dequeuebatchsize="500" > action.resumeretrycount="-1") > stop > } > > ... > > action(type="omelasticsearch" template="Generic" dynSearchIndex="on" > searchIndex="SyslogIndex" server="localhost" bulkmode="on" > queue.type="linkedlist" queue.size="10000" queue.dequeuebatchsize="500" > action.resumeretrycount="-1") > stop > > } > > input(type="imtcp" port="514" ruleset="network") > > > Altogether, I have about 20 omelasticsearch actions in the above block of > statements. My questions is -- should I use larger values for queue and > batch size than just 10000 and 500? The guide > http://blog.sematext.com/2014/01/20/rsyslog-8-1-elasticsearch-output-performance/ > recommends much larger values, but these are used for only one action > statement which handles all writes to Elasticsearch. In contrast, my setup > has many actions, and although some actions are less busy, the most active > 7-8 actions see roughly the same amount of traffic. This installations > receives 4-5 thousand messages per second, but the workload will increase > gradually. Also, what about the queue sizes for the entire ruleset, do the > current settings look reasonable? (As I have understood, each ruleset uses > its own queue, and changing the size of the main queue does not influence > the ruleset.) > > Are there any other settings I should consider, in order to increase > performance? > > Kind regards, > risto > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
