Hi all,
I am currently tuning one of my rsyslog+elasticsearch installations and
questions about optimal settings have emerged. In the web, there is a nice
guide with several recommendations
http://blog.sematext.com/2014/01/20/rsyslog-8-1-elasticsearch-output-performance/,
but it has one elasticsearch action, while my configuration has many. In a
nutshell, my current setup looks like this:
ruleset(name="network" queue.size="100000" queue.dequeuebatchsize="100"
queue.workerthreads="2") {
action(type="mmutf8fix" replacementChar="_")
if $programname contains 'app1' then {
action(type="omelasticsearch" template="App1" dynSearchIndex="on"
searchIndex="SyslogIndex" server="localhost" bulkmode="on"
queue.type="linkedlist" queue.size="10000" queue.dequeuebatchsize="500"
action.resumeretrycount="-1")
stop
}
if $programname contains 'app2' then {
action(type="omelasticsearch" template="App2" dynSearchIndex="on"
searchIndex="SyslogIndex" server="localhost" bulkmode="on"
queue.type="linkedlist" queue.size="10000" queue.dequeuebatchsize="500"
action.resumeretrycount="-1")
stop
}
...
action(type="omelasticsearch" template="Generic" dynSearchIndex="on"
searchIndex="SyslogIndex" server="localhost" bulkmode="on"
queue.type="linkedlist" queue.size="10000" queue.dequeuebatchsize="500"
action.resumeretrycount="-1")
stop
}
input(type="imtcp" port="514" ruleset="network")
Altogether, I have about 20 omelasticsearch actions in the above block of
statements. My questions is -- should I use larger values for queue and batch
size than just 10000 and 500? The guide
http://blog.sematext.com/2014/01/20/rsyslog-8-1-elasticsearch-output-performance/
recommends much larger values, but these are used for only one action
statement which handles all writes to Elasticsearch. In contrast, my setup has
many actions, and although some actions are less busy, the most active 7-8
actions see roughly the same amount of traffic. This installations receives 4-5
thousand messages per second, but the workload will increase gradually. Also,
what about the queue sizes for the entire ruleset, do the current settings look
reasonable? (As I have understood, each ruleset uses its own queue, and
changing the size of the main queue does not influence the ruleset.)
Are there any other settings I should consider, in order to increase
performance?
Kind regards,
risto
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
