I'm in the tuning & optimization phase for a large rsyslog - logstash
deployment and working through very close monitoring of system performance,
logging load and imstats.
This morning, a very large incoming log spike occurred on the rsyslog
imptcp input and was processed. That spike was routed to my omfwd action to
be shipped to a logstash tcp input.
What it appears is happening is that very large input spike went to disk
assisted queues and has created over 37K files on disk. They are being
processed and shipped and I have a steady 60Kb/s output headed towards
logstash tcp input.
My question - where should I begin to look to optimize the processing of
that output action queue or changing configuration to avoid queuing up so
much in the first place? How would I determine if this is due to rsyslog
setup or on the logstash input side?
Looking at imstats in the analyzer for logstash output (omfwd) I see
significant processed/failed numbers and then in the logstash ouput -DA- it
looks like things are as described above - large queues, with no failures
being processed gradually.
rsyslog 7.4.4 output action:
action(type="omfwd"
Target="10.x.x.x"
Port="10515"
Protocol="tcp"
Template="LogFormatDSV"
queue.filename="logstashqueue" # set file name, also enables
disk mode
queue.size="1000000"
queue.type="LinkedList" # use asynchronous processing
queue.workerthreads="5"
name="logstashforwarder"
)
Thanks for any pointers!
Doug
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
