On Fri, 27 Jun 2014, Doug McClure wrote:
I'm in the tuning & optimization phase for a large rsyslog - logstash
deployment and working through very close monitoring of system performance,
logging load and imstats.
This morning, a very large incoming log spike occurred on the rsyslog
imptcp input and was processed. That spike was routed to my omfwd action to
be shipped to a logstash tcp input.
What it appears is happening is that very large input spike went to disk
assisted queues and has created over 37K files on disk. They are being
processed and shipped and I have a steady 60Kb/s output headed towards
logstash tcp input.
My question - where should I begin to look to optimize the processing of
that output action queue or changing configuration to avoid queuing up so
much in the first place? How would I determine if this is due to rsyslog
setup or on the logstash input side?
This is probably logstash not being able to keep up with the flood of traffic
from rsyslog. Then once rsyslog spills to disk, it gets significantly slower
(one of the things that can use a revision/upgrade is the disk queue code)
you can specify a larger queue to hold more im memory (also, FixedArray seems
to be a little more efficient thatn LinkedList, but that's not your problem now)
If logstash can handle more inputs effectively, you can try adding more workers,
but that can acutally slow things down (the locking between threads can
sometimes slow things down more than the added cores working speed it up)
But start by looking on the logstash side.
David Lang
Looking at imstats in the analyzer for logstash output (omfwd) I see
significant processed/failed numbers and then in the logstash ouput -DA- it
looks like things are as described above - large queues, with no failures
being processed gradually.
rsyslog 7.4.4 output action:
action(type="omfwd"
Target="10.x.x.x"
Port="10515"
Protocol="tcp"
Template="LogFormatDSV"
queue.filename="logstashqueue" # set file name, also enables
disk mode
queue.size="1000000"
queue.type="LinkedList" # use asynchronous processing
queue.workerthreads="5"
name="logstashforwarder"
)
Thanks for any pointers!
Doug
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
