Not a bad question at all. I'd be curious to know what your TemplateFormatDSV is doing specifically, everything else looks reasonably sane, so long as it's not saturating your network with log deliveries, and on what errors are you getting on what input data? -- Eric Renfro (Psi-Jack) Linux-Help.org System Administrator
On Friday, June 27, 2014 01:05:18 PM Doug McClure wrote: > I'm in the tuning & optimization phase for a large rsyslog - logstash > deployment and working through very close monitoring of system performance, > logging load and imstats. > > This morning, a very large incoming log spike occurred on the rsyslog > imptcp input and was processed. That spike was routed to my omfwd action to > be shipped to a logstash tcp input. > > What it appears is happening is that very large input spike went to disk > assisted queues and has created over 37K files on disk. They are being > processed and shipped and I have a steady 60Kb/s output headed towards > logstash tcp input. > > My question - where should I begin to look to optimize the processing of > that output action queue or changing configuration to avoid queuing up so > much in the first place? How would I determine if this is due to rsyslog > setup or on the logstash input side? > > Looking at imstats in the analyzer for logstash output (omfwd) I see > significant processed/failed numbers and then in the logstash ouput -DA- it > looks like things are as described above - large queues, with no failures > being processed gradually. > > > rsyslog 7.4.4 output action: > > action(type="omfwd" > Target="10.x.x.x" > Port="10515" > Protocol="tcp" > Template="LogFormatDSV" > queue.filename="logstashqueue" # set file name, also enables > disk mode > queue.size="1000000" > queue.type="LinkedList" # use asynchronous processing > queue.workerthreads="5" > name="logstashforwarder" > ) > > > Thanks for any pointers! > > Doug > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
