That's just giving me a CSV formatted output. I don't see errors that I'm aware of - just massive queued up backlog (now down to about 36K files).
On Fri, Jun 27, 2014 at 2:40 PM, Eric Renfro <[email protected]> wrote: > Not a bad question at all. > > I'd be curious to know what your TemplateFormatDSV is doing specifically, > everything else looks reasonably sane, so long as it's not saturating your > network with log deliveries, and on what errors are you getting on what > input data? > -- > Eric Renfro (Psi-Jack) > Linux-Help.org System Administrator > > > On Friday, June 27, 2014 01:05:18 PM Doug McClure wrote: > > I'm in the tuning & optimization phase for a large rsyslog - logstash > > deployment and working through very close monitoring of system > performance, > > logging load and imstats. > > > > This morning, a very large incoming log spike occurred on the rsyslog > > imptcp input and was processed. That spike was routed to my omfwd action > to > > be shipped to a logstash tcp input. > > > > What it appears is happening is that very large input spike went to disk > > assisted queues and has created over 37K files on disk. They are being > > processed and shipped and I have a steady 60Kb/s output headed towards > > logstash tcp input. > > > > My question - where should I begin to look to optimize the processing of > > that output action queue or changing configuration to avoid queuing up so > > much in the first place? How would I determine if this is due to rsyslog > > setup or on the logstash input side? > > > > Looking at imstats in the analyzer for logstash output (omfwd) I see > > significant processed/failed numbers and then in the logstash ouput -DA- > it > > looks like things are as described above - large queues, with no failures > > being processed gradually. > > > > > > rsyslog 7.4.4 output action: > > > > action(type="omfwd" > > Target="10.x.x.x" > > Port="10515" > > Protocol="tcp" > > Template="LogFormatDSV" > > queue.filename="logstashqueue" # set file name, also enables > > disk mode > > queue.size="1000000" > > queue.type="LinkedList" # use asynchronous processing > > queue.workerthreads="5" > > name="logstashforwarder" > > ) > > > > > > Thanks for any pointers! > > > > Doug > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
