Batching must be supported by the plugin, it needs to implement the necessary logic. Most rsyslog-provided plugins are capable of using batch mode. Note that plugin changes can be considerable. For example, in elasticsearch I needed to implement a totally different handler and mode of operation (not due to rsyslog, but the way things work with elasticsearch). So it is best to contact the plugin author.
Rainer > -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Lars Peterson > Sent: Friday, June 01, 2012 5:28 PM > To: [email protected] > Cc: [email protected] > Subject: [rsyslog] Output plugin batching > > Hello again, > > I'm attempting to use rsyslog to process on average 20k messages /minute. At > the moment I'm outputting to both files and James Keating's omsolr plugin > (https://github.com/MrJamesK/omsolr/blob/master/omsolr.c). > > Things were humming along fine until I started throwing more traffic at it. I > set > up both main message and action queues to try and speed things up, but the > omsolr plugin appears to be severely lagging behind. > > Perhaps there's some general tuning to be done to help, but I have a hunch > that > the biggest performance gain can be made if the output plugin would send > (batch) more than more document to SOLR to index at once. I'm confused on > how to configure this or if can even be done given how omsolr is currently > implemented. The only example of configurable batching I could find was in the > omoracle plugin where I spotted a config option called "omoraclebatchsize". Is > there a way to force the batch size in the omsolr plugin using the existing > rsyslog > framework or would some code changes be required? > > Here's the snippet of my rsyslog.conf relating to the omsolr plugin: > > $WorkDirectory /var/spool/rsyslog > $ActionQueueFileName solrRuleProd > $ActionQueueMaxDiskSpace 4g > $ActionQueueDequeueBatchSize 1024 > $ActionQueueWorkerThreadMinimumMessages 100 > $ActionQueueSaveOnShutdown on > $ActionQueueType LinkedList > $ActionResumeRetryCount -1 > :fromhost-ip, !isequal, "127.0.0.1" > :omsolr:;SolrLogDefault > > Server information: > > rsyslog-5.8.7 > CentOS 6.2 > 4GB ram > 4 x 2GHz CPU (VM) > > Let me know if you need any more information. > > Thanks! > > -Lars > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards
