Thanks Rainer. I grabbed rsyslog-6.3.9 to have a look at the omelasticsearch plugin. Are your changes to support batching included in this release or is it not available to the public? I'm not seeing anything batch related in omelasticsearch.c. Am I missing something?
-Lars -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Rainer Gerhards Sent: Friday, June 01, 2012 10:34 AM To: rsyslog-users Subject: Re: [rsyslog] Output plugin batching Batching must be supported by the plugin, it needs to implement the necessary logic. Most rsyslog-provided plugins are capable of using batch mode. Note that plugin changes can be considerable. For example, in elasticsearch I needed to implement a totally different handler and mode of operation (not due to rsyslog, but the way things work with elasticsearch). So it is best to contact the plugin author. Rainer > -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Lars Peterson > Sent: Friday, June 01, 2012 5:28 PM > To: [email protected] > Cc: [email protected] > Subject: [rsyslog] Output plugin batching > > Hello again, > > I'm attempting to use rsyslog to process on average 20k messages > /minute. At the moment I'm outputting to both files and James > Keating's omsolr plugin > (https://github.com/MrJamesK/omsolr/blob/master/omsolr.c). > > Things were humming along fine until I started throwing more traffic > at it. I set up both main message and action queues to try and speed > things up, but the omsolr plugin appears to be severely lagging behind. > > Perhaps there's some general tuning to be done to help, but I have a > hunch that the biggest performance gain can be made if the output > plugin would send > (batch) more than more document to SOLR to index at once. I'm confused > on how to configure this or if can even be done given how omsolr is > currently implemented. The only example of configurable batching I > could find was in the omoracle plugin where I spotted a config option > called "omoraclebatchsize". Is there a way to force the batch size in > the omsolr plugin using the existing rsyslog framework or would some code > changes be required? > > Here's the snippet of my rsyslog.conf relating to the omsolr plugin: > > $WorkDirectory /var/spool/rsyslog > $ActionQueueFileName solrRuleProd > $ActionQueueMaxDiskSpace 4g > $ActionQueueDequeueBatchSize 1024 > $ActionQueueWorkerThreadMinimumMessages 100 $ActionQueueSaveOnShutdown > on $ActionQueueType LinkedList $ActionResumeRetryCount -1 > :fromhost-ip, !isequal, "127.0.0.1" > :omsolr:;SolrLogDefault > > Server information: > > rsyslog-5.8.7 > CentOS 6.2 > 4GB ram > 4 x 2GHz CPU (VM) > > Let me know if you need any more information. > > Thanks! > > -Lars > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards
