Search for "rsyslog Design". One of the first links will be a pdf. It has lots of Details, including a state transition diagram. The plunging needs to implement the transaction entry points. Look for queue batchsize parameters in the queue doc. Rainer
Lars Peterson hat geschrieben: Looks like a "bulkmode" boolean config variable is availablein the plugin...? Is the number of messages in the batch adjustable? Hate to ask you to do all the work for me, but is there an example of what this would like in rsyslog.conf? -Lars -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Rainer Gerhards Sent: Friday, June 01, 2012 3:09 PM To: [email protected] Subject: Re: [rsyslog] Output plugin batching Look at the master-elasticsearch branch. Rainer Lars Peterson hat geschrieben: Thanks Rainer. I grabbed rsyslog-6.3.9 to have a look at the omelasticsearch plugin. Are your changes to support batching included in this release or is it not available to the public? I'm not seeing anything batch related in omelasticsearch.c. Am I missing something? -Lars -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Rainer Gerhards Sent: Friday, June 01, 2012 10:34 AM To: rsyslog-users Subject: Re: [rsyslog] Output plugin batching Batching must be supported by the plugin, it needs to implement the necessary logic. Most rsyslog-provided plugins are capable of using batch mode. Note that plugin changes can be considerable. For example, in elasticsearch I needed to implement a totally different handler and mode of operation (not due to rsyslog, but the way things work with elasticsearch). So it is best to contact the plugin author. Rainer > -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Lars Peterson > Sent: Friday, June 01, 2012 5:28 PM > To: [email protected] > Cc: [email protected] > Subject: [rsyslog] Output plugin batching > > Hello again, > > I'm attempting to use rsyslog to process on average 20k messages > /minute. At the moment I'm outputting to both files and James > Keating's omsolr plugin > (https://github.com/MrJamesK/omsolr/blob/master/omsolr.c). > > Things were humming along fine until I started throwing more traffic > at it. I set up both main message and action queues to try and speed > things up, but the omsolr plugin appears to be severely lagging behind. > > Perhaps there's some general tuning to be done to help, but I have a > hunch that the biggest performance gain can be made if the output > plugin would send > (batch) more than more document to SOLR to index at once. I'm confused > on how to configure this or if can even be done given how omsolr is > currently implemented. The only example of configurable batching I > could find was in the omoracle plugin where I spotted a config option > called "omoraclebatchsize". Is there a way to force the batch size in > the omsolr plugin using the existing rsyslog framework or would some code > changes be required? > > Here's the snippet of my rsyslog.conf relating to the omsolr plugin: > > $WorkDirectory /var/spool/rsyslog > $ActionQueueFileName solrRuleProd > $ActionQueueMaxDiskSpace 4g > $ActionQueueDequeueBatchSize 1024 > $ActionQueueWorkerThreadMinimumMessages 100 $ActionQueueSaveOnShutdown > on $ActionQueueType LinkedList $ActionResumeRetryCount -1 > :fromhost-ip, !isequal, "127.0.0.1" > :omsolr:;SolrLogDefault > > Server information: > > rsyslog-5.8.7 > CentOS 6.2 > 4GB ram > 4 x 2GHz CPU (VM) > > Let me know if you need any more information. > > Thanks! > > -Lars > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards
