Hello again, I'm attempting to use rsyslog to process on average 20k messages /minute. At the moment I'm outputting to both files and James Keating's omsolr plugin (https://github.com/MrJamesK/omsolr/blob/master/omsolr.c).
Things were humming along fine until I started throwing more traffic at it. I set up both main message and action queues to try and speed things up, but the omsolr plugin appears to be severely lagging behind. Perhaps there's some general tuning to be done to help, but I have a hunch that the biggest performance gain can be made if the output plugin would send (batch) more than more document to SOLR to index at once. I'm confused on how to configure this or if can even be done given how omsolr is currently implemented. The only example of configurable batching I could find was in the omoracle plugin where I spotted a config option called "omoraclebatchsize". Is there a way to force the batch size in the omsolr plugin using the existing rsyslog framework or would some code changes be required? Here's the snippet of my rsyslog.conf relating to the omsolr plugin: $WorkDirectory /var/spool/rsyslog $ActionQueueFileName solrRuleProd $ActionQueueMaxDiskSpace 4g $ActionQueueDequeueBatchSize 1024 $ActionQueueWorkerThreadMinimumMessages 100 $ActionQueueSaveOnShutdown on $ActionQueueType LinkedList $ActionResumeRetryCount -1 :fromhost-ip, !isequal, "127.0.0.1" :omsolr:;SolrLogDefault Server information: rsyslog-5.8.7 CentOS 6.2 4GB ram 4 x 2GHz CPU (VM) Let me know if you need any more information. Thanks! -Lars _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards
