The biggest work is changing the plugin to handle more than one message at
a time. the 'bulkmode' flag is just notifying the core system that the
plugin can handle batches. It's a result, not a cause.
the plugin needs to dequeue multiple messages at a time (I think there is
a global batchsize variable to control how many it will grab). It then
needs to process all the messages, and implement the appropriate error
handling if the output of all the messages at once doesn't work.
I know the high level logic of what should be happening, but I haven't
looked into the details fo exactly what gets done where in the code.
David Lang
On Fri, 1 Jun 2012,
Lars Peterson wrote:
Looks like a "bulkmode" boolean config variable is availablein the plugin...?
Is the number of messages in the batch adjustable?
Hate to ask you to do all the work for me, but is there an example of what this
would like in rsyslog.conf?
-Lars
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Rainer Gerhards
Sent: Friday, June 01, 2012 3:09 PM
To: [email protected]
Subject: Re: [rsyslog] Output plugin batching
Look at the master-elasticsearch branch.
Rainer
Lars Peterson hat geschrieben:
Thanks Rainer.
I grabbed rsyslog-6.3.9 to have a look at the omelasticsearch plugin. Are your
changes to support batching included in this release or is it not available to
the public? I'm not seeing anything batch related in omelasticsearch.c. Am I
missing something?
-Lars
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Rainer Gerhards
Sent: Friday, June 01, 2012 10:34 AM
To: rsyslog-users
Subject: Re: [rsyslog] Output plugin batching
Batching must be supported by the plugin, it needs to implement the necessary
logic. Most rsyslog-provided plugins are capable of using batch mode. Note that
plugin changes can be considerable. For example, in elasticsearch I needed to
implement a totally different handler and mode of operation (not due to
rsyslog, but the way things work with elasticsearch). So it is best to contact
the plugin author.
Rainer
-----Original Message-----
From: [email protected] [mailto:rsyslog-
[email protected]] On Behalf Of Lars Peterson
Sent: Friday, June 01, 2012 5:28 PM
To: [email protected]
Cc: [email protected]
Subject: [rsyslog] Output plugin batching
Hello again,
I'm attempting to use rsyslog to process on average 20k messages
/minute. At the moment I'm outputting to both files and James
Keating's omsolr plugin
(https://github.com/MrJamesK/omsolr/blob/master/omsolr.c).
Things were humming along fine until I started throwing more traffic
at it. I set up both main message and action queues to try and speed
things up, but the omsolr plugin appears to be severely lagging behind.
Perhaps there's some general tuning to be done to help, but I have a
hunch that the biggest performance gain can be made if the output
plugin would send
(batch) more than more document to SOLR to index at once. I'm confused
on how to configure this or if can even be done given how omsolr is
currently implemented. The only example of configurable batching I
could find was in the omoracle plugin where I spotted a config option
called "omoraclebatchsize". Is there a way to force the batch size in
the omsolr plugin using the existing rsyslog framework or would some code
changes be required?
Here's the snippet of my rsyslog.conf relating to the omsolr plugin:
$WorkDirectory /var/spool/rsyslog
$ActionQueueFileName solrRuleProd
$ActionQueueMaxDiskSpace 4g
$ActionQueueDequeueBatchSize 1024
$ActionQueueWorkerThreadMinimumMessages 100 $ActionQueueSaveOnShutdown
on $ActionQueueType LinkedList $ActionResumeRetryCount -1
:fromhost-ip, !isequal, "127.0.0.1" :omsolr:;SolrLogDefault
Server information:
rsyslog-5.8.7
CentOS 6.2
4GB ram
4 x 2GHz CPU (VM)
Let me know if you need any more information.
Thanks!
-Lars
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
