you may delete the slowdown option, I also use omes with such option before, and find this option effect to DA queue too. And more, I find that the DA queue batchsize always be 8, although the queue.dequeuebatchsize much more larger.
So, yes, I want a much more faster DA queue too. In a high speed log process system, the DA queue nearly has no use... 2015-11-04 23:53 GMT+08:00 Rainer Gerhards <[email protected]>: > 2015-11-04 16:44 GMT+01:00 Joe Blow <[email protected]>: > > Ok i've played with some numbers.... this is what one of the massive > queues > > looks like now, and it *IS* dequeuing much faster (500 EPS from DA, 25k > EPS > > from in memory queue. > > > This may sound a bit strange, and I never tried it, but .. I wouldn't > be surprised if it is actually faster if you put the queue files on a > compressed directory. The idea behind that is that while this > obviously eats CPU, it will probably save you a lot of real i/o > because the data written to the disk queue can be greatly compressed. > > If you give it a try, please let us know the outcome. > > Rainer > > > Hopefully this helps some other people who have very massive, disk backed > > queues... Please feel free to comment on these values. > > > > action(type="omelasticsearch" > > name="rsys_HugeQ" > > server="10.10.10.10" > > serverport="9200" > > template="HugeQTemplate" > > asyncrepl="on" > > searchType="HugeType" > > searchIndex="HugeQindex" > > timeout="3m" > > dynSearchIndex="on" > > bulkmode="on" > > errorfile="HugeQ_err.log" > > queue.type="linkedlist" > > queue.filename="HugeQ.rsysq" > > queue.maxfilesize="2048m" > > queue.highwatermark="1000000" > > queue.lowwatermark="750000" > > queue.discardmark="499999999" > > queue.dequeueslowdown="100" > > queue.size="500000000" > > queue.saveonshutdown="on" > > queue.maxdiskspace="1000g" > > queue.dequeuebatchsize="50000" > > queue.workerthreads="8" > > queue.workerthreadminimummessages="100000" > > action.resumeretrycount="-1")stop} > > > > I'd love some feedback, but these numbers are working pretty well for > these > > massive feeds. > > > > Cheers, > > > > JB > > > > On Wed, Nov 4, 2015 at 10:26 AM, Radu Gheorghe < > [email protected]> > > wrote: > > > >> On Wed, Nov 4, 2015 at 5:19 PM, Joe Blow <[email protected]> > wrote: > >> > Radu - My checkpoint interval is set at 100k. Are you suggesting > this be > >> > lowered? raised? > >> > >> It sounds like the higher the better, but if your problem is on how > >> fast it can read... I think there's not much you can do - that seems > >> to be a setting for writes. Also note David's comment on how it might > >> only apply if syncing is enabled. > >> > >> On the read side I don't know what optimization you can do in the > >> conf. Maybe you can test with various file sizes? (queue.maxfilesize - > >> the default is 1MB so that might be too small) Though I wouldn't have > >> high hopes, it sounds like recovery is much too slow even for reading > >> 1MB files. > >> > >> Best regards, > >> Radu > >> -- > >> Performance Monitoring * Log Analytics * Search Analytics > >> Solr & Elasticsearch Support * http://sematext.com/ > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> DON'T LIKE THAT. > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
