2015-11-04 16:44 GMT+01:00 Joe Blow <[email protected]>: > Ok i've played with some numbers.... this is what one of the massive queues > looks like now, and it *IS* dequeuing much faster (500 EPS from DA, 25k EPS > from in memory queue. > This may sound a bit strange, and I never tried it, but .. I wouldn't be surprised if it is actually faster if you put the queue files on a compressed directory. The idea behind that is that while this obviously eats CPU, it will probably save you a lot of real i/o because the data written to the disk queue can be greatly compressed.
If you give it a try, please let us know the outcome. Rainer > Hopefully this helps some other people who have very massive, disk backed > queues... Please feel free to comment on these values. > > action(type="omelasticsearch" > name="rsys_HugeQ" > server="10.10.10.10" > serverport="9200" > template="HugeQTemplate" > asyncrepl="on" > searchType="HugeType" > searchIndex="HugeQindex" > timeout="3m" > dynSearchIndex="on" > bulkmode="on" > errorfile="HugeQ_err.log" > queue.type="linkedlist" > queue.filename="HugeQ.rsysq" > queue.maxfilesize="2048m" > queue.highwatermark="1000000" > queue.lowwatermark="750000" > queue.discardmark="499999999" > queue.dequeueslowdown="100" > queue.size="500000000" > queue.saveonshutdown="on" > queue.maxdiskspace="1000g" > queue.dequeuebatchsize="50000" > queue.workerthreads="8" > queue.workerthreadminimummessages="100000" > action.resumeretrycount="-1")stop} > > I'd love some feedback, but these numbers are working pretty well for these > massive feeds. > > Cheers, > > JB > > On Wed, Nov 4, 2015 at 10:26 AM, Radu Gheorghe <[email protected]> > wrote: > >> On Wed, Nov 4, 2015 at 5:19 PM, Joe Blow <[email protected]> wrote: >> > Radu - My checkpoint interval is set at 100k. Are you suggesting this be >> > lowered? raised? >> >> It sounds like the higher the better, but if your problem is on how >> fast it can read... I think there's not much you can do - that seems >> to be a setting for writes. Also note David's comment on how it might >> only apply if syncing is enabled. >> >> On the read side I don't know what optimization you can do in the >> conf. Maybe you can test with various file sizes? (queue.maxfilesize - >> the default is 1MB so that might be too small) Though I wouldn't have >> high hopes, it sounds like recovery is much too slow even for reading >> 1MB files. >> >> Best regards, >> Radu >> -- >> Performance Monitoring * Log Analytics * Search Analytics >> Solr & Elasticsearch Support * http://sematext.com/ >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
