2015-11-04 17:08 GMT+01:00 Joe Blow <[email protected]>: > I think i've spoken too soon. The in memory queues are clearing extremely > well with these settings, but the DA stuff is still pretty sluggish (slowed > down to 50-100 EPS again). I've looked at the box and the IO is around 10% > (12 disk array, which performs quite snappily), so i'm sincerely doubting > this is an IO issue. > > The huge feed in question uses around 4 workers before it has enough > workers to clear the queue as fast as it comes in (400k avg in the queue). > From my understanding that means that i've got 4 workers@50k bulk size > each, and at 200k EPS out (4 workers x 50k EPS) the in memory queue gets no > bigger. Now this is where my knowledge ends. I've set the low watermark > to 750k and high watermark to 1 million, with the thoughts that the low > watermark is below having all 8 workers full bore (8x100k) and the high > watermark is a 250k higher than that (slightly above all workers going full > bore). If i'm staying below the low watermark, and still have "free" > workers, would those workers not try and empty the DA queue? What would > help allocate more resources to clearing the DA queue?
The DA queue always run on one worker, because you can't use more than one worker with purely sequential files. TBH I think your needs simply go above what the current system can provide. As David said, the queue subsystem could well deserve an overhaul, but this is a too-big task right now given what else is going on and there also has been no sponsor for any of that disk queue work in the past years, so it doesn't seem to have a too high priority either. Rainer > > Thanks for the prompt responses. > > Cheers, > > JB > > On Wed, Nov 4, 2015 at 10:53 AM, Rainer Gerhards <[email protected]> > wrote: > >> 2015-11-04 16:44 GMT+01:00 Joe Blow <[email protected]>: >> > Ok i've played with some numbers.... this is what one of the massive >> queues >> > looks like now, and it *IS* dequeuing much faster (500 EPS from DA, 25k >> EPS >> > from in memory queue. >> > >> This may sound a bit strange, and I never tried it, but .. I wouldn't >> be surprised if it is actually faster if you put the queue files on a >> compressed directory. The idea behind that is that while this >> obviously eats CPU, it will probably save you a lot of real i/o >> because the data written to the disk queue can be greatly compressed. >> >> If you give it a try, please let us know the outcome. >> >> Rainer >> >> > Hopefully this helps some other people who have very massive, disk backed >> > queues... Please feel free to comment on these values. >> > >> > action(type="omelasticsearch" >> > name="rsys_HugeQ" >> > server="10.10.10.10" >> > serverport="9200" >> > template="HugeQTemplate" >> > asyncrepl="on" >> > searchType="HugeType" >> > searchIndex="HugeQindex" >> > timeout="3m" >> > dynSearchIndex="on" >> > bulkmode="on" >> > errorfile="HugeQ_err.log" >> > queue.type="linkedlist" >> > queue.filename="HugeQ.rsysq" >> > queue.maxfilesize="2048m" >> > queue.highwatermark="1000000" >> > queue.lowwatermark="750000" >> > queue.discardmark="499999999" >> > queue.dequeueslowdown="100" >> > queue.size="500000000" >> > queue.saveonshutdown="on" >> > queue.maxdiskspace="1000g" >> > queue.dequeuebatchsize="50000" >> > queue.workerthreads="8" >> > queue.workerthreadminimummessages="100000" >> > action.resumeretrycount="-1")stop} >> > >> > I'd love some feedback, but these numbers are working pretty well for >> these >> > massive feeds. >> > >> > Cheers, >> > >> > JB >> > >> > On Wed, Nov 4, 2015 at 10:26 AM, Radu Gheorghe < >> [email protected]> >> > wrote: >> > >> >> On Wed, Nov 4, 2015 at 5:19 PM, Joe Blow <[email protected]> >> wrote: >> >> > Radu - My checkpoint interval is set at 100k. Are you suggesting >> this be >> >> > lowered? raised? >> >> >> >> It sounds like the higher the better, but if your problem is on how >> >> fast it can read... I think there's not much you can do - that seems >> >> to be a setting for writes. Also note David's comment on how it might >> >> only apply if syncing is enabled. >> >> >> >> On the read side I don't know what optimization you can do in the >> >> conf. Maybe you can test with various file sizes? (queue.maxfilesize - >> >> the default is 1MB so that might be too small) Though I wouldn't have >> >> high hopes, it sounds like recovery is much too slow even for reading >> >> 1MB files. >> >> >> >> Best regards, >> >> Radu >> >> -- >> >> Performance Monitoring * Log Analytics * Search Analytics >> >> Solr & Elasticsearch Support * http://sematext.com/ >> >> _______________________________________________ >> >> rsyslog mailing list >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> http://www.rsyslog.com/professional-services/ >> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> >> DON'T LIKE THAT. >> >> >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com/professional-services/ >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
