2015-11-04 17:12 GMT+01:00 Rainer Gerhards <[email protected]>: > 2015-11-04 17:08 GMT+01:00 Joe Blow <[email protected]>: >> I think i've spoken too soon. The in memory queues are clearing extremely >> well with these settings, but the DA stuff is still pretty sluggish (slowed >> down to 50-100 EPS again). I've looked at the box and the IO is around 10% >> (12 disk array, which performs quite snappily), so i'm sincerely doubting >> this is an IO issue. >> >> The huge feed in question uses around 4 workers before it has enough >> workers to clear the queue as fast as it comes in (400k avg in the queue). >> From my understanding that means that i've got 4 workers@50k bulk size >> each, and at 200k EPS out (4 workers x 50k EPS) the in memory queue gets no >> bigger. Now this is where my knowledge ends. I've set the low watermark >> to 750k and high watermark to 1 million, with the thoughts that the low >> watermark is below having all 8 workers full bore (8x100k) and the high >> watermark is a 250k higher than that (slightly above all workers going full >> bore). If i'm staying below the low watermark, and still have "free" >> workers, would those workers not try and empty the DA queue? What would >> help allocate more resources to clearing the DA queue? > > The DA queue always run on one worker, because you can't use more than > one worker with purely sequential files. > > TBH I think your needs simply go above what the current system can > provide. As David said, the queue subsystem could well deserve an > overhaul, but this is a too-big task right now given what else is > going on and there also has been no sponsor for any of that disk queue > work in the past years, so it doesn't seem to have a too high priority > either.
mhhh, I should mention a potential work-around: forget the DA queue. Use failover actions. If the action fails, write log lines in native format to a file. Then, use imfile to monitor that file. Together with a smart design of the rulesets, you can probably get all you need out of such a system. Unfortunately, I am even more swamped than usual, so I cannot provide detail advised except than here pointing you to the idea. HTH Rainer > > Rainer >> >> Thanks for the prompt responses. >> >> Cheers, >> >> JB >> >> On Wed, Nov 4, 2015 at 10:53 AM, Rainer Gerhards <[email protected]> >> wrote: >> >>> 2015-11-04 16:44 GMT+01:00 Joe Blow <[email protected]>: >>> > Ok i've played with some numbers.... this is what one of the massive >>> queues >>> > looks like now, and it *IS* dequeuing much faster (500 EPS from DA, 25k >>> EPS >>> > from in memory queue. >>> > >>> This may sound a bit strange, and I never tried it, but .. I wouldn't >>> be surprised if it is actually faster if you put the queue files on a >>> compressed directory. The idea behind that is that while this >>> obviously eats CPU, it will probably save you a lot of real i/o >>> because the data written to the disk queue can be greatly compressed. >>> >>> If you give it a try, please let us know the outcome. >>> >>> Rainer >>> >>> > Hopefully this helps some other people who have very massive, disk backed >>> > queues... Please feel free to comment on these values. >>> > >>> > action(type="omelasticsearch" >>> > name="rsys_HugeQ" >>> > server="10.10.10.10" >>> > serverport="9200" >>> > template="HugeQTemplate" >>> > asyncrepl="on" >>> > searchType="HugeType" >>> > searchIndex="HugeQindex" >>> > timeout="3m" >>> > dynSearchIndex="on" >>> > bulkmode="on" >>> > errorfile="HugeQ_err.log" >>> > queue.type="linkedlist" >>> > queue.filename="HugeQ.rsysq" >>> > queue.maxfilesize="2048m" >>> > queue.highwatermark="1000000" >>> > queue.lowwatermark="750000" >>> > queue.discardmark="499999999" >>> > queue.dequeueslowdown="100" >>> > queue.size="500000000" >>> > queue.saveonshutdown="on" >>> > queue.maxdiskspace="1000g" >>> > queue.dequeuebatchsize="50000" >>> > queue.workerthreads="8" >>> > queue.workerthreadminimummessages="100000" >>> > action.resumeretrycount="-1")stop} >>> > >>> > I'd love some feedback, but these numbers are working pretty well for >>> these >>> > massive feeds. >>> > >>> > Cheers, >>> > >>> > JB >>> > >>> > On Wed, Nov 4, 2015 at 10:26 AM, Radu Gheorghe < >>> [email protected]> >>> > wrote: >>> > >>> >> On Wed, Nov 4, 2015 at 5:19 PM, Joe Blow <[email protected]> >>> wrote: >>> >> > Radu - My checkpoint interval is set at 100k. Are you suggesting >>> this be >>> >> > lowered? raised? >>> >> >>> >> It sounds like the higher the better, but if your problem is on how >>> >> fast it can read... I think there's not much you can do - that seems >>> >> to be a setting for writes. Also note David's comment on how it might >>> >> only apply if syncing is enabled. >>> >> >>> >> On the read side I don't know what optimization you can do in the >>> >> conf. Maybe you can test with various file sizes? (queue.maxfilesize - >>> >> the default is 1MB so that might be too small) Though I wouldn't have >>> >> high hopes, it sounds like recovery is much too slow even for reading >>> >> 1MB files. >>> >> >>> >> Best regards, >>> >> Radu >>> >> -- >>> >> Performance Monitoring * Log Analytics * Search Analytics >>> >> Solr & Elasticsearch Support * http://sematext.com/ >>> >> _______________________________________________ >>> >> rsyslog mailing list >>> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> >> http://www.rsyslog.com/professional-services/ >>> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> >> DON'T LIKE THAT. >>> >> >>> > _______________________________________________ >>> > rsyslog mailing list >>> > http://lists.adiscon.net/mailman/listinfo/rsyslog >>> > http://www.rsyslog.com/professional-services/ >>> > What's up with rsyslog? Follow https://twitter.com/rgerhards >>> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
