Ok i've played with some numbers.... this is what one of the massive queues
looks like now, and it *IS* dequeuing much faster (500 EPS from DA, 25k EPS
from in memory queue.
Hopefully this helps some other people who have very massive, disk backed
queues... Please feel free to comment on these values.
action(type="omelasticsearch"
name="rsys_HugeQ"
server="10.10.10.10"
serverport="9200"
template="HugeQTemplate"
asyncrepl="on"
searchType="HugeType"
searchIndex="HugeQindex"
timeout="3m"
dynSearchIndex="on"
bulkmode="on"
errorfile="HugeQ_err.log"
queue.type="linkedlist"
queue.filename="HugeQ.rsysq"
queue.maxfilesize="2048m"
queue.highwatermark="1000000"
queue.lowwatermark="750000"
queue.discardmark="499999999"
queue.dequeueslowdown="100"
queue.size="500000000"
queue.saveonshutdown="on"
queue.maxdiskspace="1000g"
queue.dequeuebatchsize="50000"
queue.workerthreads="8"
queue.workerthreadminimummessages="100000"
action.resumeretrycount="-1")stop}
I'd love some feedback, but these numbers are working pretty well for these
massive feeds.
Cheers,
JB
On Wed, Nov 4, 2015 at 10:26 AM, Radu Gheorghe <[email protected]>
wrote:
> On Wed, Nov 4, 2015 at 5:19 PM, Joe Blow <[email protected]> wrote:
> > Radu - My checkpoint interval is set at 100k. Are you suggesting this be
> > lowered? raised?
>
> It sounds like the higher the better, but if your problem is on how
> fast it can read... I think there's not much you can do - that seems
> to be a setting for writes. Also note David's comment on how it might
> only apply if syncing is enabled.
>
> On the read side I don't know what optimization you can do in the
> conf. Maybe you can test with various file sizes? (queue.maxfilesize -
> the default is 1MB so that might be too small) Though I wouldn't have
> high hopes, it sounds like recovery is much too slow even for reading
> 1MB files.
>
> Best regards,
> Radu
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
