On Sun, Oct 13, 2013 at 7:31 PM, Boylan, James <[email protected]>wrote:
> Turns out I'm already using FixedArray. > > While the rest of the information in the previous emails would be nice to > know/clarify, my primary focus is the changing of the QueueSize for the > omelasticsearch plugin. I've tried changing the queue size in the > configuration of the action, but it doesn't seem to change the end result. > Do I need to configure the $ActionQueueSize as well as the queue size for > the configured omelasticsearch action? > > ie > *.*;syslog.!=debug > action(type="omelasticsearch" > server=" > eslog-master-001.cswm.orbitz.net" > > serverport="9200" > > template="elasticsearchJSON" > > dynSearchIndex="on" > > searchIndex="dynElasticsearchIndex" > > searchType="logs-type" > bulkmode="on" > > queue.type="FixedArray" > > queue.size="5000000" <----- This setting > That should work. Please start rsyslog in debug mode and provide the generated debug log. Startup only is OK, I would just like to see what the config processor and a bit later the queue startup tell. > > queue.dequeuebatchsize="20" > That's VERY low, especially for ES. Each batch requires a http request! > > action.resumeretrycount="-1") > > And $ActionQueueSize as well? > > No, not at all. Old-style does never affect new-style (was a design goal). Rainer > -- James > ________________________________________ > From: [email protected] [[email protected]] > On Behalf Of Boylan, James [[email protected]] > Sent: Sunday, October 13, 2013 11:31 AM > To: [email protected]; [email protected] > Subject: Re: [rsyslog] Queues And Max Sizes > > I figured that was the case with the queue structure. I just wanted to > make sure. > > As to the performance issue, I admit the primary issue is in relation to > impstats as well as queue management. I currently have the instances > configured with a max size of 25,000,000 and I find that the output of > impstats can be significantly delayed or stop altogether. I have > experienced this with 7.2.5, though I haven't upgraded our production > instances to 7.4.4 yet. > > Basically I'm working on building a logging infrastructure that can handle > an average of about 5 million messages per minute with bursts as high as 20 > million per minute. I have a configuration that works for taking the > traffic, putting it to disk and transferring it to other locations, but has > trouble outputting to omelasticsearch because the queue associated with it > maxes at 1,000,000 and it starts throwing queue full errors. > > Obviously with a main queue of 25 million, and an average throughput of > over 5 million messages per minute, not having the omelasticsearch queue > scaled appropriately could pose an issue. > > Part of this was me clarifying how the queues work and if large queue > sizes would potentially impact the overall performance of an instance, but > the bigger issue is my inability so far to get the action queue max size to > change when it relates to omelasticsearch. > > I'm going to try changing to FixedArray to see how that works. But any > thoughts or insight is appreciated. > > Thanks! > > -- James > > ----- Reply message ----- > From: "David Lang" <[email protected]> > To: "rsyslog-users" <[email protected]> > Subject: [rsyslog] Queues And Max Sizes > Date: Sun, Oct 13, 2013 10:04 am > > > > I can't answer your questions about the max sizes, but your second question > shows a misunderstanding of rsyslog internals > > There is one main queue (unless you tie rulesets to inputs and define a new > queue for the ruleset) > > input modules add messages to the main queue > > MainMsgQueueWorkerThreads pull messages off of the main queue and run > through > the various tests in the config file. If the test results in the message > going > to a new queue (action queue or ruleset queue), the worker thread > transfers the > message to that queue, and then ActionQueueWorkerThreads pull messages > from that > queue to send them out the action (or with rulesets, the process the > ruleset and > possibly transfer to yet another tier of queues) > > The only limit I was aware of on queue sizes was available RAM. If you have > queues larger than RAM so that you start swapping, performance will > plummet. > > Remember that each queue entry takes up MaxMessageSize + overhead (I don't > know > how much overhead) > > I've found that FixedArray is faster than LinkedList for large queues. > > where did you get the info that things slow down when the main queue gets > over > 5M entries? > > David Lang > > On Sun, 13 Oct 2013, Boylan, James wrote: > > > Hey all. > > > > So a few questions. > > > > Is there a maximum queues size that you should keep an Rsyslog instance > > running under? I know that various parts of the applications stop > performing > > as well once $MainMsgQueueSize gets over about 5 million. For example > impstats > > really gets unhappy the larger this setting is. Is there a top number > that one > > should try and keep this queue size under? > > > > Is the max queue size per $MainMsgQueueWorkerThreads thread or is it the > total > > of all messages in all threads? > > > > Finally I know that you are supposed to be able to change queue settings > as > > described in http://www.rsyslog.com/doc/queues.html which if I'm reading > > correctly I should be able to set $ActionQueueSize to increase the > maximum > > size of the Action queues. However in my testing of them with > omelasticsearch, > > I find that the maximum appears to be hard locked at 1,000,000. Do I > need to > > be using $ElasticsearchQueueSize to change that upper limit? I'm > currently > > running into an issue where the Queue for omelasticsearch is too small > to be > > able to keep hold the amount of messages the Main Queue is attempting to > shift > > into it. > > > > -- James > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
