On Sun, Oct 13, 2013 at 6:31 PM, Boylan, James <[email protected]>wrote:
> I figured that was the case with the queue structure. I just wanted to > make sure. > > As to the performance issue, I admit the primary issue is in relation to > impstats as well as queue management. I currently have the instances > configured with a max size of 25,000,000 and I find that the output of > impstats can be significantly delayed or stop altogether. I have > experienced this with 7.2.5, though I haven't upgraded our production > instances to 7.4.4 yet. > I guess you run impstats in-band (the default). In that case, it's output goes to the main queue, as any other message does. So if the main queue takes looong to processs, impstats messages get delayed as well. They may be discarded, too, based on queue settings and queue full status. Note that you can run impstats of of band, by instructing it to directly write to file. There is nothing inherant in impstats or rsyslog that prevents large queues, as long as there is sufficient physical main memory available. > Basically I'm working on building a logging infrastructure that can handle > an average of about 5 million messages per minute with bursts as high as 20 > million per minute. I have a configuration that works for taking the > traffic, putting it to disk and transferring it to other locations, but has > trouble outputting to omelasticsearch because the queue associated with it > maxes at 1,000,000 and it starts throwing queue full errors. > > Obviously with a main queue of 25 million, and an average throughput of > over 5 million messages per minute, not having the omelasticsearch queue > scaled appropriately could pose an issue. > > Part of this was me clarifying how the queues work and if large queue > sizes would potentially impact the overall performance of an instance, but > the bigger issue is my inability so far to get the action queue max size to > change when it relates to omelasticsearch. > > I don't understand this fully. Highly suggest to let impstats log to file and post the result here. > I'm going to try changing to FixedArray to see how that works. But any > thoughts or insight is appreciated. > > As you deal with a slow output, I don't think this will have much effect. Rainer > Thanks! > > -- James > > ----- Reply message ----- > From: "David Lang" <[email protected]> > To: "rsyslog-users" <[email protected]> > Subject: [rsyslog] Queues And Max Sizes > Date: Sun, Oct 13, 2013 10:04 am > > > > I can't answer your questions about the max sizes, but your second question > shows a misunderstanding of rsyslog internals > > There is one main queue (unless you tie rulesets to inputs and define a new > queue for the ruleset) > > input modules add messages to the main queue > > MainMsgQueueWorkerThreads pull messages off of the main queue and run > through > the various tests in the config file. If the test results in the message > going > to a new queue (action queue or ruleset queue), the worker thread > transfers the > message to that queue, and then ActionQueueWorkerThreads pull messages > from that > queue to send them out the action (or with rulesets, the process the > ruleset and > possibly transfer to yet another tier of queues) > > The only limit I was aware of on queue sizes was available RAM. If you have > queues larger than RAM so that you start swapping, performance will > plummet. > > Remember that each queue entry takes up MaxMessageSize + overhead (I don't > know > how much overhead) > > I've found that FixedArray is faster than LinkedList for large queues. > > where did you get the info that things slow down when the main queue gets > over > 5M entries? > > David Lang > > On Sun, 13 Oct 2013, Boylan, James wrote: > > > Hey all. > > > > So a few questions. > > > > Is there a maximum queues size that you should keep an Rsyslog instance > > running under? I know that various parts of the applications stop > performing > > as well once $MainMsgQueueSize gets over about 5 million. For example > impstats > > really gets unhappy the larger this setting is. Is there a top number > that one > > should try and keep this queue size under? > > > > Is the max queue size per $MainMsgQueueWorkerThreads thread or is it the > total > > of all messages in all threads? > > > > Finally I know that you are supposed to be able to change queue settings > as > > described in http://www.rsyslog.com/doc/queues.html which if I'm reading > > correctly I should be able to set $ActionQueueSize to increase the > maximum > > size of the Action queues. However in my testing of them with > omelasticsearch, > > I find that the maximum appears to be hard locked at 1,000,000. Do I > need to > > be using $ElasticsearchQueueSize to change that upper limit? I'm > currently > > running into an issue where the Queue for omelasticsearch is too small > to be > > able to keep hold the amount of messages the Main Queue is attempting to > shift > > into it. > > > > -- James > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
