I figured that was the case with the queue structure. I just wanted to make sure.
As to the performance issue, I admit the primary issue is in relation to impstats as well as queue management. I currently have the instances configured with a max size of 25,000,000 and I find that the output of impstats can be significantly delayed or stop altogether. I have experienced this with 7.2.5, though I haven't upgraded our production instances to 7.4.4 yet. Basically I'm working on building a logging infrastructure that can handle an average of about 5 million messages per minute with bursts as high as 20 million per minute. I have a configuration that works for taking the traffic, putting it to disk and transferring it to other locations, but has trouble outputting to omelasticsearch because the queue associated with it maxes at 1,000,000 and it starts throwing queue full errors. Obviously with a main queue of 25 million, and an average throughput of over 5 million messages per minute, not having the omelasticsearch queue scaled appropriately could pose an issue. Part of this was me clarifying how the queues work and if large queue sizes would potentially impact the overall performance of an instance, but the bigger issue is my inability so far to get the action queue max size to change when it relates to omelasticsearch. I'm going to try changing to FixedArray to see how that works. But any thoughts or insight is appreciated. Thanks! -- James ----- Reply message ----- From: "David Lang" <[email protected]> To: "rsyslog-users" <[email protected]> Subject: [rsyslog] Queues And Max Sizes Date: Sun, Oct 13, 2013 10:04 am I can't answer your questions about the max sizes, but your second question shows a misunderstanding of rsyslog internals There is one main queue (unless you tie rulesets to inputs and define a new queue for the ruleset) input modules add messages to the main queue MainMsgQueueWorkerThreads pull messages off of the main queue and run through the various tests in the config file. If the test results in the message going to a new queue (action queue or ruleset queue), the worker thread transfers the message to that queue, and then ActionQueueWorkerThreads pull messages from that queue to send them out the action (or with rulesets, the process the ruleset and possibly transfer to yet another tier of queues) The only limit I was aware of on queue sizes was available RAM. If you have queues larger than RAM so that you start swapping, performance will plummet. Remember that each queue entry takes up MaxMessageSize + overhead (I don't know how much overhead) I've found that FixedArray is faster than LinkedList for large queues. where did you get the info that things slow down when the main queue gets over 5M entries? David Lang On Sun, 13 Oct 2013, Boylan, James wrote: > Hey all. > > So a few questions. > > Is there a maximum queues size that you should keep an Rsyslog instance > running under? I know that various parts of the applications stop performing > as well once $MainMsgQueueSize gets over about 5 million. For example impstats > really gets unhappy the larger this setting is. Is there a top number that one > should try and keep this queue size under? > > Is the max queue size per $MainMsgQueueWorkerThreads thread or is it the total > of all messages in all threads? > > Finally I know that you are supposed to be able to change queue settings as > described in http://www.rsyslog.com/doc/queues.html which if I'm reading > correctly I should be able to set $ActionQueueSize to increase the maximum > size of the Action queues. However in my testing of them with omelasticsearch, > I find that the maximum appears to be hard locked at 1,000,000. Do I need to > be using $ElasticsearchQueueSize to change that upper limit? I'm currently > running into an issue where the Queue for omelasticsearch is too small to be > able to keep hold the amount of messages the Main Queue is attempting to shift > into it. > > -- James > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
