I can't answer your questions about the max sizes, but your second question
shows a misunderstanding of rsyslog internals
There is one main queue (unless you tie rulesets to inputs and define a new
queue for the ruleset)
input modules add messages to the main queue
MainMsgQueueWorkerThreads pull messages off of the main queue and run through
the various tests in the config file. If the test results in the message going
to a new queue (action queue or ruleset queue), the worker thread transfers the
message to that queue, and then ActionQueueWorkerThreads pull messages from that
queue to send them out the action (or with rulesets, the process the ruleset and
possibly transfer to yet another tier of queues)
The only limit I was aware of on queue sizes was available RAM. If you have
queues larger than RAM so that you start swapping, performance will plummet.
Remember that each queue entry takes up MaxMessageSize + overhead (I don't know
how much overhead)
I've found that FixedArray is faster than LinkedList for large queues.
where did you get the info that things slow down when the main queue gets over
5M entries?
David Lang
On Sun, 13 Oct 2013, Boylan, James wrote:
Hey all.
So a few questions.
Is there a maximum queues size that you should keep an Rsyslog instance
running under? I know that various parts of the applications stop performing
as well once $MainMsgQueueSize gets over about 5 million. For example impstats
really gets unhappy the larger this setting is. Is there a top number that one
should try and keep this queue size under?
Is the max queue size per $MainMsgQueueWorkerThreads thread or is it the total
of all messages in all threads?
Finally I know that you are supposed to be able to change queue settings as
described in http://www.rsyslog.com/doc/queues.html which if I'm reading
correctly I should be able to set $ActionQueueSize to increase the maximum
size of the Action queues. However in my testing of them with omelasticsearch,
I find that the maximum appears to be hard locked at 1,000,000. Do I need to
be using $ElasticsearchQueueSize to change that upper limit? I'm currently
running into an issue where the Queue for omelasticsearch is too small to be
able to keep hold the amount of messages the Main Queue is attempting to shift
into it.
-- James
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
