1- There are about 13000 files of 1.1 MB accumulated to 13 GB on disk. When
I remove low and high water marks on main and action queue to make it only
In-memmory queues, it still create these files. Why is that?
what makes the disk queues isn't the watermark settings, but the fact that you
have a queue.filename
2- How about if rsyslog receive at 30000EPS but main queue is about 500k
and action queue 100k.
what's your question? a 500K main queue and 100K action queue may be just fine
at 30K logs/sec, if you can deliver the logs at that rate. If you can't deliver
the logs that fast, no amount of queue is going to be enough.
3- what is meant by dequeuebatch size and how can i add delay to control
speed.
rsyslog adds logs to the queues as they arrive with threads running the input
modules.
Other threads (worker threads) grab a 'batch' of messages from the queue, and
then run through the ruleset, doing the tests and then running the code from
output modules to deliver the logs (or if there is an action or ruleset that's
called that has it's own queue, the worker thread delivers the message(s) to
that queue and another layer of worker threads runs to read messages from those
queues)
dequeuebatchsize is the maximum number of logs that a worker thread will pull
from the queue in one loop through the process.
The reason for batches instead of always pulling individual messages is twofold
1. there is a significant amount of overhead to manipulate the locks on the
queue, only having to do so once per 100 (or 1000) messages instead of once per
message is a very significant speed improvement
2. for some output modules (think datbases), there is very little difference in
the cost of delivering a lot of messages instead of one message (for example, I
tested inserts into postgres about a decade ago, I could insert 1000 records as
one transaction in the same time that I could insert 2 records as 2 separate
transactions). Batches give the output modules the ability to handle multiple
records at once.
so if you have the dequeuebatchdelay set for 1/10 of a second, then rsyslog will
deliver slightly less than 10 batches of logs every second max. If each batch is
1 message, this will be just under 10 logs per second. If each batch can be 100
logs, than this could be up to 1000 logs per second
I say just under because it takes time to process the logs, so 1/10 sec of sleep
+ log processing time will be > 1 second, so you would not quite process 10
batches in a second, but you would probably be close to processing 99 batches in
100 seconds, so it's not much less.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
