Dear David,
You always have been very helpful for us. I have solved many problem of our
rsyslog. Could you please give me some suggestions.
If I received messages at about 20000EPS, what main queue and action queue
size dequeuebatch size should I use.
Is there anyway to get stats of messages received on TCP port and how can
we clearly check how much RAM is being used by queues.
Logs are being written in syslog file with 20000 EPS but to control
processing at fluentd, i am using dequeuebatch size 2000 EPS with delay of
1 sec. I think there is some wrong.
main_queue(
#queue.filename="main_queue" # write to disk if needed
#queue.maxdiskspace="1g" # when to stop writing to disk
#queue.highwatermark="400000" # start spilling to disk at this size
#queue.lowwatermark="100000" # stop spilling when it gets back to this size
#queue.saveonshutdown="on" # write queue contents to disk on shutdown
queue.dequeueBatchSize="2000"
queue.workerthreads="2"
queue.size="500000" # absolute max queue size
)
if ($fromhost-ip != "127.0.0.1" and $fromhost-ip != "10.16.28.160" and
$fromhost-ip != "192.168.10.34" and $fromhost-ip != "172.18.40.134" and
$fromhost-ip != "192.168.122.1") then {
action(type="omfwd" target="127.0.0.1" port="5170" protocol="tcp"
template="msgonly" queue.size="100000"
queue.dequeuebatchsize="2000" queue.dequeueslowdown="1000000"
queue.type="LinkedList" )
stop ## To Avoid writing in syslog
}
I only have 6 GB of RAM.
Thanks.
On Fri, Dec 18, 2015 at 10:32 AM, David Lang <[email protected]> wrote:
> 1- There are about 13000 files of 1.1 MB accumulated to 13 GB on disk. When
>> I remove low and high water marks on main and action queue to make it only
>> In-memmory queues, it still create these files. Why is that?
>>
>
> what makes the disk queues isn't the watermark settings, but the fact that
> you have a queue.filename
>
> 2- How about if rsyslog receive at 30000EPS but main queue is about 500k
>> and action queue 100k.
>>
>
> what's your question? a 500K main queue and 100K action queue may be just
> fine at 30K logs/sec, if you can deliver the logs at that rate. If you
> can't deliver the logs that fast, no amount of queue is going to be enough.
>
> 3- what is meant by dequeuebatch size and how can i add delay to control
>> speed.
>>
>
> rsyslog adds logs to the queues as they arrive with threads running the
> input modules.
>
> Other threads (worker threads) grab a 'batch' of messages from the queue,
> and then run through the ruleset, doing the tests and then running the code
> from output modules to deliver the logs (or if there is an action or
> ruleset that's called that has it's own queue, the worker thread delivers
> the message(s) to that queue and another layer of worker threads runs to
> read messages from those queues)
>
> dequeuebatchsize is the maximum number of logs that a worker thread will
> pull from the queue in one loop through the process.
>
> The reason for batches instead of always pulling individual messages is
> twofold
>
> 1. there is a significant amount of overhead to manipulate the locks on
> the queue, only having to do so once per 100 (or 1000) messages instead of
> once per message is a very significant speed improvement
>
> 2. for some output modules (think datbases), there is very little
> difference in the cost of delivering a lot of messages instead of one
> message (for example, I tested inserts into postgres about a decade ago, I
> could insert 1000 records as one transaction in the same time that I could
> insert 2 records as 2 separate transactions). Batches give the output
> modules the ability to handle multiple records at once.
>
> so if you have the dequeuebatchdelay set for 1/10 of a second, then
> rsyslog will deliver slightly less than 10 batches of logs every second
> max. If each batch is 1 message, this will be just under 10 logs per
> second. If each batch can be 100 logs, than this could be up to 1000 logs
> per second
>
> I say just under because it takes time to process the logs, so 1/10 sec of
> sleep + log processing time will be > 1 second, so you would not quite
> process 10 batches in a second, but you would probably be close to
> processing 99 batches in 100 seconds, so it's not much less.
>
>
> David Lang
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
