Thanks David for being in touch. Please give your expert comments. http://pastebin.com/FRzdy7Ff
On Tue, Dec 22, 2015 at 1:57 AM, David Lang <[email protected]> wrote: > On Mon, 21 Dec 2015, Muhammad Asif wrote: > > Dear David, >> >> You always have been very helpful for us. I have solved many problem of >> our >> rsyslog. Could you please give me some suggestions. >> >> If I received messages at about 20000EPS, what main queue and action queue >> size dequeuebatch size should I use. >> Is there anyway to get stats of messages received on TCP port and how can >> we clearly check how much RAM is being used by queues. >> > > configure the impstats module, it gives you all these details > > Logs are being written in syslog file with 20000 EPS but to control >> processing at fluentd, i am using dequeuebatch size 2000 EPS with delay of >> 1 sec. I think there is some wrong. >> > > This will cauuse rsyslog to send up to 20000 messages in one chunk, then > sleep for a second. you probably want to do something more like batch size > of 2000 and sleep for 1/10 of a second (or 200 and 1/100 of a second) so > that the logs are far more spread out > > what are the limits in fluentd and what happens when you exceed them? > > David Lang > > > main_queue( >> #queue.filename="main_queue" # write to disk if needed >> #queue.maxdiskspace="1g" # when to stop writing to disk >> #queue.highwatermark="400000" # start spilling to disk at this size >> #queue.lowwatermark="100000" # stop spilling when it gets back to this >> size >> #queue.saveonshutdown="on" # write queue contents to disk on shutdown >> queue.dequeueBatchSize="2000" >> queue.workerthreads="2" >> queue.size="500000" # absolute max queue size >> ) >> >> >> if ($fromhost-ip != "127.0.0.1" and $fromhost-ip != "10.16.28.160" and >> $fromhost-ip != "192.168.10.34" and $fromhost-ip != "172.18.40.134" and >> $fromhost-ip != "192.168.122.1") then { >> action(type="omfwd" target="127.0.0.1" port="5170" protocol="tcp" >> template="msgonly" queue.size="100000" >> queue.dequeuebatchsize="2000" queue.dequeueslowdown="1000000" >> queue.type="LinkedList" ) >> stop ## To Avoid writing in syslog >> } >> >> I only have 6 GB of RAM. >> >> >> Thanks. >> >> >> >> On Fri, Dec 18, 2015 at 10:32 AM, David Lang <[email protected]> wrote: >> >> 1- There are about 13000 files of 1.1 MB accumulated to 13 GB on disk. >>> When >>> >>>> I remove low and high water marks on main and action queue to make it >>>> only >>>> In-memmory queues, it still create these files. Why is that? >>>> >>>> >>> what makes the disk queues isn't the watermark settings, but the fact >>> that >>> you have a queue.filename >>> >>> 2- How about if rsyslog receive at 30000EPS but main queue is about 500k >>> >>>> and action queue 100k. >>>> >>>> >>> what's your question? a 500K main queue and 100K action queue may be just >>> fine at 30K logs/sec, if you can deliver the logs at that rate. If you >>> can't deliver the logs that fast, no amount of queue is going to be >>> enough. >>> >>> 3- what is meant by dequeuebatch size and how can i add delay to control >>> >>>> speed. >>>> >>>> >>> rsyslog adds logs to the queues as they arrive with threads running the >>> input modules. >>> >>> Other threads (worker threads) grab a 'batch' of messages from the queue, >>> and then run through the ruleset, doing the tests and then running the >>> code >>> from output modules to deliver the logs (or if there is an action or >>> ruleset that's called that has it's own queue, the worker thread delivers >>> the message(s) to that queue and another layer of worker threads runs to >>> read messages from those queues) >>> >>> dequeuebatchsize is the maximum number of logs that a worker thread will >>> pull from the queue in one loop through the process. >>> >>> The reason for batches instead of always pulling individual messages is >>> twofold >>> >>> 1. there is a significant amount of overhead to manipulate the locks on >>> the queue, only having to do so once per 100 (or 1000) messages instead >>> of >>> once per message is a very significant speed improvement >>> >>> 2. for some output modules (think datbases), there is very little >>> difference in the cost of delivering a lot of messages instead of one >>> message (for example, I tested inserts into postgres about a decade ago, >>> I >>> could insert 1000 records as one transaction in the same time that I >>> could >>> insert 2 records as 2 separate transactions). Batches give the output >>> modules the ability to handle multiple records at once. >>> >>> so if you have the dequeuebatchdelay set for 1/10 of a second, then >>> rsyslog will deliver slightly less than 10 batches of logs every second >>> max. If each batch is 1 message, this will be just under 10 logs per >>> second. If each batch can be 100 logs, than this could be up to 1000 logs >>> per second >>> >>> I say just under because it takes time to process the logs, so 1/10 sec >>> of >>> sleep + log processing time will be > 1 second, so you would not quite >>> process 10 batches in a second, but you would probably be close to >>> processing 99 batches in 100 seconds, so it's not much less. >>> >>> >>> David Lang >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >>> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >> _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
