Bond, Yes, rsyslog will drop messages when its queues overflow. Statistics about that can be gathered by the impstats module. See this page for details:
http://www.rsyslog.com/how-to-use-impstats/ Also, it would help to know what version of rsyslog you're using, and your config file, etc.. I'm interested in knowing more about your setup, as I'm routinely handling 90k+ EPS (or so I think!) and rsyslog doesn't seem to be taxed at all. I'm pretty sure that people here will speak of their setups handling 100k+ EPS as well. As for your other questions: 1. Yes 2. Yes, see impstats above 3. I don't think so. You could throttle the input, but not UDP, and eventually the queue will just fill up and you'll have to discard. 4. Need more info about config, etc. to start troubleshooting Cheers! Robert -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Masuda, Bond Sent: Wednesday, July 2, 2014 1:05 PM To: rsyslog-users Subject: [rsyslog] question on reliability of message processing hi rsyslogers: we've been doing some load testing of syslog messages over UDP/514 to rsyslog. we write all incoming messages to a file in /data/logs/incoming-all.log. In our load test, we generated about 29 million messages in 300 seconds. On the server side, we are receiving about 25 million messages; and about 4 million messages are lost on the network (not an rsyslog issue). However, of the 25 million messages we know arrive at the server, we are also seeing message lost in /data/logs/incoming-all.log, albeit to a much lesser degree than the network problem. The actual numbers are: 29,561,113 messages generated and sent in 300 seconds 24,802,441 messages arrive at the rsyslog server (counting UDP packets via NETFILTER/mangle-PREROUTING accounting rule) 24,774,587 messages written to /data/logs/incoming-all.log So it would seem that we lost 27854 messages within rsyslog. My question is this: 1. Does rsyslog drop messages when its message queues are overflowing? 2. If answer to #1 is yes, does it keep any accounting of the lost messages and how can I see those numbers? or at least warn that its queues are overflowing? 3. if answer to #1 is yes, is there some configuration setting to make rsyslog guarantee not to drop messages, potentially as trade off with some other problem? Or is it just a matter of increasing queue sizes? 4. If answer to #1 is no, what's the best way to go about troubleshooting why messages are being lost? BTW, under less stressful conditions, all the in/out numbers perfectly match. We only start seeing "lost messages/packets" when we go above ~50,000 messages per second. Would appreciate any insights... -Bond _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
