On Wed, 2 Jul 2014, Masuda, Bond wrote:
hi rsyslogers:
we've been doing some load testing of syslog messages over UDP/514 to rsyslog.
we write all incoming messages to a file in /data/logs/incoming-all.log.
In our load test, we generated about 29 million messages in 300 seconds. On the
server side, we are receiving about 25 million messages; and about 4 million
messages are lost on the network (not an rsyslog issue). However, of the 25
million messages we know arrive at the server, we are also seeing message lost
in /data/logs/incoming-all.log, albeit to a much lesser degree than the network
problem.
The actual numbers are:
29,561,113 messages generated and sent in 300 seconds
24,802,441 messages arrive at the rsyslog server (counting UDP packets via
NETFILTER/mangle-PREROUTING accounting rule)
24,774,587 messages written to /data/logs/incoming-all.log
So it would seem that we lost 27854 messages within rsyslog.
My question is this:
1. Does rsyslog drop messages when its message queues are overflowing?
Yes, if the queue is full, what else can rsyslog do?
2. If answer to #1 is yes, does it keep any accounting of the lost messages and
how can I see those numbers? or at least warn that its queues are overflowing?
yes, enable impstats
3. if answer to #1 is yes, is there some configuration setting to make rsyslog
guarantee not to drop messages, potentially as trade off with some other
problem? Or is it just a matter of increasing queue sizes?
not with UDP, because if rsyslog can't deal with the message _NOW_ the OS will
drop the packet
4. If answer to #1 is no, what's the best way to go about troubleshooting why
messages are being lost?
BTW, under less stressful conditions, all the in/out numbers perfectly match.
We only start seeing "lost messages/packets" when we go above ~50,000 messages
per second.
first question, what version are you running, 8.2 is worlds better than 5.x
after that, have you tried disabling DNS lookups
run tcpdump (saving to a file, not outputting to the screen) to check if the
packets are really being lost on the network or if they are being lost in the
network stack of the receiving machine)
configure impstats to see what your queue sizes look like
what are you doing with the logs? it's possible that your bottleneck is in
outputting the logs and fixing that will solve your problem
you could configure disk assisted queues to spool logs to disk, but that is
slower than just outputting them to a file, so it doesn't solve the problem
a tuned rsyslog setup can do gig-E wire speed or faster, you are well below that
limit, so we should be able to help you.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
