On Wed, 23 Dec 2009, Brad Knowles wrote: >> You could probably also log to Syslog and have it write those logs to a >> remote Syslog server rather than writing them locally. That won't >> require a network-aware logging module. > > THe disadvantage to remote syslog is that the default protocol is UDP, > and in my experience is very sensitive to latency. On networks that > were less than 1% utilized, I've seen UDP syslog drop over 75% of all > the log data. Packages like syslog-ng allow you the option of doing > syslog over a TCP connection, but that also has some issues.
I see this as more likely a problem with the receiving software than with UDP itself (although it could also be your network gear) using a tuned rsyslog receiver on a mid-range switch (Cisco 3650) I have basicly hit wire speed Gig-E without packet loss (250 byte messages at >375,000 messages/sec, up to ram capacity of the receiver, rsyslog could not write messages to disk this quickly) remember that the old syslog daemon would receive the packet, do a reverse DNS lookup on the source IP, parse the message, process the message through it's filters, write the message to disk (with an fsync), before looking for he next packet to process. just eliminating the DNS lookup would save huge amounts of time in this process and drasticly cut down on the packet loss. but the fact that all of the other processing needs to be done for each packet can cause a problem if you don't have large enough OS buffers to queue up a burst of traffic. Over the last year rsyslog has gotten a lot of performance attention (I've driven a lot of it with my testing) and is looking very good for high performance logging now. I haven't had a chance to stress test the latest version (which includes significant output side improvements), but it can do an amazingly good job at handling bursts of traffic. it does this at the expense of reliability by default, but I have also tested the extreme safety mode (where it writes everything to disk at every step), and there it could do up to ~8k messages/sec with good hardware and the right filesystem (same hardware with the wrong filesystem couldn't reach 2K messages/sec) David Lang _______________________________________________ Tech mailing list [email protected] http://lopsa.org/cgi-bin/mailman/listinfo/tech This list provided by the League of Professional System Administrators http://lopsa.org/
