On Wed, 22 Feb 2012, Chris McCraw wrote:
Hi list,
Longtime user, first time optimizer of rsyslog. Here's my situation:
We just upgraded a machine that gets a ridiculous amount of log
traffic from one IP (our load balancer)--firehose levels, hundreds of
MB/minute. This machine also takes logs of a few dozen low-traffic
servers on the same subnet. With the upgrade from v4.6.2 to v5.8.5,
we gained UDP Multiruleset binding, yay! We've moved all of our
logging via the firehose from TCP to UDP, because the TCP logging was
very fragile and would simply stop if the rsyslog restart for log
rotation took a microsecond too long.
Logging works great. Our nameserver load shot way up, because it
seems our TCP-only 4.6.2 setup was not doing a DNS lookup for every
message...yet using the same file (with the addition of the UDP
ruleset binding) with v5.8.5 and -c5 instead of -c4 on the command
line for rsyslog has changed the lookup behavior of rsyslog, and named
is spinning constantly, presumably on the same host name.
Any pointers to the docs on how to mitigate this? We're open to any
number of solutions (hopefully not including upgrading to v6)--put all
hostnames in /etc/hosts, for instance. Since the firehose is all
bound to specific files anyway, those logs don't even need DNS
lookups--we know exactly where they come from. We don't want to turn
off DNS entirely if we can avoid it, but we could partition into
"normal port 514 tcp traffic gets lookups and other port UDP traffic
doesn't". I'm guessing there is more than one way to do this =)
Unfortunantly in v5 it's all or nothing, -x will disable DNS lookups
entirely. It does have a little bit of caching so that it won't lookup the
same IP again and again, but that doesn't help a lot if you have a bunch
of active sources.
putting the names in /etc/hosts will help, but there's still a very
significant amount of time spend doing the lookup, even if everything is
local.
version 6.3 has a full DNS cache.
other than that, consider running two copies of rsyslog, one with DNS
disabled and one with it enabled. you don't even need to forward from one
to the other in your case.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
