Thanks for the good discussion. I am lacking somewhat behind, but will review it in depth tomorrow morning.
I just wanted to stress the point that an external cache does not really help, much for the reason David mentioned: if you process messages at very high data rates, the context switch overhead involved with any external solution is extremely costly. Also, in the usual cases, I may do several million queries within a few seconds for just a handful of hosts. With an internal cache, the overhead in doing so is very minimal. With an external solution, the overhead in calling the external cache causes a lot of performance degredation, what in the case of UDP also implies (heavy!) message loss. Rainer > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of [email protected] > Sent: Wednesday, September 30, 2009 8:54 PM > To: rsyslog-users > Subject: Re: [rsyslog] DNS cache and expiration > > On Wed, 30 Sep 2009, Kenneth Marshall wrote: > > >>> mechanism to map IP addresses to names for the purposes of logging > >>> error messages. As such, pretty much the only piece that needs to > >>> be tracked within rsyslog is the TTL for the entry and the ip -> > >>> name mapping. A thread would be responsible for expiring entries > >>> from the cache (or refreshing the timeout) after validating the > >>> correctness of the mapping. I think the DNS lookups should be > >>> handled by a good resolver like pdns-recursor, djbdns,... The > >>> goal here is to allow names in the log entries and not just IP > >>> addresses and in a very high performance logging environment. > >> > >> the trouble is that doing _proper_ TTL expiration isn't as > simple as it > >> sounds. > >> > >> and if you are willing to back away from 'proper' > expiration to something > >> that will work in practice, why not go much further (as I > have detailed in > >> the other messages) > >> > >> David Lang > > > > I agree. I only mention TTL values as a reasonable upperbound of the > > refresh check. The advantage is to remove the large pause in logging > > due to a DNS refresh after a HUP to rsyslog, if that were the only > > method to flush/refresh. Like you mention, the IPs/names of systems > > being logged change rarely so we should tune this for speed and > > not worry about expiration correctness. I do not agree so other > > statements that an external cache will perform as well as > an internal > > cache. Too many software products that I work with have > needed exactly > > that functionality to support very high levels of performance. > > actually, you could have the cache be configurable in three modes > > 1. no caching > > 2. blank the cache on HUP > > 3. never blank the cache (i.e. require a full restart to clear it) > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
