On Wed, Sep 30, 2009 at 11:15:42AM -0700, [email protected] wrote: > On Wed, 30 Sep 2009, Kenneth Marshall wrote: > > > On Wed, Sep 30, 2009 at 09:55:09AM -0600, RB wrote: > >> On Wed, Sep 30, 2009 at 08:31, Rainer Gerhards <[email protected]> > >> wrote: > >>> After my bughunt looks almost completed, I have come back to implementing > >>> the > >>> name lookup cache. However, I just found out that obtaining the expiration > >>> period of the name lookup seems not to be covered by the "usual" socket > >>> calls. Or did I just miss them? > >> > >> Unfortunately not - most resolver libraries provide only what the > >> programmer usually wants - the symbolic (name) or numeric (IP) result > >> of a query. I've not looked carefully at APIs like res_query, though, > >> and that might bring what you need. > >> > >>> Any advise, comments and hints regarding name caching and expiration would > >>> deeply be appreciated. > >> > >> This was my greatest concern with doing *good* internal caching in > >> rsyslog - you're almost guaranteed to use and/or implement a large > >> chunk of proper resolver functionality. Depending on how readable you > >> find Perl, the Net::DNS infrastructure may provide some good pointers > >> on implementing custom resolution toolkits. The djbdns 'dnscache' > >> program (and perhaps the djbdns client resolver library itself) could > >> also be good pointers. > > > > I do not think that the goal of this feature in rsyslog is to > > re-implement resolver functionality but to provide a fast-path > > mechanism to map IP addresses to names for the purposes of logging > > error messages. As such, pretty much the only piece that needs to > > be tracked within rsyslog is the TTL for the entry and the ip -> > > name mapping. A thread would be responsible for expiring entries > > from the cache (or refreshing the timeout) after validating the > > correctness of the mapping. I think the DNS lookups should be > > handled by a good resolver like pdns-recursor, djbdns,... The > > goal here is to allow names in the log entries and not just IP > > addresses and in a very high performance logging environment. > > the trouble is that doing _proper_ TTL expiration isn't as simple as it > sounds. > > and if you are willing to back away from 'proper' expiration to something > that will work in practice, why not go much further (as I have detailed in > the other messages) > > David Lang
I agree. I only mention TTL values as a reasonable upperbound of the refresh check. The advantage is to remove the large pause in logging due to a DNS refresh after a HUP to rsyslog, if that were the only method to flush/refresh. Like you mention, the IPs/names of systems being logged change rarely so we should tune this for speed and not worry about expiration correctness. I do not agree so other statements that an external cache will perform as well as an internal cache. Too many software products that I work with have needed exactly that functionality to support very high levels of performance. Regards, Ken _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
