On Wed, 30 Sep 2009, Rainer Gerhards wrote: > Hi all, > > I think we had discussed this some time in the past, but I cannot find a > record of it. So I thought I ask (again?): > > After my bughunt looks almost completed, I have come back to implementing the > name lookup cache. However, I just found out that obtaining the expiration > period of the name lookup seems not to be covered by the "usual" socket > calls. Or did I just miss them?
no they don't, they are name resolution calls, not DNS calls. there are many sources of name resolution (/etc/hosts, LDAP, wins, NIS, etc) and most of them do not have a concept of expiration, and those that do have specific rules for how expiration works (for DNS you have a time after which you are supposed to try and re-resolve it, but can continue to use the name, and a different time after which you are not supposed to use the name for example) > Any advise, comments and hints regarding name caching and expiration would > deeply be appreciated. going back to basics here. Why is this feature desired? why not just use a caching nameserver listening on localhost? 1. doing a name lookup can cause lost logs as you saw when you were doing testing recently, doing name lookups on each received UDP log can cause you to loose log messages when the OS can no longer queue them up. 2. throughput in a high volume site, the cost of doing a name lookup for each log message can be high enough to be a problem. even a local nameserver can be expensive if you are dealing with 10's of thousands of messages/sec what if you were to move the name resolution from the input module to the output module? that would solve problem #1 immediatly by just eliminating any lookups as the messages are received. note: this may not be possible due to name based rules for what hosts to accept logs from, although the answer here may be to lookup the names when you startup and do the filtering by IP while running. if you delay the name resolution until the output module, you may be able to only do it if the output module needs it (if it uses a name property in the template or ruleset), and if it doesn't you skip the work entirely. in anything short of a very high volume site a local caching nameserver will satisfy the throughput issue nicely (especially if the name resolution is delayed to the output as I mentioned above). in a high volume site I really think that it can be good enough to just throw away the name cache when you do a HUP. a high volume site is going to be doing a HUP on a frequent basis anyway to rotate the logs. This avoids a LOT of overhead and complications in managing expirations. In my site I send rsyslog a HUP every 5 min currently (and have some cases where I plan to change this to every 1 min in the near future) David Lang _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
