> -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of [email protected] > Sent: Wednesday, September 30, 2009 9:53 PM > To: rsyslog-users > Subject: Re: [rsyslog] DNS cache and expiration > > On Wed, 30 Sep 2009, Rainer Gerhards wrote: > > > Thanks for the good discussion. I am lacking somewhat > behind, but will review > > it in depth tomorrow morning. > > > > I just wanted to stress the point that an external cache > does not really > > help, much for the reason David mentioned: if you process > messages at very > > high data rates, the context switch overhead involved with > any external > > solution is extremely costly. Also, in the usual cases, I > may do several > > million queries within a few seconds for just a handful of > hosts. With an > > internal cache, the overhead in doing so is very minimal. > With an external > > solution, the overhead in calling the external cache causes a lot of > > performance degredation, what in the case of UDP also > implies (heavy!) > > message loss. > > the message loss problem with UDP will not be solved completely by an > internal cache. when the source is not in the cache and you > have to go out > to find it the lookup can take several seconds. > > moving the lookup out of the input module and into the output > module would > address this, anything else would leave you with losses as > the cache gets > populated.
That's right and that's one reason why I intend to move this (optionally) over to the "backend" processing. However, even that does not completely solve the message loss problem, as we, in extreme cases, may loose messages when the queue is full - and for a myriad of other reasons, like routers discarding frames and such. Of course, you know that, but I'd like to mention if for those folks that at some time find our conversation via Google ;) Rainer > > David Lang > > > Rainer > > > >> -----Original Message----- > >> From: [email protected] > >> [mailto:[email protected]] On Behalf Of > [email protected] > >> Sent: Wednesday, September 30, 2009 8:54 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] DNS cache and expiration > >> > >> On Wed, 30 Sep 2009, Kenneth Marshall wrote: > >> > >>>>> mechanism to map IP addresses to names for the purposes > of logging > >>>>> error messages. As such, pretty much the only piece > that needs to > >>>>> be tracked within rsyslog is the TTL for the entry and the ip -> > >>>>> name mapping. A thread would be responsible for expiring entries > >>>>> from the cache (or refreshing the timeout) after validating the > >>>>> correctness of the mapping. I think the DNS lookups should be > >>>>> handled by a good resolver like pdns-recursor, djbdns,... The > >>>>> goal here is to allow names in the log entries and not just IP > >>>>> addresses and in a very high performance logging environment. > >>>> > >>>> the trouble is that doing _proper_ TTL expiration isn't as > >> simple as it > >>>> sounds. > >>>> > >>>> and if you are willing to back away from 'proper' > >> expiration to something > >>>> that will work in practice, why not go much further (as I > >> have detailed in > >>>> the other messages) > >>>> > >>>> David Lang > >>> > >>> I agree. I only mention TTL values as a reasonable > upperbound of the > >>> refresh check. The advantage is to remove the large pause > in logging > >>> due to a DNS refresh after a HUP to rsyslog, if that were the only > >>> method to flush/refresh. Like you mention, the IPs/names > of systems > >>> being logged change rarely so we should tune this for speed and > >>> not worry about expiration correctness. I do not agree so other > >>> statements that an external cache will perform as well as > >> an internal > >>> cache. Too many software products that I work with have > >> needed exactly > >>> that functionality to support very high levels of performance. > >> > >> actually, you could have the cache be configurable in three modes > >> > >> 1. no caching > >> > >> 2. blank the cache on HUP > >> > >> 3. never blank the cache (i.e. require a full restart to clear it) > >> > >> David Lang > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
