On Thu, 1 Oct 2009, Rainer Gerhards wrote: > I have now reviewed the discussion in depth. Many thanks for all the good > thoughts. > > I will now first begin to move the name resolution calls to the "backend" > threads (away from the inputs). However, I will create an option that the > name resolution may be done before enqueing the message, which may be used by > some folks for whatever reason (plus, it is trivially to do). I also think I > will leave in the capability that inputs do the name resolution themselfs, at > least if they prefer. For example, imuxsock always has the local host as > input, so it makes limited sense to do name resolution calls for messages > received from that input. Similarly, imtcp uses the name that was (once) > obtained when the session was created (btw, this is also some expiry-less > "name resolution", but I think here it is even harder to argue against it > ;)).
on note on this, if you can delay the resolution to the point where something actually accesses the property that uses the name you may be able to skip doing name resolution entirely in many cases. David Lang > When I am done with this, I have a nice central place where the name > resolution is done. In that place, I will then build a central name cache, at > first with only two options > > - cache disabled > - cache without expiration (requires shutdown) > > After that, I'll probably look into HUP plus probably provide a maximum TTL > to rsyslog that says e.g. "check entry after n seconds". Even if n is low > (maybe 10 seconds), the extra effort for doing these lookups is minimal: > > If the system has low throughput, the lookup cost is irrelevant in any case. > If it is a high performance system, with more than e.g. 10,000 msgs/s, then > n=10 means we do one lookup for at most 1/100,000th message, which doesn't > matter either. > > Most importantly, I do not like the idea to use a third-party resolver > library, as this creates another set of dependencies, and this for a core > module. Booting rsyslog would get even more complicated. So I think the > "expire ever n seconds" approach is a very practical one. > > You will see move these changes gradually into the master branch. > > One word on the performance: an avl-tree based internal cache can resolve > names *very* fast. It has a depth of almost log n where n being the number of > names cached. Building and maintaining the tree is almost for free, because > the structure is rather static. Expiration, as I intend to implement it, is > also a very simple operation, the node does not need to be removed and > re-inserted but rather I need to update the resolution info by means of a new > lookup (what costs the majority of time). So for a typical search, I would > assume that the name can be resolved with less than 500 machine instructions > (what I consider a conservative choice). The way I intend to implement it > probably does not include any mutex calls, so there is no context switch > involved. > > I consider the cost of a single context switch to be much higher than such an > avl-tree lookup. Now consider any external solution. You need to do at least > four context switches (rsyslog -> OS -> resolver -> OS -> rsyslog), probably > more. Also, at least the OS scheduler is involved to activate the threads > that are synchronized, plus probably a bit more of OS logic. Then, the > external resolver can at best perform at the same rate rsyslog itself can > (using a similar algorithm), but probably slower because it must be more > generic. Looking at this picture, I think, clarifies why an (proper) internal > implementation is so much more efficient than any external solution. Also > note that I assume that in almost all cases, due to the structure of syslog > traffic, no real lookup needs to be done, everything sits already in the > resolver caches. > > I hope this is useful information. If someone has any additional comments, I > would love to hear them. > > Rainer > >> -----Original Message----- >> From: [email protected] [mailto:rsyslog- >> [email protected]] On Behalf Of Rainer Gerhards >> Sent: Wednesday, September 30, 2009 9:57 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] DNS cache and expiration >> >>> -----Original Message----- >>> From: [email protected] >>> [mailto:[email protected]] On Behalf Of [email protected] >>> Sent: Wednesday, September 30, 2009 9:53 PM >>> To: rsyslog-users >>> Subject: Re: [rsyslog] DNS cache and expiration >>> >>> On Wed, 30 Sep 2009, Rainer Gerhards wrote: >>> >>>> Thanks for the good discussion. I am lacking somewhat >>> behind, but will review >>>> it in depth tomorrow morning. >>>> >>>> I just wanted to stress the point that an external cache >>> does not really >>>> help, much for the reason David mentioned: if you process >>> messages at very >>>> high data rates, the context switch overhead involved with >>> any external >>>> solution is extremely costly. Also, in the usual cases, I >>> may do several >>>> million queries within a few seconds for just a handful of >>> hosts. With an >>>> internal cache, the overhead in doing so is very minimal. >>> With an external >>>> solution, the overhead in calling the external cache causes a lot >> of >>>> performance degredation, what in the case of UDP also >>> implies (heavy!) >>>> message loss. >>> >>> the message loss problem with UDP will not be solved completely by an >>> internal cache. when the source is not in the cache and you >>> have to go out >>> to find it the lookup can take several seconds. >>> >>> moving the lookup out of the input module and into the output >>> module would >>> address this, anything else would leave you with losses as >>> the cache gets >>> populated. >> >> That's right and that's one reason why I intend to move this >> (optionally) >> over to the "backend" processing. However, even that does not >> completely >> solve the message loss problem, as we, in extreme cases, may loose >> messages >> when the queue is full - and for a myriad of other reasons, like >> routers >> discarding frames and such. Of course, you know that, but I'd like to >> mention >> if for those folks that at some time find our conversation via Google >> ;) >> >> Rainer >>> >>> David Lang >>> >>>> Rainer >>>> >>>>> -----Original Message----- >>>>> From: [email protected] >>>>> [mailto:[email protected]] On Behalf Of >>> [email protected] >>>>> Sent: Wednesday, September 30, 2009 8:54 PM >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] DNS cache and expiration >>>>> >>>>> On Wed, 30 Sep 2009, Kenneth Marshall wrote: >>>>> >>>>>>>> mechanism to map IP addresses to names for the purposes >>> of logging >>>>>>>> error messages. As such, pretty much the only piece >>> that needs to >>>>>>>> be tracked within rsyslog is the TTL for the entry and the ip - >>> >>>>>>>> name mapping. A thread would be responsible for expiring >> entries >>>>>>>> from the cache (or refreshing the timeout) after validating the >>>>>>>> correctness of the mapping. I think the DNS lookups should be >>>>>>>> handled by a good resolver like pdns-recursor, djbdns,... The >>>>>>>> goal here is to allow names in the log entries and not just IP >>>>>>>> addresses and in a very high performance logging environment. >>>>>>> >>>>>>> the trouble is that doing _proper_ TTL expiration isn't as >>>>> simple as it >>>>>>> sounds. >>>>>>> >>>>>>> and if you are willing to back away from 'proper' >>>>> expiration to something >>>>>>> that will work in practice, why not go much further (as I >>>>> have detailed in >>>>>>> the other messages) >>>>>>> >>>>>>> David Lang >>>>>> >>>>>> I agree. I only mention TTL values as a reasonable >>> upperbound of the >>>>>> refresh check. The advantage is to remove the large pause >>> in logging >>>>>> due to a DNS refresh after a HUP to rsyslog, if that were the >> only >>>>>> method to flush/refresh. Like you mention, the IPs/names >>> of systems >>>>>> being logged change rarely so we should tune this for speed and >>>>>> not worry about expiration correctness. I do not agree so other >>>>>> statements that an external cache will perform as well as >>>>> an internal >>>>>> cache. Too many software products that I work with have >>>>> needed exactly >>>>>> that functionality to support very high levels of performance. >>>>> >>>>> actually, you could have the cache be configurable in three modes >>>>> >>>>> 1. no caching >>>>> >>>>> 2. blank the cache on HUP >>>>> >>>>> 3. never blank the cache (i.e. require a full restart to clear it) >>>>> >>>>> David Lang >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
