> -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Aaron Wiebe > Sent: Thursday, June 10, 2010 3:11 PM > To: rsyslog-users > Subject: Re: [rsyslog] discussion request: performance enhancement for > imtcp > > On Thu, Jun 10, 2010 at 8:46 AM, Rainer Gerhards > <[email protected]> wrote: > > > > I think you misunderstand that cache. If there would be no cache, > that would > > mean we would need to close the current dynafile and open a new one > whenever > > the action needs to route a message to a different file. That would > have > > enormous performance impact. Think about a dynafile action as one > that > > potentially writes to 100 files at once. Then think about having only > one > > file open at one time and then think about the message stream. > > I should rephrase - I do understand the requirement for such a caching > system, however I dislike the implementation. I would prefer a > simpler lookup method without this idea of a cache size, forced > expiration, or files open forever. There is no deterministic > expectation on what happens when a file enters the cache, it is > entirely dependent upon what takes place with _other_ files. > > I would prefer to have a simpler lookup cache with a negative result > triggering an open, and that being the only expectation of the cache.
To all of this, I simply agree ;) > The only limit to the number of files currently open should be the > open files limit provided by the OS. ummm... think that sockets take form the same camp, so there must be a limit. Also, even with a timeout, some files may be dormant for at least a couple of minutes (or seconds -- user option). >I am curious: how is an output > file identified at present? via it's name, so search is relatively expensive So I simply misunderstood your earlier comment. The current implementation is not quite well. A new one should also solve the different-actions-write-to-same-file problem. But all of that comes after the current round of optimizations (you need to start at one spot ;)). > > There are a number of subtle issues, one being the dynafile cache > (one action > > can write to more than one file), another one being the potential for > async > > io (which is not really enabled in v4 but will be in v5). If we had > just a > > single lock for (logical) writing, we could not fill up a buffer and > > physically write it out only if it were full. So, for ZIP, you would > need to > > compress single messages. That means a sharp drop in compression > ratio (I > > guess you can not achieve more than 10 to 20% on single messages if > you would > > like to have recovery records). > > I don't fully understand how the async writer interacts with an action > thread yet... is it a one-action-many-writers relationship? right now, it is one-on-one, as the action is guarded by the action lock (and remember that on my list is removing that lock for select actions, omfile being one of them!). The async writer is simply set to do async-io, it's essentially double-buffering in mainframe terms (and the current implementation isn't even very efficient). Rainer > > -Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
