> -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Aaron Wiebe > Sent: Thursday, June 10, 2010 2:17 PM > To: rsyslog-users > Subject: Re: [rsyslog] discussion request: performance enhancement for > imtcp > > On Thu, Jun 10, 2010 at 1:58 AM, Rainer Gerhards > <[email protected]> wrote: > > > > Looking at omfile implementation as an example, there *are* lots of > things > > guarded by this look. Just think about the dynafile cache, the ZIP > writer or > > the background write process. All of them use relatively simple > algorithms > > based on the assumption that the core guarantees exclusive use of the > > instance data structures. Removing that guarantee is a non-trivial > task. > > I'd like to see the dynafile cache go away, personally. That section > of code has always felt problematic, and it seems that there would be > significantly more effective ways of doing the same task. I > personally prefer the timeout method where files idle for x seconds > are closed out. I am of the opinion that the current approach is just > generally bad.
I think you misunderstand that cache. If there would be no cache, that would mean we would need to close the current dynafile and open a new one whenever the action needs to route a message to a different file. That would have enormous performance impact. Think about a dynafile action as one that potentially writes to 100 files at once. Then think about having only one file open at one time and then think about the message stream. Note that this is NOT related to the question if a file is closed after an inactivity timeout or only after cache eviction. Inactivity timeout could (and probably will) be implemented together with the dynafile cache. Anyhow, suggestions for handling dynafiles in a different way are very welcome. This is something I have under consideration. > In the case of compression, couldn't that simply sit inside the same > lock used for writing, just prior to the write() itself? There are a number of subtle issues, one being the dynafile cache (one action can write to more than one file), another one being the potential for async io (which is not really enabled in v4 but will be in v5). If we had just a single lock for (logical) writing, we could not fill up a buffer and physically write it out only if it were full. So, for ZIP, you would need to compress single messages. That means a sharp drop in compression ratio (I guess you can not achieve more than 10 to 20% on single messages if you would like to have recovery records). Rainer _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
