> -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Aaron Wiebe > Sent: Thursday, June 10, 2010 3:31 PM > To: rsyslog-users > Subject: Re: [rsyslog] discussion request: performance enhancement for > imtcp > > On Thu, Jun 10, 2010 at 9:17 AM, Rainer Gerhards > <[email protected]> wrote: > > > >> The only limit to the number of files currently open should be the > >> open files limit provided by the OS. > > > > ummm... think that sockets take form the same camp, so there must be > a limit. > > Also, even with a timeout, some files may be dormant for at least a > couple of > > minutes (or seconds -- user option). > > Well yes - but those pool sizes are defined through max listeners and > max sessions, so we could have anything that was left over.
agreed -- but I will look into these details once I am there ;) > > >>I am curious: how is an output > >> file identified at present? > > > > via it's name, so search is relatively expensive > > Sounds like the cache should be implemented as a hash table. :) Yes, but I need to find good hash functions. If someone already has suggestions, they are welcome (while I will not tackle this immediately, I would record that information). > > >> I don't fully understand how the async writer interacts with an > action > >> thread yet... is it a one-action-many-writers relationship? > > > > right now, it is one-on-one, as the action is guarded by the action > lock (and > > remember that on my list is removing that lock for select actions, > omfile > > being one of them!). The async writer is simply set to do async-io, > it's > > essentially double-buffering in mainframe terms (and the current > > implementation isn't even very efficient). > > So... just to clarify the separation of work, the async writer buffers > and writes, and that's it? Let's tell in terms of producer and consumer. The producer is omfile, as far as it delivers data to the buffer. The consumer than uses the buffer filled by the producer, ZIPs it if required to do so and emits the write request. Right now, there are always two buffers, so while the consumer processes one, the producer fills the other one. If the consumer is too slow the producer will block. And if you look carefully at the code, you'll see that producer and consumer barely run in parallel due to a "design issue" that I did not want (for various reasons) sort out in v4. Once I do so, performance should be even better in v5 (this has the potential to double the speed of file writes -- but not total message processing time). But this is part of the overall omfile refactoring. Rainer > > -Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
