On Tue, 18 May 2010, Rainer Gerhards wrote: > Hi all, > > I am currently thinking about various optimizations for even more throughput > inside the v5 engine. During this effort, I am also doing some review of > existing literature on lock-free and wait-free algorithms. One of my former > ideas re-appeared, and I wanted to get feedback if that would be a useful > addition. Even if it is seen as valuable, I will probably not implement it > immediately, but I would try to incorporate it into the new design, which I > plan to implement later this year. > > Looking at the queue modes, we have some overhead inside queues because > queues need to do a lot of things. Things like race-limiting, blocking on > full queue, even going to disk if the queue fills up too much. Also, support > for thread pools and all that is needed. Note that all of this overhead is > also necessary if the queue is used to run an action asynchronously. > > I think I could implement a considerably faster queue, if I limit its > features. Most importantly, that means: > > - no support for going to the disk (that should not be an issue, I think) > - no support for race-limiting > - capability to accept message loss if queue is full > - message loss victim not selected based on priority > - message loss on shutdown acceptable > > In short, the queue would provide simple in-memory queueing services > including synchronization between multiple producers and consumers, but no > advanced services at all. At this price, it could probably reduced the queue > overhead very considerably. > > I think such a queue could be useful for the (common) case when data needs to > be shuffled to files, and some loss is acceptable (e.g. UDP is the input). > I'd say we could probably improve the performance for this use case by a > factor of two. > > Would this be useful? Thoughts are appreciated.
yes, this sounds useful. David Lang _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
