Hi all, I am currently thinking about various optimizations for even more throughput inside the v5 engine. During this effort, I am also doing some review of existing literature on lock-free and wait-free algorithms. One of my former ideas re-appeared, and I wanted to get feedback if that would be a useful addition. Even if it is seen as valuable, I will probably not implement it immediately, but I would try to incorporate it into the new design, which I plan to implement later this year.
Looking at the queue modes, we have some overhead inside queues because queues need to do a lot of things. Things like race-limiting, blocking on full queue, even going to disk if the queue fills up too much. Also, support for thread pools and all that is needed. Note that all of this overhead is also necessary if the queue is used to run an action asynchronously. I think I could implement a considerably faster queue, if I limit its features. Most importantly, that means: - no support for going to the disk (that should not be an issue, I think) - no support for race-limiting - capability to accept message loss if queue is full - message loss victim not selected based on priority - message loss on shutdown acceptable In short, the queue would provide simple in-memory queueing services including synchronization between multiple producers and consumers, but no advanced services at all. At this price, it could probably reduced the queue overhead very considerably. I think such a queue could be useful for the (common) case when data needs to be shuffled to files, and some loss is acceptable (e.g. UDP is the input). I'd say we could probably improve the performance for this use case by a factor of two. Would this be useful? Thoughts are appreciated. Rainer _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
