I have two initial thoughts from experience concerning heavy use of the queue system in high traffic (6+ billion log lines a day) environments - specifically concerning DA queues.
1. The queues for the most part were very reliable 2. The performance unspooling from a "fail" state (defined as "we had to spool to disk") was sub optimal. Using fast io subsystems also seemed to have little impact on the unspool speed - it seemed stuck at around 50 to 60k messages a second in our environment, and seemed to be more CPU bound than IO bound. So, in our production we always tried to set the in memory portion of the queues very high - the disk backed portion was there in case of a disaster and we daily prayed we didn't end up having to fail over to them. We lived in fear of finding ourselves in a situation where we could not unspool fast enough to catch back up. The queue performance was a definite limiting factor for us. It was probably are biggest pain point with rsyslog. I don't say this to trivialize the complexity of the problem - and to be clear, we rarely had -reliability- issues with the queues. Brian On Wed, Oct 31, 2012 at 12:55 PM, Rainer Gerhards <[email protected]> wrote: > Hi all, > > There is the dangling issue that rsyslog has grown out of its current queue > subsystem. I am currently considering a refactoring or a complete redesign. I > initially wanted to write a large blog post with all details and ideas, but > have now opted to split this in a couple of parts - both because I have > problems to find time to do the "big one" at once; and also it probably is > smarter to get feedback asap. > > So here is the initial part: > > http://blog.gerhards.net/2012/10/rsyslog-disk-queues-refactor-or.html > > This will get anyone interested in the queue subsystem a broad understanding > of how it works - and why. Please share any concerns you have about the > current system as well as wishes/suggestions on what should improve. Deeply > technical information is fine, actually appreciated. > > I intend to let the discussion run and write the other parts of the blog > series when "events warrant it" ;) Due to other projects, I can probably not > discuss 10 hours a day, but will try to be as active as possible (which > hopefully means "much"). The intent is to come up with a solution that will > be good for the next five years to come... > > Thanks, > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
