On Thu, Apr 19, 2012 at 12:02 AM, Rainer Gerhards <[email protected]> wrote: >> -----Original Message----- >> From: [email protected] [mailto:rsyslog- >> [email protected]] On Behalf Of Chris McCraw >> Sent: Thursday, April 19, 2012 2:18 AM >> To: rsyslog-users >> Subject: [rsyslog] rsyslogd 5.8.5 + heavy message load + compression - >> failure mode? >> >> Hi folks, >> >> I probably missed it, but after awhile searching the docs fruitlessly, >> I decided I'd ask the experts. >> >> We have an rsyslog server that handles a couple million messages (from >> a single remote server) per minute. It logs these to several logfiles >> with a cpu load of about 40% of a single CPU core. logrotate >> currently takes about 12 hours to single-threadedly, consecutively, >> compress and rotate these logs. It's been suggested that we take >> logrotate out of the loop and just have rsyslog write compressed >> files. This seems like a great idea, but I'm curious about how it >> scales, since I don't have a good test environment (just some >> underpowered VM's which don't seem to generate comparable load no >> matter how I try) to work with. >> >> Suppose that rsyslog needs more than a core's worth of CPU to do the >> compression realtime. What happens then? Is rsyslogd multithreaded >> enough (or can it be setup to be multithreaded enough) to spin up more >> threads to handle the compressed writes? Will it ever drop messages? > > We'll I can't do the actual lab for you (well, under a support contract...). > But what I can say is that I have the strong feeling this will work for you. > I know at least of one datacenter which has a far higher data rate than you > have and they work very successfully with that feature. But YMMV: you need to > do some testing, which will identify potential bottlenecks, if there are any.
I am thrilled to do testing and can even do some in production, but I'm not sure how to be sure no messages are dropped. Any suggestions for trackable high-load-generation? I've been using logger and/or nc in a loop from the command line to log "# rather long message..." where # increases sequentially to be sure no messages were dropped, but the production log stream is just a bunch of http requests and I don't have any gauge of when one doesn't make it through, so real-world testing might not be informative. >> - we're willing to change some configuration, but here's the only >> special config we have now: >> - $MainMsgQueueType Direct > > Outch - why that? This practically disables all multi-threading and thightly > couples producer and consumer. Hmm, it was there when I arrived and I never researched it. I suspect it was put in place for debugging purposes. I'll remove it when I try this out. -- Chris McCraw | Operations New Relic - http://blog.newrelic.com - @NewRelic on Twitter _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards
