> -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Aaron Nichols > Sent: Friday, October 23, 2009 3:22 PM > To: rsyslog-users > Subject: Re: [rsyslog] > Enabling$ActionFileEnableSynccausedmassiveincreaseinwrite > volume(bytes/sec) to NAS -cansomeone helpshed light? > > Thank you everyone for your responses - rather than reply to each > post, here is some feedback. > > The storage system - by all measure we have a "high end" system - It's > an EMC Clarion CX3-80 attached to a Celerra NS80 NAS head with 10gb > interfaces. The network file system is NFS, the Celerra can also > present CIFS shares but I doubt those would work any better. This is > not easily replaced with a linux box so that I can do rsync. The logs > must be made available to multiple hosts and our legacy implementation > *is* a Linux box writing to direct attached disk sharing NFS... it is > crumbling under our write/read load so we had to come up with another > solution. This solution is understood to be a bit of a compromise - > the ideal situation would be direct attached fiberchannel disk and a > cluster filesystem but that wasn't an option - mostly for cost. For > all the complaints about NFS it performs just fine - if we enable file > sync the write load just goes through the roof, but the NAS keeps up. > > Client bufferring (or throttling) is still possible however because we > are transitioning to this new system many of these systems are > dual-logging to two destinations so their logging behavior can be > observed on two different systems. On the old system running syslog-ng > these messages are written to disk and visible near real-time and they > don't seem to be doing any rate limiting or otherwise.
Note that the debug run is *much* slower than the actual run. It's the usual problem, if you instrument the system you also affect it. Performance-wise, we have seen that a debug build can be up to ten times slower (sometimes more, depending on config), but even the regular debug log alone, if turned on, heavily changes performance and order of execution. The root cause is the many debug outputs, which require at least some mutex snyc and that obviously affects the overall system... > The debug log > that was taken however had file sync enable - and when that's enabled > this delay is not seen so I'm not suprised it wasn't visible. I can > produce a debug log with file sync disabled for comparison, that's > probably something I should have done in the first place. >From my POV, that actually doesn't help at all. The reason is that I don't see any unusual in the log, and that means all I can see is that data is being continously written... > > I have tried an strace on the rsyslog process(es) when running > normally and haven't been able to get much out of them but that could > be my own limited knowledge of strace. `strace -p <pid>` yields a line > of information and then just stops, even though the rsyslog process is > still working fine and is obviously doing something. I've tried > attaching to the different threads as well with the same result. ... the strace will probably provide much better info, because here we see the API calls. One thing I forgot to mention is that you should NOT use it together with the debug log. The debug output api calls will make the strace output unreadable. For example, I have these line in one of my test scripts: strace -T -tt -f -F tools/rsyslogd -c..... I don't know out of my head what each option does, but that line may give you a starting point. The end result is not a single line but rather a complete log of each API called PLUS how much time the processing took. That, too, is a quite enormous log... Rainer > > I will see if I can get a comparison debug log with file sync disabled > & if there is some guidance on getting a good strace I'm happy to > provide that output. Yesterday we had to move the server from RHEL 5.3 > to RHEL 5.1 due to some discovered limits in tcp connections - either > intentionally or unintentionally different in our installation of 5.3. > The server is now performing (with file sync disabled) at an even > higher capacity on RH5.1 however I haven't had a chance to review > whether the write latency is still observed - hopefully today I can > look at that. > > Thanks again for all the information. > > Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
