> -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Aaron Nichols > Sent: Wednesday, October 21, 2009 7:29 AM > To: rsyslog-users > Subject: Re: [rsyslog] Enabling $ActionFileEnableSync caused > massiveincrease in write volume (bytes/sec) to NAS - can someonehelp > shed light? > > On Tue, Oct 20, 2009 at 11:12 PM, <[email protected]> wrote: > > > hmm, how large are the log entries? I'm wondering if the fact that > every > > log entry then updates the metadata (size, time modified, etc) could > > account for most, if not all, of your extra traffic. > > > > Most log entries are small (under 2k) but there is one log destination > which > produces very large log entries. We have had to increase the max > message > size to 64k to accommodate these messages. This is only one destination > but > comes from multiple sources (collated into a single file) - so with a > better > understanding of how the "$ActionFileEnableSync" option vs. using "-" > on a > particular action works, I may be able to enable sync for log > destinations > other than that one high volume, large message size destination.
Sorry, I forgot to answer that in my posting yesterday. $ActionFileEnableSync is position dependent. So I assume that it is giving right at the top of rsyslog.conf, before any action. If set to on, it *enables* the sync facility. So you can use the dash in front of the file to specify whether or not a sync should happen. If it is set to "off" (the default), the "dash-rule" is always ignored and a sync never happens. This was done because many default configs have the sync option set just out of habit and the performance toll, as you have seen, is immense. > > > > the other thing that you could be seeing is if the NFS server records > > write traffic based on # of blocks written * block size. > > > > when doing a sync after each write you will do a lot of blocks (but > > re-write the same block many times), if it counts each of these > writes as > > full block that would greatly magnify the write totals. > > > > I think you will really like the message batching in v5. you > configure it > > to output every X messages or Y ms, whichever comes first. doing a > sync > > after every write will have _far_ less effect. > > > > > I probably will enjoy v5 but am not yet ready to run it in production. > I > need to find a good balance between log update frequency and keeping > NAS > performance sane. I did pass this feedback along to our storage guys to > see > if there is any tuning we can do around this. > > I will also be gathering a debug log soon - awaiting approvals to run > the > test. I assume the debug output would show some indication of where > this > might be causing problems? My primary interest in the debug log is to see if something works other than I would expect it. If that is not the case, you need to dig down into what NFS does to cause this behavior. Doing an strace may also be a good idea, depending on the output of the debug log. The core problem from my POV is that in 4.4.1 there is no code that could create the behavior you describe (at least not to the best of my knowledge, and this is what the debug log helps verify ;)). Rainer _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
