Hi Aaron, first and very important question: which version of rsyslog are you using?
Rainer > -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Aaron Nichols > Sent: Tuesday, October 20, 2009 5:32 PM > To: rsyslog-users > Subject: [rsyslog] Enabling $ActionFileEnableSync caused massive > increase inwrite volume (bytes/sec) to NAS - can someone help shed > light? > > Hello, > We have two rsyslog "writer" instances which are receiving syslog > messages and writing them to an NFS mount on a NAS. The NAS is > connected via > 10gb ethernet but has a number of other applications writing to it as > well. > Typical write volume for our log writers is 2-3Mbps to the NAS. > Yesterday I > attempted to resolve some complaints with logs being "bursty" - events > appear to be written in batches which sounded consistent with buffering > on > the rsyslog daemon or not syncing the file with every message. These > bursts > could come as infrequently as every 30 seconds however, which seems > excessive. I took a number of steps to remedy this but ultimately had > to > reverse the changes because write volume (measured at the disks) to the > NAS > went from 2Mbps up to 9Mbps and iops went from 30 up to 900 at the > peak. I'm > sure there is an explanation for this - but there are some subtle > configuration changes which had an impact and I don't understand > exactly > why. > > First, the main change was to set "$ActionFileEnableSync on" in each > logging > servers configuration. From what I can tell this globally enables the > ability to turn on file sync and it defaults to "off". This change, > combined > with removing the "-" from all the actions in my configuration should > have > resulted in all files being synced all the time - probably the worst > case > scenario and I'm not all that suprised this caused problems. > > Second, before rolling the change back 100% - I removed the > "$ActionFileEnableSync on" configuration from the servers but left the > action configurations without the "-" - assuming that the global > configuration option would completely disable file sync and using a "-" > would have no effect. This did not seem to be the case - it did > dramatically > decrease the volume of traffic to the NAS but still the volume was > about > double (5Mbps) the normal volume. > > Third, I added "-" before all the action lines in the configuration and > this > brought volume back down to the 2-3Mbps we are used to. > > This leaves me with a few questions: > > 1) I would expect file sync to increase transactions with the NAS but > not > increase the volume of data being written to the NAS as measured at the > disk. Can someone shed light on why this would so profoundly impact > write > volume? Load on the rsyslog servers also went down substantially after > making this change - presumably because there were fewer queued > transactions > - not sure. > > 2) Is there some difference between what "$ActionFileEnableSync on" > does vs. > adding a "-" before an action other than one is global and one is > per-action? I thought these were just two different levels of > granularity > for configuring the same thing but that doesn't appear to be the case. > > 3) Is it expected to have rsyslog take up to 30 seconds to flush > messages to > a logfile? These are BUSY logs - being written to multiple times per > second, > so it's not a delay on the client. I can observe these same clients > logging > messages to another logging server (the old server) running syslog-ng > and > messages are observable in near real-time. > > Ultimate - #3 is the problem I am trying to solve, but I'm very curious > why > these changes had the impact they did and why globally disabling file > sync > using "$ActionFileEnableSync" would not entire decrease log volume > until I > add "-" to all the actions. > > Let me know if I can add info that would help shed light on this. > > Thanks, > Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
