2015-10-05 9:32 GMT+02:00 Kendall Green <[email protected]>: > Hi David, thank you for the suggestions. > > We've tried setting the checkpoint, flush interval, flush tx on end, and > looked for options to disable or turn off compression, but there might be > some outdated or misinformation in the omfile docs. It's probably > unimportant or unrelated but there's at least one contradiction in the > descriptions for default parameters for if robust zip default is off or on. > > Regardless, changes to settings had no affect on messages failing to be > delivered to the fuse mount. Further testing showed, that messages would > eventually disappear from the queue if waiting long enough before restart > or HUP. Waiting for messages to reach checkpoint of flush interval would > sometimes result in only a single message, depending on how long before > deciding to HUP/restart. > > As a test with 100 messages to be parsed to the dfs fuse mount, if HUP or > restart takes place immediately after the messages get stuck in transit, > all 100 appear on the dfs file system. Increasing this to 1000 messages > would process the entire batch and deliver when signal HUP, but restart > seems to write only about 350/1000 messages before shutdown completes, > losing the remaining logs. When sending in hundred of thousands of > messages, about 20k showed up after HUP, so the messages don't appear to > write when buffer is full. The buffer appears to overflow instead when > using dfs fuse driver that doesn't support random read write access. > > "Attempting random read write" is a consistent error message which appears > when rsyslog fails writing to the dfs-fuse mount. > Unfortunately there is no option for replacing the vendor supported dfs > fuse driver, or I'd try an alternate which support random read write. > > Does it make sense for this error as indicator for what is preventing the > parser from normal delivery of logs to dfs fuse mount? > Is there a way to set rsyslog to perform other method (guessing sequential) > omfile output without attempting random read write access?
well, omfile does do *sequential* writes! However, it needs to *append* to the file and my uneducated guess is that the driver uses random write access to implement the open append option. Rainer > Are there any specific configuration settings I should try? > > Any recommendations are greatly appreciated! > > > Regards, > Kendall > > > > On Thu, Oct 1, 2015 at 7:50 PM, David Lang <[email protected]> wrote: > >> On Thu, 1 Oct 2015, Kendall Green wrote: >> >> Rsyslog 8.13 and 8.12 have issues when writing omfile to dfs fuse mount. >>> >>> The messages are stuck in limbo until stopping or restarting rsyslog. >>> >>> Changing the omfile destination from the fuse mount path to a local disk >>> directory results in logs output immediately. >>> >>> There are no apparent issues with the fuse mounted filesystem (other than >>> rsyslog omfile delivery is stalled until service shutdown/restart). >>> >>> Has anyone ran across this problem? >>> >> >> I'll bet that if you HUP rsyslog the contents will appear as well. It >> sounds like the data is not getting flushed out to where it's visible to >> readers on these more complex filesystems, but on a local filesystem you >> are seeing the data flushed much more rapidly. >> >> I'd also bet that if you keep writing data, eventually you will cross some >> threshold and a large chunk of data will appear. >> >> writing data out compressed increases the window for this because the >> compression algorithm really wants a full buffer before it processes the >> data and writes it out. >> >> if you look at the omfile documentation, you will see soem flush and >> checkpoint paramters that you can use to tweak this behavior. It sounds as >> if there may also be some stuff you can do with fuse to adjust the amount >> of buffering and when the data becomes visible. >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
