Hi David, thank you for the suggestions. We've tried setting the checkpoint, flush interval, flush tx on end, and looked for options to disable or turn off compression, but there might be some outdated or misinformation in the omfile docs. It's probably unimportant or unrelated but there's at least one contradiction in the descriptions for default parameters for if robust zip default is off or on.
Regardless, changes to settings had no affect on messages failing to be delivered to the fuse mount. Further testing showed, that messages would eventually disappear from the queue if waiting long enough before restart or HUP. Waiting for messages to reach checkpoint of flush interval would sometimes result in only a single message, depending on how long before deciding to HUP/restart. As a test with 100 messages to be parsed to the dfs fuse mount, if HUP or restart takes place immediately after the messages get stuck in transit, all 100 appear on the dfs file system. Increasing this to 1000 messages would process the entire batch and deliver when signal HUP, but restart seems to write only about 350/1000 messages before shutdown completes, losing the remaining logs. When sending in hundred of thousands of messages, about 20k showed up after HUP, so the messages don't appear to write when buffer is full. The buffer appears to overflow instead when using dfs fuse driver that doesn't support random read write access. "Attempting random read write" is a consistent error message which appears when rsyslog fails writing to the dfs-fuse mount. Unfortunately there is no option for replacing the vendor supported dfs fuse driver, or I'd try an alternate which support random read write. Does it make sense for this error as indicator for what is preventing the parser from normal delivery of logs to dfs fuse mount? Is there a way to set rsyslog to perform other method (guessing sequential) omfile output without attempting random read write access? Are there any specific configuration settings I should try? Any recommendations are greatly appreciated! Regards, Kendall On Thu, Oct 1, 2015 at 7:50 PM, David Lang <[email protected]> wrote: > On Thu, 1 Oct 2015, Kendall Green wrote: > > Rsyslog 8.13 and 8.12 have issues when writing omfile to dfs fuse mount. >> >> The messages are stuck in limbo until stopping or restarting rsyslog. >> >> Changing the omfile destination from the fuse mount path to a local disk >> directory results in logs output immediately. >> >> There are no apparent issues with the fuse mounted filesystem (other than >> rsyslog omfile delivery is stalled until service shutdown/restart). >> >> Has anyone ran across this problem? >> > > I'll bet that if you HUP rsyslog the contents will appear as well. It > sounds like the data is not getting flushed out to where it's visible to > readers on these more complex filesystems, but on a local filesystem you > are seeing the data flushed much more rapidly. > > I'd also bet that if you keep writing data, eventually you will cross some > threshold and a large chunk of data will appear. > > writing data out compressed increases the window for this because the > compression algorithm really wants a full buffer before it processes the > data and writes it out. > > if you look at the omfile documentation, you will see soem flush and > checkpoint paramters that you can use to tweak this behavior. It sounds as > if there may also be some stuff you can do with fuse to adjust the amount > of buffering and when the data becomes visible. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
