I've set asyncWriting="off" and the only difference is, that with 1000 message test, all were delivered on HUP and/or service restart, but apparently not on flushInterval or buffer full. I'm currently running another test with 100k messages iterating about 10 logs/sec, and watching the output for number of lines. The output file is created on dfs-fuse mount upon receipt of the first log, but 0 lines written.
This current test will hopefully show how many logs get processed without intervention over the next ~3 hours, and if still no logs appear will see how many are written upon HUP signal. I'm unsure how the asyncWriting off impacts the rest of the config, and if the current config has disabled compression... or not with no zip. This note is from omfile asyncWriting doc: *Note that in order to enable FlushInterval, AsyncWriting must be set to “on”. Otherwise, the flush interval will be ignored. Also note that when FlushOnTXEnd is “on” but AsyncWriting is off, output will only be written when the buffer is full. This may take several hours, or even require a rsyslog shutdown. However, a buffer flush can be forced in that case by sending rsyslogd a HUP signal.* Please help clarify how these parameters correspond and what the settings can achieve expedited log transport, avoiding long delays waiting for buffering or flush intervals. This goal is why have chosen low settings for IOBufferSize and flushInterval. Also, the current config test appears to have no change on delivering any of the ~18k logs so far, but will let this run 100k and see. Then will repeat fresh using a single message to track with debugging enabled. Unfortunately I'm not able to provide debug directly from the client but can look for something specific, then will transpose or summarize in my next reply for deeper troubleshooting of the issue. It is also interesting that system messages show random read write errors appear for each log sent to rsyslog for dfs-fuse destination. Logs are received, rsyslog attempt to random read-write dfs-fuse, log get stuck, and eventually logs written when services are restarted. Thanks again, Kendall On Mon, Oct 5, 2015 at 1:58 PM, David Lang <[email protected]> wrote: > try disabling asyncWriting. I've run into problems with it when using it > with compression. > > David Lang > > On Mon, 5 Oct 2015, Kendall Green wrote: > > Date: Mon, 5 Oct 2015 13:54:56 -0700 >> >> From: Kendall Green <[email protected]> >> Reply-To: rsyslog-users <[email protected]> >> To: rsyslog-users <[email protected]> >> Subject: Re: [rsyslog] omfile fuse >> >> When testing, the problem appeared when receiving imptcp and queuing, then >> changed the config for troubleshooting without queue or input module, and >> simply tested with looping logger -plocal0.notice messages locally, and >> comparing the counts. >> >> Here is the most recent config, but will try other parameters you >> recommend. >> >> local0.notice { >> action(type="omfile" >> name="dfs-fuse_local0-notice" >> createDirs="on" >> dirCreateMode="0755" >> dirOwner="root" >> dirGroup="dfs-grp" >> fileCreateMode="0644" >> fileOwner="root" >> fileGroup="dfs-grp" >> closeTimeout="1" >> zipLevel="0" >> veryRobustZip="off" >> flushInterval="1" >> asyncWriting="on" >> flushOnTXEnd="on" >> IOBufferSize="1k" >> file="/dfs-fuse-data/test/dfs-appliance_test.log" >> } >> >> http://www.rsyslog.com/doc/master/configuration/modules/omfile.html >> >> Thank you for the help! >> >> >> -Kendall >> >> On Mon, Oct 5, 2015 at 5:28 AM, Rainer Gerhards <[email protected] >> > >> wrote: >> >> 2015-10-05 12:45 GMT+02:00 David Lang <[email protected]>: >>> >>>> I have seen problems with omfile and async writers (especially when >>>> compressing), so for now I just don't do tht >>>> >>>> could you post your config? >>>> >>>> do you have a disk queue? that would do a random read. But the output >>>> >>> from >>> >>>> omfile should not be doing any reading. >>>> >>> >>> 99% (but not 100%) sure: I think we read and write queue files >>> sequentially. >>> >>> Rainer >>> >>> >>>> David Lang >>>> >>>> On Mon, 5 Oct 2015, Kendall Green wrote: >>>> >>>> Date: Mon, 5 Oct 2015 00:32:48 -0700 >>>>> From: Kendall Green <[email protected]> >>>>> Reply-To: rsyslog-users <[email protected]> >>>>> To: rsyslog-users <[email protected]> >>>>> Subject: Re: [rsyslog] omfile fuse >>>>> >>>>> >>>>> Hi David, thank you for the suggestions. >>>>> >>>>> We've tried setting the checkpoint, flush interval, flush tx on end, >>>>> and >>>>> looked for options to disable or turn off compression, but there might >>>>> >>>> be >>> >>>> some outdated or misinformation in the omfile docs. It's probably >>>>> unimportant or unrelated but there's at least one contradiction in the >>>>> descriptions for default parameters for if robust zip default is off or >>>>> on. >>>>> >>>>> Regardless, changes to settings had no affect on messages failing to be >>>>> delivered to the fuse mount. Further testing showed, that messages >>>>> would >>>>> eventually disappear from the queue if waiting long enough before >>>>> >>>> restart >>> >>>> or HUP. Waiting for messages to reach checkpoint of flush interval would >>>>> sometimes result in only a single message, depending on how long before >>>>> deciding to HUP/restart. >>>>> >>>>> As a test with 100 messages to be parsed to the dfs fuse mount, if HUP >>>>> >>>> or >>> >>>> restart takes place immediately after the messages get stuck in transit, >>>>> all 100 appear on the dfs file system. Increasing this to 1000 messages >>>>> would process the entire batch and deliver when signal HUP, but restart >>>>> seems to write only about 350/1000 messages before shutdown completes, >>>>> losing the remaining logs. When sending in hundred of thousands of >>>>> messages, about 20k showed up after HUP, so the messages don't appear >>>>> to >>>>> write when buffer is full. The buffer appears to overflow instead when >>>>> using dfs fuse driver that doesn't support random read write access. >>>>> >>>>> "Attempting random read write" is a consistent error message which >>>>> >>>> appears >>> >>>> when rsyslog fails writing to the dfs-fuse mount. >>>>> Unfortunately there is no option for replacing the vendor supported dfs >>>>> fuse driver, or I'd try an alternate which support random read write. >>>>> >>>>> Does it make sense for this error as indicator for what is preventing >>>>> >>>> the >>> >>>> parser from normal delivery of logs to dfs fuse mount? >>>>> Is there a way to set rsyslog to perform other method (guessing >>>>> sequential) >>>>> omfile output without attempting random read write access? >>>>> Are there any specific configuration settings I should try? >>>>> >>>>> Any recommendations are greatly appreciated! >>>>> >>>>> >>>>> Regards, >>>>> Kendall >>>>> >>>>> >>>>> >>>>> On Thu, Oct 1, 2015 at 7:50 PM, David Lang <[email protected]> wrote: >>>>> >>>>> On Thu, 1 Oct 2015, Kendall Green wrote: >>>>>> >>>>>> Rsyslog 8.13 and 8.12 have issues when writing omfile to dfs fuse >>>>>> >>>>> mount. >>> >>>> >>>>>>> >>>>>>> The messages are stuck in limbo until stopping or restarting rsyslog. >>>>>>> >>>>>>> Changing the omfile destination from the fuse mount path to a local >>>>>>> >>>>>> disk >>> >>>> directory results in logs output immediately. >>>>>>> >>>>>>> There are no apparent issues with the fuse mounted filesystem (other >>>>>>> than >>>>>>> rsyslog omfile delivery is stalled until service shutdown/restart). >>>>>>> >>>>>>> Has anyone ran across this problem? >>>>>>> >>>>>>> >>>>>> I'll bet that if you HUP rsyslog the contents will appear as well. It >>>>>> sounds like the data is not getting flushed out to where it's visible >>>>>> >>>>> to >>> >>>> readers on these more complex filesystems, but on a local filesystem >>>>>> >>>>> you >>> >>>> are seeing the data flushed much more rapidly. >>>>>> >>>>>> I'd also bet that if you keep writing data, eventually you will cross >>>>>> some >>>>>> threshold and a large chunk of data will appear. >>>>>> >>>>>> writing data out compressed increases the window for this because the >>>>>> compression algorithm really wants a full buffer before it processes >>>>>> >>>>> the >>> >>>> data and writes it out. >>>>>> >>>>>> if you look at the omfile documentation, you will see soem flush and >>>>>> checkpoint paramters that you can use to tweak this behavior. It >>>>>> sounds >>>>>> as >>>>>> if there may also be some stuff you can do with fuse to adjust the >>>>>> >>>>> amount >>> >>>> of buffering and when the data becomes visible. >>>>>> >>>>>> David Lang >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com/professional-services/ >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>> >>>>> myriad >>> >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>>> DON'T LIKE THAT. >>>>>> >>>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com/professional-services/ >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>> myriad >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>> >>>> DON'T >>> >>>> LIKE THAT. >>>>> >>>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>> >>> of >>> >>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>> DON'T >>>> LIKE THAT. >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >>> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >> _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
