When testing, the problem appeared when receiving imptcp and queuing, then
changed the config for troubleshooting without queue or input module, and
simply tested with looping logger -plocal0.notice messages locally, and
comparing the counts.
Here is the most recent config, but will try other parameters you recommend.
local0.notice {
action(type="omfile"
name="dfs-fuse_local0-notice"
createDirs="on"
dirCreateMode="0755"
dirOwner="root"
dirGroup="dfs-grp"
fileCreateMode="0644"
fileOwner="root"
fileGroup="dfs-grp"
closeTimeout="1"
zipLevel="0"
veryRobustZip="off"
flushInterval="1"
asyncWriting="on"
flushOnTXEnd="on"
IOBufferSize="1k"
file="/dfs-fuse-data/test/dfs-appliance_test.log"
}
http://www.rsyslog.com/doc/master/configuration/modules/omfile.html
Thank you for the help!
-Kendall
On Mon, Oct 5, 2015 at 5:28 AM, Rainer Gerhards <[email protected]>
wrote:
> 2015-10-05 12:45 GMT+02:00 David Lang <[email protected]>:
> > I have seen problems with omfile and async writers (especially when
> > compressing), so for now I just don't do tht
> >
> > could you post your config?
> >
> > do you have a disk queue? that would do a random read. But the output
> from
> > omfile should not be doing any reading.
>
> 99% (but not 100%) sure: I think we read and write queue files
> sequentially.
>
> Rainer
>
> >
> > David Lang
> >
> > On Mon, 5 Oct 2015, Kendall Green wrote:
> >
> >> Date: Mon, 5 Oct 2015 00:32:48 -0700
> >> From: Kendall Green <[email protected]>
> >> Reply-To: rsyslog-users <[email protected]>
> >> To: rsyslog-users <[email protected]>
> >> Subject: Re: [rsyslog] omfile fuse
> >>
> >>
> >> Hi David, thank you for the suggestions.
> >>
> >> We've tried setting the checkpoint, flush interval, flush tx on end, and
> >> looked for options to disable or turn off compression, but there might
> be
> >> some outdated or misinformation in the omfile docs. It's probably
> >> unimportant or unrelated but there's at least one contradiction in the
> >> descriptions for default parameters for if robust zip default is off or
> >> on.
> >>
> >> Regardless, changes to settings had no affect on messages failing to be
> >> delivered to the fuse mount. Further testing showed, that messages would
> >> eventually disappear from the queue if waiting long enough before
> restart
> >> or HUP. Waiting for messages to reach checkpoint of flush interval would
> >> sometimes result in only a single message, depending on how long before
> >> deciding to HUP/restart.
> >>
> >> As a test with 100 messages to be parsed to the dfs fuse mount, if HUP
> or
> >> restart takes place immediately after the messages get stuck in transit,
> >> all 100 appear on the dfs file system. Increasing this to 1000 messages
> >> would process the entire batch and deliver when signal HUP, but restart
> >> seems to write only about 350/1000 messages before shutdown completes,
> >> losing the remaining logs. When sending in hundred of thousands of
> >> messages, about 20k showed up after HUP, so the messages don't appear to
> >> write when buffer is full. The buffer appears to overflow instead when
> >> using dfs fuse driver that doesn't support random read write access.
> >>
> >> "Attempting random read write" is a consistent error message which
> appears
> >> when rsyslog fails writing to the dfs-fuse mount.
> >> Unfortunately there is no option for replacing the vendor supported dfs
> >> fuse driver, or I'd try an alternate which support random read write.
> >>
> >> Does it make sense for this error as indicator for what is preventing
> the
> >> parser from normal delivery of logs to dfs fuse mount?
> >> Is there a way to set rsyslog to perform other method (guessing
> >> sequential)
> >> omfile output without attempting random read write access?
> >> Are there any specific configuration settings I should try?
> >>
> >> Any recommendations are greatly appreciated!
> >>
> >>
> >> Regards,
> >> Kendall
> >>
> >>
> >>
> >> On Thu, Oct 1, 2015 at 7:50 PM, David Lang <[email protected]> wrote:
> >>
> >>> On Thu, 1 Oct 2015, Kendall Green wrote:
> >>>
> >>> Rsyslog 8.13 and 8.12 have issues when writing omfile to dfs fuse
> mount.
> >>>>
> >>>>
> >>>> The messages are stuck in limbo until stopping or restarting rsyslog.
> >>>>
> >>>> Changing the omfile destination from the fuse mount path to a local
> disk
> >>>> directory results in logs output immediately.
> >>>>
> >>>> There are no apparent issues with the fuse mounted filesystem (other
> >>>> than
> >>>> rsyslog omfile delivery is stalled until service shutdown/restart).
> >>>>
> >>>> Has anyone ran across this problem?
> >>>>
> >>>
> >>> I'll bet that if you HUP rsyslog the contents will appear as well. It
> >>> sounds like the data is not getting flushed out to where it's visible
> to
> >>> readers on these more complex filesystems, but on a local filesystem
> you
> >>> are seeing the data flushed much more rapidly.
> >>>
> >>> I'd also bet that if you keep writing data, eventually you will cross
> >>> some
> >>> threshold and a large chunk of data will appear.
> >>>
> >>> writing data out compressed increases the window for this because the
> >>> compression algorithm really wants a full buffer before it processes
> the
> >>> data and writes it out.
> >>>
> >>> if you look at the omfile documentation, you will see soem flush and
> >>> checkpoint paramters that you can use to tweak this behavior. It sounds
> >>> as
> >>> if there may also be some stuff you can do with fuse to adjust the
> amount
> >>> of buffering and when the data becomes visible.
> >>>
> >>> David Lang
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com/professional-services/
> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad
> >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> >>> DON'T LIKE THAT.
> >>>
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com/professional-services/
> >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T
> >> LIKE THAT.
> >>
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of
> > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> > LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
