try disabling asyncWriting. I've run into problems with it when using it with
compression.
David Lang
On Mon, 5 Oct 2015, Kendall Green wrote:
Date: Mon, 5 Oct 2015 13:54:56 -0700
From: Kendall Green <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] omfile fuse
When testing, the problem appeared when receiving imptcp and queuing, then
changed the config for troubleshooting without queue or input module, and
simply tested with looping logger -plocal0.notice messages locally, and
comparing the counts.
Here is the most recent config, but will try other parameters you recommend.
local0.notice {
action(type="omfile"
name="dfs-fuse_local0-notice"
createDirs="on"
dirCreateMode="0755"
dirOwner="root"
dirGroup="dfs-grp"
fileCreateMode="0644"
fileOwner="root"
fileGroup="dfs-grp"
closeTimeout="1"
zipLevel="0"
veryRobustZip="off"
flushInterval="1"
asyncWriting="on"
flushOnTXEnd="on"
IOBufferSize="1k"
file="/dfs-fuse-data/test/dfs-appliance_test.log"
}
http://www.rsyslog.com/doc/master/configuration/modules/omfile.html
Thank you for the help!
-Kendall
On Mon, Oct 5, 2015 at 5:28 AM, Rainer Gerhards <[email protected]>
wrote:
2015-10-05 12:45 GMT+02:00 David Lang <[email protected]>:
I have seen problems with omfile and async writers (especially when
compressing), so for now I just don't do tht
could you post your config?
do you have a disk queue? that would do a random read. But the output
from
omfile should not be doing any reading.
99% (but not 100%) sure: I think we read and write queue files
sequentially.
Rainer
David Lang
On Mon, 5 Oct 2015, Kendall Green wrote:
Date: Mon, 5 Oct 2015 00:32:48 -0700
From: Kendall Green <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] omfile fuse
Hi David, thank you for the suggestions.
We've tried setting the checkpoint, flush interval, flush tx on end, and
looked for options to disable or turn off compression, but there might
be
some outdated or misinformation in the omfile docs. It's probably
unimportant or unrelated but there's at least one contradiction in the
descriptions for default parameters for if robust zip default is off or
on.
Regardless, changes to settings had no affect on messages failing to be
delivered to the fuse mount. Further testing showed, that messages would
eventually disappear from the queue if waiting long enough before
restart
or HUP. Waiting for messages to reach checkpoint of flush interval would
sometimes result in only a single message, depending on how long before
deciding to HUP/restart.
As a test with 100 messages to be parsed to the dfs fuse mount, if HUP
or
restart takes place immediately after the messages get stuck in transit,
all 100 appear on the dfs file system. Increasing this to 1000 messages
would process the entire batch and deliver when signal HUP, but restart
seems to write only about 350/1000 messages before shutdown completes,
losing the remaining logs. When sending in hundred of thousands of
messages, about 20k showed up after HUP, so the messages don't appear to
write when buffer is full. The buffer appears to overflow instead when
using dfs fuse driver that doesn't support random read write access.
"Attempting random read write" is a consistent error message which
appears
when rsyslog fails writing to the dfs-fuse mount.
Unfortunately there is no option for replacing the vendor supported dfs
fuse driver, or I'd try an alternate which support random read write.
Does it make sense for this error as indicator for what is preventing
the
parser from normal delivery of logs to dfs fuse mount?
Is there a way to set rsyslog to perform other method (guessing
sequential)
omfile output without attempting random read write access?
Are there any specific configuration settings I should try?
Any recommendations are greatly appreciated!
Regards,
Kendall
On Thu, Oct 1, 2015 at 7:50 PM, David Lang <[email protected]> wrote:
On Thu, 1 Oct 2015, Kendall Green wrote:
Rsyslog 8.13 and 8.12 have issues when writing omfile to dfs fuse
mount.
The messages are stuck in limbo until stopping or restarting rsyslog.
Changing the omfile destination from the fuse mount path to a local
disk
directory results in logs output immediately.
There are no apparent issues with the fuse mounted filesystem (other
than
rsyslog omfile delivery is stalled until service shutdown/restart).
Has anyone ran across this problem?
I'll bet that if you HUP rsyslog the contents will appear as well. It
sounds like the data is not getting flushed out to where it's visible
to
readers on these more complex filesystems, but on a local filesystem
you
are seeing the data flushed much more rapidly.
I'd also bet that if you keep writing data, eventually you will cross
some
threshold and a large chunk of data will appear.
writing data out compressed increases the window for this because the
compression algorithm really wants a full buffer before it processes
the
data and writes it out.
if you look at the omfile documentation, you will see soem flush and
checkpoint paramters that you can use to tweak this behavior. It sounds
as
if there may also be some stuff you can do with fuse to adjust the
amount
of buffering and when the data becomes visible.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T
LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
