On Sep 12, 2014, at 9:09 AM, Pranith Kumar Karampuri <[email protected]> wrote:
> > On 09/11/2014 11:38 PM, mike wrote: >> Any more to this thread? I don't mean to nag, but this seems like a pretty >> serious issue. > Did you get a chance to try this test case when write-behind translator is > disabled? > > gluster volume set <volname> performance.write-behind off I just ran the test and it seems that disabling write-behind makes the problem go away. I'll update the bug, thanks. > > Pranith >> >> How can I help? >> >> On Sep 7, 2014, at 9:51 AM, mike <[email protected]> wrote: >> >>> I don't think I have these enabled. How can I confirm that? >>> >>> On Sep 7, 2014, at 12:57 AM, Anand Avati <[email protected]> wrote: >>> >>>> The only reason O_APPEND gets stripped on the server side, is because of >>>> one of the following xlators: >>>> >>>> - stripe >>>> - quiesce >>>> - crypt >>>> >>>> If you have any of these, please try unloading/reconfiguring without these >>>> features and try again. >>>> >>>> Thanks >>>> >>>> >>>> On Sat, Sep 6, 2014 at 3:31 PM, mike <[email protected]> wrote: >>>> I was able to narrow it down to smallish python script. >>>> >>>> I've attached that to the bug. >>>> >>>> https://bugzilla.redhat.com/show_bug.cgi?id=1138970 >>>> >>>> >>>> On Sep 6, 2014, at 1:05 PM, Justin Clift <[email protected]> wrote: >>>> >>>> > Thanks Mike, this is good stuff. :) >>>> > >>>> > + Justin >>>> > >>>> > >>>> > On 06/09/2014, at 8:19 PM, mike wrote: >>>> >> I upgraded the client to Gluster 3.5.2, but there is no difference. >>>> >> >>>> >> The bug is almost certainly in the Fuse client. If I remount the >>>> >> filesystem with NFS, the problem is no longer observable. >>>> >> >>>> >> I spent a little time looking through the xlator/fuse-bridge to see >>>> >> where the offsets are coming from, but I'm really not familiar enough >>>> >> with the code, so it is slow going. >>>> >> >>>> >> Unfortunately, I'm still having trouble reproducing this in a python >>>> >> script that could be readily attached to a bug report. >>>> >> >>>> >> I'll take a crack at that again, but I will a file a bug anyway for >>>> >> completeness. >>>> >> >>>> >> On Sep 5, 2014, at 7:10 PM, mike <[email protected]> wrote: >>>> >> >>>> >>> I have narrowed down the source of the bug. >>>> >>> >>>> >>> Here is an strace of glusterfsd http://fpaste.org/131455/40996378/ >>>> >>> >>>> >>> The first line represents a write that does *not* make it into the >>>> >>> underlying file. >>>> >>> >>>> >>> The last line is the write that stomps the earlier write. >>>> >>> >>>> >>> As I said, the client file is opened in O_APPEND mode, but on the >>>> >>> glusterfsd side, the file is just O_CREAT|O_WRONLY. The means the >>>> >>> offsets to pwrite() need to be valid. >>>> >>> >>>> >>> I correlated this to a tcpdump I took and I can see that in fact, the >>>> >>> RPCs being sent have the wrong offset. Interestingly, >>>> >>> glusterfs.write-is-append = 0, which I wouldn't have expected. >>>> >>> >>>> >>> I think the bug lies in the glusterfs fuse client. >>>> >>> >>>> >>> As to your question about Gluster 3.5.2, I may be able to do that if I >>>> >>> am unable to find the bug in the source. >>>> >>> >>>> >>> -Mike >>>> >>> >>>> >>> On Sep 5, 2014, at 6:16 PM, Justin Clift <[email protected]> wrote: >>>> >>> >>>> >>>> On 06/09/2014, at 12:10 AM, mike wrote: >>>> >>>>> I have found that the O_APPEND flag is key to this failure - I had >>>> >>>>> overlooked that flag when reading the strace and trying to cobble up >>>> >>>>> a minimal reproduction. >>>> >>>>> >>>> >>>>> I now have a small pair of python scripts that can reliably >>>> >>>>> reproduce this failure. >>>> >>>> >>>> >>>> >>>> >>>> As a thought, is there a reasonable way you can test this on >>>> >>>> GlusterFS 3.5.2? >>>> >>>> >>>> >>>> There were some important bug fixes in 3.5.2 (from 3.5.1). >>>> >>>> >>>> >>>> Note I'm not saying yours is one of them, I'm just asking if it's >>>> >>>> easy to test and find out. :) >>>> >>>> >>>> >>>> Regards and best wishes, >>>> >>>> >>>> >>>> Justin Clift >>>> >>>> >>>> >>>> -- >>>> >>>> GlusterFS - http://www.gluster.org >>>> >>>> >>>> >>>> An open source, distributed file system scaling to several >>>> >>>> petabytes, and handling thousands of clients. >>>> >>>> >>>> >>>> My personal twitter: twitter.com/realjustinclift >>>> >>>> >>>> >>> >>>> >> >>>> > >>>> > -- >>>> > GlusterFS - http://www.gluster.org >>>> > >>>> > An open source, distributed file system scaling to several >>>> > petabytes, and handling thousands of clients. >>>> > >>>> > My personal twitter: twitter.com/realjustinclift >>>> > >>>> >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> [email protected] >>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users >>>> >>> >> >> >> >> _______________________________________________ >> Gluster-users mailing list >> [email protected] >> http://supercolony.gluster.org/mailman/listinfo/gluster-users > >
_______________________________________________ Gluster-users mailing list [email protected] http://supercolony.gluster.org/mailman/listinfo/gluster-users
