With some additional offline information from Benjamin the problem has been tracked down to dbpf_bstream_direct_write_op_svc(). The issue is that two write calls to different, contiguous, sections of the file occur without locking around retrieval of the current file size. The flow is similar to this, assuming two writes X -> Y, Y+1 - >Z both writes enter dbpf_bstream_direct_write_op_svc() write X->Y gets the current file size write X->Y makes the pwrite call write Y+1 -> Z gets the current file size write X->Y updates the file size write Y+1 -> Z makes the pwrite call (padding zeros from the previous end of file) write Y+1 -> Z updates the file size
I can certainly add some locking to prevent this. Mostly to Phil or Sam, was there something in place that should be preventing this before I add another wheel? I did try moving the flocks from direct_locked_write() around the get file size and update but it looks like the fd is being closed causing the locks to be tossed. Thanks, Michael On Mon, Jun 13, 2011 at 4:47 PM, Becky Ligon <[email protected]> wrote: > Benjamin: > > Thanks for the extra information. I have added it to our tracking system. > > Becky > -- > Becky Ligon > HPC Admin Staff > PVFS/OrangeFS Developer > Clemson University/Omnibond.com OrangeFS Support > 864-650-4065 > > > On Wednesday, June 08, 2011 07:33:45 Benjamin Severs wrote: > >> On Wednesday, June 08, 2011 06:21:45 AM Michael Moore wrote: > >> > Hi Benjamin, > >> > > >> > I don't have a quick fix, I don't know if others have seen this kind > >> of > >> > issue before. I've created a ticket in Trac for it: > >> > https://www.orangefs.org/trac/orangefs/ticket/34 > >> > > >> > I'm planning on just copying your setup to recreate the problem. > >> Anything > >> > special about the contents of the 10GB file or just urandom type data? > >> > > >> > Thanks for letting us know about this! > >> > > >> > Michael > >> > >> There was nothing special about the file. My file was just a repeating > >> sequence of characters (ex. 200 'A's, followed by 200 'B's, and so on). > >> > >> - Benjamin Severs > > > > More information... > > > > I believe this issue is caused by some race condition among the directio > > threads. From my testing, it appears that the corruption stops (or at > > least > > hasn't shown up in my testing) if I either switch to using the alt-aio > > trove > > method or if I reduce the number of directio threads to 1. > > > > Hope this is useful. > > > > -- > > Benjamin Severs > > _______________________________________________ > > Pvfs2-developers mailing list > > [email protected] > > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers > > > > > _______________________________________________ > Pvfs2-developers mailing list > [email protected] > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers >
_______________________________________________ Pvfs2-developers mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
