Hi Steve, Thanks for your response - what you simulate looks pretty close to what is going on here - so you are right it could be something else... network, memory, cpus, hdd..... :(
Appreciate your efforts Regards Mark On 3/3/07, Steve <[EMAIL PROTECTED]> wrote:
Hi, Yes, last night I was copying and comparing repeatedly a 300meg file on one client, the other client/samba box was pulling files off via a windows xp machine which was copying them back to another directory back on the share. Whilst my I/O box is 2 pvfs2 services in one this should simulate 2 separate servers pretty well. We have been using the pvfs2 store for several weeks, whilst the usage isnt heavy, we have on many occasions had simultaneous reads and writes, sometimes simultaneous writes. Its very low end hardware and files range from 6meg to 4gb currently with 570gb in a 1tb store. Ive seen no corruption You seem to have something strange going on. Steve -------Original Message------- From: Mark Van De Vyver Date: 03/03/2007 02:28:05 To: Steve Cc: [email protected] Subject: Re: [Pvfs2-users] PVFS 2.6.2 intermittent cmp/diff failure Hi Steve > I will try the script in the meantime I have tried to hammer it this > afternoon copying a 500meg ISO and repeatedly doing cmp, I saw no errors. > Also it came to me that when I copied the 500gb in I used a mirroring > application which would have highlighted any bad copies as files updating, I > saw none. I can also see no errors if I just have one machine copying/verifying To the PVFS2 area. Is your error free run from a case when several Machine are accessing/writing to the one PVFS2 area? Regards Mark > > > > > > -------Original Message------- > > > > From: Mark Van De Vyver > > Date: 02/03/2007 19:17:40 > > To: Steve > > Cc: [email protected] > > Subject: Re: [Pvfs2-users] PVFS 2.6.2 intermittent cmp/diff failure > > > > Hi Steve, > > I don't have access to the cluster now, but the following script has a > > Few fixes. > > I haven't yet tested copying from a non-pvfs area to pvfs with pvfs 2.6.2. > > I saw something similar in pvfs 1.5.1 when copyinf from a tmpfs area to > pvfs2. > > Running `mount` should show you if the dvd is auto-mounted and under > > What directory, in which case my mount below is redundant and you'll > > Need to replace the '/media/cdrom/' references. > > > > # untested script start > > Mkdir /media/cdrom > > # you may have to insert your systems dev name here > > Mount /dev/hdb /media/cdrom > > > > For fn in `ls /media/cdrom/*.*|sed -e 'S/\/media\/cdrom\///G` > > Do > > If [ -f "/mnt/pvfs2/${fn}" ] > > Then > > # This should 'fail' more frequently than the cmp in the else clause > > Cmp /media/cdrom/${fn} /mnt/pvfs2/${fn} > > If [ $? != 0 ] > > Then > > Echo "Prexisting copy not exact - more frequent and random?" > > If > > Else > > Cp /media/cdrom/${fn} /mnt/pvfs2/${fn} > > Cmp /media/cdrom/${fn} /mnt/pvfs2/${fn} > > If [ $? != 0 ] > > Then > > Echo " Initial copy not exact - less frequent and random" > > If > > If > > Done > > # untested script end > > > > Thanks > > Mark > > > > On 3/2/07, Steve <[EMAIL PROTECTED]> wrote: > > > Well I thought id try manual cp > > > > > > > > > > > > I never mounted a dvd under link only cdrom. I mounted a movie dvd and get > > > > an I/O error when trying to copy. I mounted a data dvd burned under > windows > > > and the mount fails as wrong filesystem. > > > > > > > > > > > > Whats your mount command syntax ? > > > > > > > > > > > > BTW do you get the same if you copy your files to local non pvfs2 disk and > > > > then use your script ? > > > > > > > > > > > > > > > > > > -------Original Message------- > > > > > > > > > > > > From: Mark Van De Vyver > > > > > > Date: 02/03/2007 09:40:30 > > > > > > To: Steve > > > > > > Cc: [email protected] > > > > > > Subject: Re: [Pvfs2-users] PVFS 2.6.2 intermittent cmp/diff failure > > > > > > > > > > > > Thanks Steve, > > > > > > I don't see any problem until I run the diff or cmp and even then > > > > > > These indicate the files are identical if the cmp is run _immediately_ > > > > > > After the file copy. > > > > > > Cmp and diff only indicate a difference when a file is 'checked' after > > > > > > Some other files have been copied-checked. > > > > > > > > > > > > The files are from the NYSE trade and quote (TAQ) DVD's, so they are > > > > > > Text stored as binary. > > > > > > > > > > > > You might be able to try the following with a dozen or so large binary > > > > > > Files, I have approx 300-400GB stored in the PVFS area. > > > > > > > > > > > > Ideally the following should be run on two or more PVFS2 servers at > > > > > > The same time, apply this to several DVD's that have not been copied > > > > > > To the PVFS area, then reapply the script to the same DVD's after they > > > > > > Have been copied. > > > > > > The following is a slightly simplified version of my script - here I > > > > > > Don't delete and re-copy when an existing file fails the cmp > > > > > > Verification: > > > > > > > > > > > > # untested script start > > > > > > For fn in `ls /dvd/*large.bin|sed -e 'S/\/dev\//G` > > > > > > Do > > > > > > If [ -f /mnt/pvfs2/${fn} ] > > > > > > Then > > > > > > # This should 'fail' more frequently than the cmp in the else clause > > > > > > Cmp ${fn} /mnt/pvfs2/${fn} > > > > > > If [ $? != 0 ] > > > > > > Then > > > > > > Echo "Prexisting copy not exact - more frequent and random?" > > > > > > If > > > > > > Else > > > > > > Cp ${fn} /mnt/pvfs2/${fn} > > > > > > Cmp ${fn} /mnt/pvfs2/${fn} > > > > > > If [ $? != 0 ] > > > > > > Then > > > > > > Echo " Initial copy not exact - less frequent and random" > > > > > > If > > > > > > Done > > > > > > # untested script end > > > > > > > > > > > > Regards > > > > > > Mark > > > > > > > > > > > > On 3/2/07, Steve <[EMAIL PROTECTED]> wrote: > > > > > > > My setup is a little different in that at the moment I have 2 I/O > services > > > > > > > > > > running on one box, a metadata on another and a client/samba server on a > > > > > > > > third. I have moved in the data via samba. We have copied in mp3's and > > > > > > > avi/mpg's as well as large ISO's plus software exe's. Surely after > several > > > > > > > > > > week of use we would notice some problem ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > I do have another box set up as a client that happens to have a dvd ROM > > > > > > > drive in it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > What type of files ? A vob ? > > > > > > > > > > > > > > What sequence of commands would I need to do you test your problem ? > > > > > > > > > > > > > > If I get a little spare time I could try for U ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > Steve > > > > > > > > > > > > > > > > > > > > > > > > > > > > -------Original Message------- > > > > > > > > > > > > > > > > > > > > > > > > > > > > From: Mark Van De Vyver > > > > > > > > > > > > > > Date: 02/03/2007 08:18:11 > > > > > > > > > > > > > > To: Steve > > > > > > > > > > > > > > Subject: Re: [Pvfs2-users] PVFS 2.6.2 intermittent cmp/diff failure > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Steve, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Not sure if this helps any but I have copied over 500gb of media files > > > > to > > > > > > > > > > > > > > > pvfs2 running on old dell's 533 to 866 CPU with very little ram > running > > > on > > > > > > > > > > > > > > > > > > > > > > caos3 beta 3. Although I havent done any checks other than using the > > > media > > > > > > > > > > > > > > > > > > > > > > I havent noticed any problems. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The failures might be spurious....? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Could you have problems with the dvd device ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > I doubt it - but it may not be impossible? > > > > > > > > > > > > > > This happens with the DVD drives on all three nodes, and when I just > > > > > > > > > > > > > > Have one node 'working the diif/cmp failures either don't occur or > > > > > > > > > > > > > > Very, very rarely. Start all three nodes 'working' and I see roughly > > > > > > > > > > > > > > 1 out of 2 binary files fail the initial diff/cmp check, but very very > > > > > > > > > > > > > > Few (one every couple of DVD's fail the cmp/diff check immediately > > > > > > > > > > > > > > After the copy is done..... > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > Mark > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -------Original Message------- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From: Mark Van De Vyver > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Date: 02/03/2007 03:26:40 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > To: [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Subject: [Pvfs2-users] PVFS 2.6.2 intermittent cmp/diff failure > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This is a follow up on an earlier email where I reported that PVFS > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1.5.1 failed copy binary files from several DVD's. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm running a 3 node Rocks 4.2.1 Cluster, CentOS4.4, x86_64, nodes are > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Connected via an unmanaged switch. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I have reinstalled the Rocks Cluster (all nodes), including the PVFS2 > > > Roll > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The cluster is set up with the frontend as the metadaat server and the > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Other two nodes are PVFS2 I/O servers and clients. The /mnt.pvfs2 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Area is on a 3 disk RAID 0 partition formatted as ext3. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > After installing I ran the test steps in the "PVFS2 Quick Start > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Guide". The test steps ran without error. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I upgraded to PVFS 2.6.2 on all nodes and re-ran the test steps, again > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > No errors or problems. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I build PVFS 2.6.2 with the following: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ./configure --with-kernel=</path/to/kernel26/> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --enable-kernel-sendfile --prefix=/usr/local/pvfs2/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Then type > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Make all > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Make kmod_install > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Make install > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On each node I have a script that lists the files on the DVD disc > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Loaded on that node. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Each file is copied if it does not exist on the HDD (PVFS area) and > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The copy is immediately verified: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cp /dvd/file1 /mnt/pvfs2/file1 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cmp /dvd/file1 /mnt/pvfs2/file1 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > `cmp` does not report any error. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This has been done for 60-70 DVD. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If I insert a DVD that has previously been copied my script finds that > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > A file exists in the PVFS area and does a `cmp` with the DVD file, if > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The file fails this comparison the file is deleted, copied, verified > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > (cmp). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I notice that frequently and randomly the previously copied files will > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Fail the _initial_ `cmp` check if more than one node is 'active', I e. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Processing a DVD. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Once deleted and copied the second `cmp` check is passed. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Some details: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The files do not fail the `cmp` check immediately after being copied - > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Only when checking a previously copied file. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The `cmp` result indicates a different byte at which the files differ. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Re-inserting the same dvd several times results if different files > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Failing the first `cmp` check. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The second check (immediately after the copy is finished) is always > > > passed > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This occurs rarely, if at all (I.e. I haven't noticed it), when only > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > One node is processing a DVD. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This only occurs with binary files - which are relatively large 200MB > - > > > 2 > > > > > > > GB > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This never occurs with text files - which are also small 100'sKB > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The pvfs2-client.log file is empty on each node. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I have tried using diff and experience the same results. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This is similar to an error I was seeing in PVFS 1.5.1 - hence the > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Upgrade. I've also changed my previous script which `dd` copied the > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > DVD to memory (approx 8GB), then wrote this ISO file to the PVFS2 area > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - this worked fine for initial copies, but failed for re-copies. At > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > That time I wasn't verifiying the copy, so it was the copy to the > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > PVFS2 area that failed..... > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Finally, on one occasion when manually running `cmp` on a file I > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Noticed the following sequence. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cmp file1 file2 (pass) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cmp file1 file2 (pass) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Difffile1 file2 (fail) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cmp file1 file2 (fail) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Is this known behavior with a known workaround/configuration setting? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The behavior I see made me guess a caching or network issue (there are > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > No other machines on the cluster network). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Can anyone suggest PVFS configuration settings that will make PVFS > more > > > > > > > > > > > > > > > robust. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm not a programmer or linux guru - I just spent this summer > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Converting from winxp... > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm happy to explore some possible fixes, but don't assume too much :) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks in advance > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Mark > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Pvfs2-users mailing list > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
_______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
