Re: [Pvfs2-users] PVFS 2.6.2 intermittent cmp/diff failure

[Sam Lang]

Hi Mark,
Comments inline.

-sam

On Mar 3, 2007, at 5:23 PM, Mark Van De Vyver wrote:

----snip----

It seemed the pvfs2-client start-up wasn't going to time out so I hard
reset - may have been impatient.
I take your point about the reboots.

The timeouts are set quite high (5 minutes I think).  You can adjust  
both the timeout value and number of retries in the fs.conf.  Your  
best bet is to make sure the servers are started before trying to mount.


---snip---

The primary intended use for the pvfs2 area was to to hold MySQL
tables - but it starts to look like this may be too bleeding edge for
my level of skill & experience?
The nodes would run condor jobs that access & store data in the  
MySQL db .
Using pvfs would mean I don't have to split the db's over 3 MySQL
servers - one on each node - but there is some performance cost - all
the data has to pass through the frontend.
So, I'm using PVFS just to get access to the combined space from the
three nodes - approx 3Tb on a tight budget.

Hmm...it sounds like you really want an abstraction layer with a  
global namespace to the data distributed over your three nodes.  PVFS  
can be good for that.  One thing to remember though, PVFS by default  
stripes file data over all the IO nodes in the filesystem, which is  
really only useful with more of an asymmetric setup, and a more  
complex client/server network.  You can avoid this, so that PVFS  
stores an entire file on one node, which might improve your  
performance.  This can be done by setting an extended attribute on  
the root of the fs.

setfattr -n "pvfs2.num_dfiles" -v "1" /mnt/pvfs2/


Not having PVFS is not a show stopper, but seemed to offer the  
simplest set-up.
I would appreciate any suggested improvements/advice - including 'we
don't advise that' :)

As Murali mentioned in a previous email, PVFS doesn't provide file  
locks, but if you're accessing your tables from just the frontend  
node, that shouldn't be a problem.  That being said, while a MySQL  
server might run successfully on a PVFS mount, the performance of  
such a setup is likely going to be worse than a distributed MySQL  
setup.  PVFS will stripe the tables over the IO servers in what  
undoubtedly ends up being a non-optimal fashion.


Could you send us the fs.conf file you're using on each of the nodes?

I've attached the config files - these were in /opt/pvfs2/etc my
understanding is they are what Rocks distributes to the nodes as part
of the PVFS2 Roll configuration step.

First off, the PVFS2 roll configuration is a bit wonky.  They appear  
to call metadata servers 'frontend' nodes, and IO servers 'compute'  
nodes.  This means that you're only using the frontend node disk for  
metadata (which really doesn't come close to its ~ 1TB capacity), and  
your two compute nodes hold all the actual file data.  It sounds like  
you really want to distribute all the data amongst the three nodes in  
your system (frontend, compute0, compute1), and use one of them as  
the metadata server as well.  You can do this by modifying the  
fs.conf so that 'frontend' appears in the <DataHandleRanges> context  
with a unique range for its IO handles (something outside the ranges  
already chosen by compute0 and compute1).

As for the diff of files failing, it seems odd that its happening  
_after_ the diff has already succeeded.

Is it possible that when the dvd is mounted again, the filenames or  
their content have somehow changed between the last time it was  
mounted?  Or with the way you grab the dvd labels off the disk using  
dd, they somehow end up being the same, and the filenames, are the  
same on different dvds, but their contents differ?

It would be helpful to know how the diff fails.  Would it be possible  
to abort the script at the first sign of a diff returning non-zero,  
and look at the differences between the files?  How much of the file  
on the PVFS volume is corrupted?  Is it all at the end, etc.?


>
> I have 3 terminals open showing:
> tail -f /var/log/messages
> tail -f /var/log/dmsg
> tail -f /var/log/syslog
> I see no output to these files throughout this exercise.
>
> The pvfs2-client.log files from each machine are attached.
>
> It seems this problem occurs when PVFS2 is under load?

Possibly.  When you get to the end of those steps, are you still able
to access files in the file system?

Yes I can.  I can open a text file copied from the DVD-A, but then it
seems text or small files never fail the cmp/diff verification.

Thanks again for the time and effort put into these exchanges and the
development of PVFS.

Regards
Mark

Thanks,

-sam

>
> Hope this helps.
> Mark
>
>> Thanks,
>>
>> Rob
>>
>> Mark Van De Vyver wrote:
>> > Thanks Steve,
>> > I don't see any problem until I run the diff or cmp and even  
then
>> > these indicate the files are identical if the cmp is run
>> _immediately_
>> > after the file copy.
>> > cmp and diff only indicate a difference when a file is 'checked'
>> after
>> > some other files have been copied-checked.
>> >
>> > The files are from the NYSE trade and quote (TAQ) DVD's, so they
>> are
>> > text stored as binary.
>> >
>> > You might be able to try the following with a dozen or so large
>> binary
>> > files, I have approx 300-400GB stored in the PVFS area.
>> >
>> > Ideally the following should be run on two or more PVFS2  
servers at
>> > the same time, apply this to several DVD's that have not been
>> copied
>> > to the PVFS area, then reapply the script to the same DVD's
>> after they
>> > have been copied.
>> > The following is a slightly simplified version of my script -
>> here I
>> > don't delete and re-copy when an existing file fails the cmp
>> > verification:
>> >
>> > # untested script start
>> > for fn in `ls /dvd/*large.bin|sed -e 's/\/dev\//g'`
>> >  do
>> >      if [ -f /mnt/pvfs2/${fn} ]
>> >        then
>> >          # This should 'fail' more frequently than the cmp in
>> the else
>> > clause
>> >          cmp ${fn} /mnt/pvfs2/${fn}
>> >          if [ $? != 0 ]
>> >            then
>> >              echo "Prexisting copy not exact - more frequent and
>> random?"
>> >          fi
>> >        else
>> >           cp ${fn} /mnt/pvfs2/${fn}
>> >           cmp ${fn} /mnt/pvfs2/${fn}
>> >           if [ $? != 0 ]
>> >              then
>> >                echo "    Initial copy not exact - less frequent
>> and random"
>> >        fi
>> >  done
>> > # untested script end
>> >
>> > Regards
>> > Mark
>> >
>> > On 3/2/07, Steve <[EMAIL PROTECTED]> wrote:
>> >> My setup is a little different in that at the moment I have  
2 I/O
>> >> services
>> >> running on one box, a metadata on another and a client/samba
>> server on a
>> >> third. I have moved in the data via samba. We have copied in
>> mp3's and
>> >> avi/mpg's as well as large ISO's plus software exe's. Surely  
after
>> >> several
>> >> week of use we would notice some problem ?
>> >>
>> >>
>> >>
>> >> I do have another box set up as a client that happens to have a
>> dvd ROM
>> >> drive in it.
>> >>
>> >>
>> >>
>> >> What type of files ? A vob ?
>> >>
>> >> What sequence of  commands would I need to do you test your
>> problem ?
>> >>
>> >> If I get a little spare time I could try for U ?
>> >>
>> >>
>> >>
>> >> Steve
>> >>
>> >>
>> >>
>> >> -------Original Message-------
>> >>
>> >>
>> >>
>> >> From: Mark Van De Vyver
>> >>
>> >> Date: 02/03/2007 08:18:11
>> >>
>> >> To: Steve
>> >>
>> >> Subject: Re: [Pvfs2-users] PVFS 2.6.2 intermittent cmp/diff
>> failure
>> >>
>> >>
>> >>
>> >> Hi Steve,
>> >>
>> >>
>> >>
>> >> > Not sure if this helps any but I have copied over 500gb of  
media
>> >> files to
>> >>
>> >> > pvfs2 running on old dell's 533 to 866 CPU with very  
little ram
>> >> running on
>> >>
>> >>
>> >> > caos3 beta 3. Although I havent done any checks other than
>> using the
>> >> media
>> >>
>> >>
>> >> > I havent noticed any problems.
>> >>
>> >> >
>> >>
>> >>
>> >>
>> >> The failures might be spurious....?
>> >>
>> >>
>> >>
>> >> > Could you have problems with the dvd device ?
>> >>
>> >>
>> >>
>> >> I doubt it - but it may not be impossible?
>> >>
>> >> This happens with the DVD drives on all three nodes, and when I
>> just
>> >>
>> >> Have one node 'working the diif/cmp failures either don't  
occur or
>> >>
>> >> Very, very rarely. Start all three nodes 'working' and I see
>> roughly
>> >>
>> >> 1 out of 2 binary files fail the initial diff/cmp check, but
>> very very
>> >>
>> >> Few (one every couple of DVD's fail the cmp/diff check  
immediately
>> >>
>> >> After the copy is done.....
>> >>
>> >>
>> >>
>> >> Thanks
>> >>
>> >> Mark
>> >>
>> >>
>> >>
>> >> >
>> >>
>> >> > -------Original Message-------
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > From: Mark Van De Vyver
>> >>
>> >> >
>> >>
>> >> > Date: 02/03/2007 03:26:40
>> >>
>> >> >
>> >>
>> >> > To: pvfs2-users@beowulf-underground.org
>> >>
>> >> >
>> >>
>> >> > Subject: [Pvfs2-users] PVFS 2.6.2 intermittent cmp/diff  
failure
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > Hi,
>> >>
>> >> >
>> >>
>> >> > This is a follow up on an earlier email where I reported that
>> PVFS
>> >>
>> >> >
>> >>
>> >> > 1.5.1 failed copy binary files from several DVD's.
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > I'm running a 3 node Rocks 4.2.1 Cluster, CentOS4.4, x86_64,
>> nodes are
>> >>
>> >> >
>> >>
>> >> > Connected via an unmanaged switch.
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > I have reinstalled the Rocks Cluster (all nodes),  
including the
>> >> PVFS2 Roll
>> >>
>> >>
>> >> >
>> >>
>> >> > The cluster is set up with the frontend as the metadaat
>> server and the
>> >>
>> >> >
>> >>
>> >> > Other two nodes are PVFS2 I/O servers and clients. The /
>> mnt.pvfs2
>> >>
>> >> >
>> >>
>> >> > Area is on a 3 disk RAID 0 partition formatted as ext3.
>> >>
>> >> >
>> >>
>> >> > After installing I ran the test steps in the "PVFS2 Quick  
Start
>> >>
>> >> >
>> >>
>> >> > Guide". The test steps ran without error.
>> >>
>> >> >
>> >>
>> >> > I upgraded to PVFS 2.6.2 on all nodes and re-ran the test
>> steps, again
>> >>
>> >> >
>> >>
>> >> > No errors or problems.
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > I build PVFS 2.6.2 with the following:
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > ./configure --with-kernel=</path/to/kernel26/>
>> >>
>> >> >
>> >>
>> >> > --enable-kernel-sendfile --prefix=/usr/local/pvfs2/
>> >>
>> >> >
>> >>
>> >> > Then type
>> >>
>> >> >
>> >>
>> >> > Make all
>> >>
>> >> >
>> >>
>> >> > Make kmod_install
>> >>
>> >> >
>> >>
>> >> > Make install
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > On each node I have a script that lists the files on the DVD
>> disc
>> >>
>> >> >
>> >>
>> >> > Loaded on that node.
>> >>
>> >> >
>> >>
>> >> > Each file is copied if it does not exist on the HDD (PVFS
>> area) and
>> >>
>> >> >
>> >>
>> >> > The copy is immediately verified:
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > Cp /dvd/file1 /mnt/pvfs2/file1
>> >>
>> >> >
>> >>
>> >> > Cmp /dvd/file1 /mnt/pvfs2/file1
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > `cmp` does not report any error.
>> >>
>> >> >
>> >>
>> >> > This has been done for 60-70 DVD.
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > If I insert a DVD that has previously been copied my script
>> finds that
>> >>
>> >> >
>> >>
>> >> > A file exists in the PVFS area and does a `cmp` with the DVD
>> file, if
>> >>
>> >> >
>> >>
>> >> > The file fails this comparison the file is deleted, copied,
>> verified
>> >>
>> >> >
>> >>
>> >> > (cmp).
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > I notice that frequently and randomly the previously copied
>> files will
>> >>
>> >> >
>> >>
>> >> > Fail the _initial_ `cmp` check if more than one node is
>> 'active', I.e.
>> >>
>> >> >
>> >>
>> >> > Processing a DVD.
>> >>
>> >> >
>> >>
>> >> > Once deleted and copied the second `cmp` check is passed.
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > Some details:
>> >>
>> >> >
>> >>
>> >> > The files do not fail the `cmp` check immediately after being
>> copied -
>> >>
>> >> >
>> >>
>> >> > Only when checking a previously copied file.
>> >>
>> >> >
>> >>
>> >> > The `cmp` result indicates a different byte at which the
>> files differ.
>> >>
>> >> >
>> >>
>> >> > Re-inserting the same dvd several times results if different
>> files
>> >>
>> >> >
>> >>
>> >> > Failing the first `cmp` check.
>> >>
>> >> >
>> >>
>> >> > The second check (immediately after the copy is finished) is
>> always
>> >> passed
>> >>
>> >>
>> >> >
>> >>
>> >> > This occurs rarely, if at all (I.e. I haven't noticed it),
>> when only
>> >>
>> >> >
>> >>
>> >> > One node is processing a DVD.
>> >>
>> >> >
>> >>
>> >> > This only occurs with binary files - which are relatively  
large
>> >> 200MB - 2
>> >> GB
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > This never occurs with text files - which are also small  
100'sKB
>> >>
>> >> >
>> >>
>> >> > The pvfs2-client.log file is empty on each node.
>> >>
>> >> >
>> >>
>> >> > I have tried using diff and experience the same results.
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > This is similar to an error I was seeing in PVFS 1.5.1 -
>> hence the
>> >>
>> >> >
>> >>
>> >> > Upgrade. I've also changed my previous script which `dd`
>> copied the
>> >>
>> >> >
>> >>
>> >> > DVD to memory (approx 8GB), then wrote this ISO file to the
>> PVFS2 area
>> >>
>> >> >
>> >>
>> >> > - this worked fine for initial copies, but failed for re-
>> copies. At
>> >>
>> >> >
>> >>
>> >> > That time I wasn't verifiying the copy, so it was the copy to
>> the
>> >>
>> >> >
>> >>
>> >> > PVFS2 area that failed.....
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > Finally, on one occasion when manually running `cmp` on a  
file I
>> >>
>> >> >
>> >>
>> >> > Noticed the following sequence.
>> >>
>> >> >
>> >>
>> >> > Cmp file1 file2 (pass)
>> >>
>> >> >
>> >>
>> >> > Cmp file1 file2 (pass)
>> >>
>> >> >
>> >>
>> >> > Difffile1 file2 (fail)
>> >>
>> >> >
>> >>
>> >> > Cmp file1 file2 (fail)
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > Is this known behavior with a known workaround/configuration
>> setting?
>> >>
>> >> >
>> >>
>> >> > The behavior I see made me guess a caching or network issue
>> (there are
>> >>
>> >> >
>> >>
>> >> > No other machines on the cluster network).
>> >>
>> >> >
>> >>
>> >> > Can anyone suggest PVFS configuration settings that will make
>> PVFS more
>> >>
>> >> > robust.
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > I'm not a programmer or linux guru - I just spent this summer
>> >>
>> >> >
>> >>
>> >> > Converting from winxp...
>> >>
>> >> >
>> >>
>> >> > I'm happy to explore some possible fixes, but don't assume
>> too much :)
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > Thanks in advance
>> >>
>> >> >
>> >>
>> >> > Mark
>> >>
>> >> >
>> >>
>> >> > _______________________________________________
>> >>
>> >> >
>> >>
>> >> > Pvfs2-users mailing list
>> >>
>> >> >
>> >>
>> >> > Pvfs2-users@beowulf-underground.org
>> >>
>> >> >
>> >>
>> >> > http://www.beowulf-underground.org/mailman/listinfo/pvfs2- 
users
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >>
>> >>
>> > _______________________________________________
>> > Pvfs2-users mailing list
>> > Pvfs2-users@beowulf-underground.org
>> > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>> >
>>
>> <copy-taq-dvd-monitor.sh>
>> <copy-taq-dvd.sh>
>> <pvfs2-client.log.frontend>
>> <pvfs2-client.log.compute-0-0>
>> <pvfs2-client.log.compute-0-1>
> _______________________________________________
> Pvfs2-users mailing list
> Pvfs2-users@beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users

<pvfs2-fs.conf>
<pvfs2-server.conf-frontend>
<pvfs2-server.conf-pvfs2-compute-0-0>
<pvfs2-server.conf-pvfs2-compute-0-1>

_______________________________________________
Pvfs2-users mailing list
Pvfs2-users@beowulf-underground.org
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users