Thanks for the detailed response Sam. My replies are below.
These messages in your client logs are the same messages I commented on back on February 15th. Its possible that the messages in these logs are just stale, but the timestamps in these logs differ from the ones you sent on Feb. 15th, so I'm wary that your network issues haven't been resolved. When I see these logs, they suggest that your network is not configured properly, and that you're not actually using PVFS in the way you intend. To be sure that errors you see in the client logs are recent, you can pass the --logstamp=datetime argument to pvfs2-client. That will show full dates, instead of just times.
Thanks for the -logstamp tip. I've re-installed the cluster since the last set of logs I sent - just in case there was a network issue. In the current log file you see all messages related to pvfs2 over the period since I installed the machine. The error messages you see are either from when there was a network cable unplugged or from when accessing a PVFS area when all the nodes hadn't fully started up. The [10:16:20] timestamp is a day later than the [10:32:57]. I pass the test steps set out in the Quickstart Guide, but that may not prove a correct setup.
I've included some further comments inline. On Mar 2, 2007, at 8:14 PM, Mark Van De Vyver wrote: > Hi Rob, > Thanks for taking time to look at this. > > I'm still wondering if the warnings I saw when compiling PVFS 2.6.2 > are related (see my earlier post on upgrading from 1.5.1 to 2.6.2) I'll try to reply to that email as well. Essentially though you can ignore those warnings. > >> Are you seeing any block device errors in your dmesg/syslog output? >> Given the fsck that you had to perform previously, it is possible >> that >> there's a problem with the local disk or FS. > > I'm not sure hat lay behind those disk errors - I've changed a power > supply and haven't seen any more. I've also completely rebuilt the > cluster since then - so that PVFS 2.6.2 was upgraded froma on a fresh > install of 1.5.1. > > No errors in dmesg, messages or syslog - I now see some in the pvfs2- > client.log (attached, see below for a description of what I do on the > machines during this time) > >> What local FS are you using, by the way? > > ext3 on a raid 0, all nodes are partitioned identically. > > To elaborate/clarify my earlier post: > > I take 3 DVD that have previously been copied to the PVFS area: DVD A, > B, C, thr files on these all passed the diff verification after they > were copied. > > I boot the cluster - the two PVFS2 I/O compute nodes start fine, > the frontend hangs when loading the PVFS2-client (server loads [ OK > ]). This correlates to the client log on the frontend. It sounds like you're booting the PVFS IO nodes and frontend client node simultaneously? The PVFS servers have to be done initializing before you can attempt to mount the volume on the frontend, otherwise the mount will fail (which is what you're seeing). If I'm reading the frontend node logs right, it looks like the mount attempt occurs before your network is even setup, giving you those 'No route to host' errors. > After a hard reset, then checking the filesystem the frontend > boots fine. Out of curiousity, do you perform this booting/hard reset step for a particular reason, or is that just what windows has trained you to do? ;-) PVFS is implemented to allow you to leave these machines running, and perform the necessary updates and configuration changes without rebooting each time you make a change.
It seemed the pvfs2-client start-up wasn't going to time out so I hard reset - may have been impatient. I take your point about the reboots.
> > Load DVD-A on to the frontend and run the script > copy-taq-dvd-monitor.sh, which calls copy-taq-dvd.sh (both attached) > Two binary files fail the verification check, they are deleted, > coppied and verified - both pass. > Now all the files on DVD-A have been verified once. > > Reinsert DVD-A on the frontend and rerun the script - all files pass > the verification - no files need to be deleted/recopied/verified. > Now all the files on the PVFS2 area have been _twice_ verified as the > same as the DVD. > > Insert DVD-B and DVD-C on the two nodes and DVD-A on the frontend, run > the same script on all three machines. > Now 3 files on DVD-A fail the initial verification. One of these is > the same that failed the first 'run'. > DVD-B and C have 3-4 binary files that fail. > All failed files are deleted/copied/verified and pass the > verification. > > Stop the script running on the two compute nodes (DVD-B, C), restart > the script on the frontend to re-process DVD-A. > The diff fails reporting a 'broken pipe', attempts to restart the > script result in the `cp` operation for this file hanging. > At this point I stopped. Based on these steps, it sounds like your setup consists of trying to use PVFS in a symmetric mode, where you run three servers as part of your file system, and mount that file system from those same three nodes with servers running on them. This of course can be done, but its an uncommon setup for PVFS, and you may benefit from an alternative setup.
Your setup description is correct.
Besides fixing the corruption problems you're seeing, we might be able to help with your setup as well. What workload will you be using PVFS for mostly? By that I mean, do you intend to just copy DVDs to the file system once, and then access them repeatedly at a (hopefully) high bandwidth? Did you want to be able to distribute the DVDs over the nodes in your cluster so they can be accessed from anywhere? How big are the disks on each of the nodes in your cluster?
Each node has 3 x 400GB disks, with 3 x 311GB set-up as a raid 0, formatted as ext3. This is where Rocks stores the /mnt/pvfs2 data. So I have approx 3TB available for access via pvfs. The cluster has a single user and does not have to be 'high availability'. The DVD storage for is an intermediate processing step, largely for convenience - so I don't have to track what data is on what node - I also wanted to use it to 'kick the pvfs2 tyres'. Eventually the DVD contents would be copied to a external backup drive and accessed only occasionally. Having three nodes to copy the DVD also makes recovering quicker if the raid0 fails for some reason. The primary intended use for the pvfs2 area was to to hold MySQL tables - but it starts to look like this may be too bleeding edge for my level of skill & experience? The nodes would run condor jobs that access & store data in the MySQL db . Using pvfs would mean I don't have to split the db's over 3 MySQL servers - one on each node - but there is some performance cost - all the data has to pass through the frontend. So, I'm using PVFS just to get access to the combined space from the three nodes - approx 3Tb on a tight budget. Not having PVFS is not a show stopper, but seemed to offer the simplest set-up. I would appreciate any suggested improvements/advice - including 'we don't advise that' :)
Could you send us the fs.conf file you're using on each of the nodes?
I've attached the config files - these were in /opt/pvfs2/etc my understanding is they are what Rocks distributes to the nodes as part of the PVFS2 Roll configuration step.
> > I have 3 terminals open showing: > tail -f /var/log/messages > tail -f /var/log/dmsg > tail -f /var/log/syslog > I see no output to these files throughout this exercise. > > The pvfs2-client.log files from each machine are attached. > > It seems this problem occurs when PVFS2 is under load? Possibly. When you get to the end of those steps, are you still able to access files in the file system?
Yes I can. I can open a text file copied from the DVD-A, but then it seems text or small files never fail the cmp/diff verification. Thanks again for the time and effort put into these exchanges and the development of PVFS. Regards Mark
Thanks,
-sam
>
> Hope this helps.
> Mark
>
>> Thanks,
>>
>> Rob
>>
>> Mark Van De Vyver wrote:
>> > Thanks Steve,
>> > I don't see any problem until I run the diff or cmp and even then
>> > these indicate the files are identical if the cmp is run
>> _immediately_
>> > after the file copy.
>> > cmp and diff only indicate a difference when a file is 'checked'
>> after
>> > some other files have been copied-checked.
>> >
>> > The files are from the NYSE trade and quote (TAQ) DVD's, so they
>> are
>> > text stored as binary.
>> >
>> > You might be able to try the following with a dozen or so large
>> binary
>> > files, I have approx 300-400GB stored in the PVFS area.
>> >
>> > Ideally the following should be run on two or more PVFS2 servers at
>> > the same time, apply this to several DVD's that have not been
>> copied
>> > to the PVFS area, then reapply the script to the same DVD's
>> after they
>> > have been copied.
>> > The following is a slightly simplified version of my script -
>> here I
>> > don't delete and re-copy when an existing file fails the cmp
>> > verification:
>> >
>> > # untested script start
>> > for fn in `ls /dvd/*large.bin|sed -e 's/\/dev\//g'`
>> > do
>> > if [ -f /mnt/pvfs2/${fn} ]
>> > then
>> > # This should 'fail' more frequently than the cmp in
>> the else
>> > clause
>> > cmp ${fn} /mnt/pvfs2/${fn}
>> > if [ $? != 0 ]
>> > then
>> > echo "Prexisting copy not exact - more frequent and
>> random?"
>> > fi
>> > else
>> > cp ${fn} /mnt/pvfs2/${fn}
>> > cmp ${fn} /mnt/pvfs2/${fn}
>> > if [ $? != 0 ]
>> > then
>> > echo " Initial copy not exact - less frequent
>> and random"
>> > fi
>> > done
>> > # untested script end
>> >
>> > Regards
>> > Mark
>> >
>> > On 3/2/07, Steve <[EMAIL PROTECTED]> wrote:
>> >> My setup is a little different in that at the moment I have 2 I/O
>> >> services
>> >> running on one box, a metadata on another and a client/samba
>> server on a
>> >> third. I have moved in the data via samba. We have copied in
>> mp3's and
>> >> avi/mpg's as well as large ISO's plus software exe's. Surely after
>> >> several
>> >> week of use we would notice some problem ?
>> >>
>> >>
>> >>
>> >> I do have another box set up as a client that happens to have a
>> dvd ROM
>> >> drive in it.
>> >>
>> >>
>> >>
>> >> What type of files ? A vob ?
>> >>
>> >> What sequence of commands would I need to do you test your
>> problem ?
>> >>
>> >> If I get a little spare time I could try for U ?
>> >>
>> >>
>> >>
>> >> Steve
>> >>
>> >>
>> >>
>> >> -------Original Message-------
>> >>
>> >>
>> >>
>> >> From: Mark Van De Vyver
>> >>
>> >> Date: 02/03/2007 08:18:11
>> >>
>> >> To: Steve
>> >>
>> >> Subject: Re: [Pvfs2-users] PVFS 2.6.2 intermittent cmp/diff
>> failure
>> >>
>> >>
>> >>
>> >> Hi Steve,
>> >>
>> >>
>> >>
>> >> > Not sure if this helps any but I have copied over 500gb of media
>> >> files to
>> >>
>> >> > pvfs2 running on old dell's 533 to 866 CPU with very little ram
>> >> running on
>> >>
>> >>
>> >> > caos3 beta 3. Although I havent done any checks other than
>> using the
>> >> media
>> >>
>> >>
>> >> > I havent noticed any problems.
>> >>
>> >> >
>> >>
>> >>
>> >>
>> >> The failures might be spurious....?
>> >>
>> >>
>> >>
>> >> > Could you have problems with the dvd device ?
>> >>
>> >>
>> >>
>> >> I doubt it - but it may not be impossible?
>> >>
>> >> This happens with the DVD drives on all three nodes, and when I
>> just
>> >>
>> >> Have one node 'working the diif/cmp failures either don't occur or
>> >>
>> >> Very, very rarely. Start all three nodes 'working' and I see
>> roughly
>> >>
>> >> 1 out of 2 binary files fail the initial diff/cmp check, but
>> very very
>> >>
>> >> Few (one every couple of DVD's fail the cmp/diff check immediately
>> >>
>> >> After the copy is done.....
>> >>
>> >>
>> >>
>> >> Thanks
>> >>
>> >> Mark
>> >>
>> >>
>> >>
>> >> >
>> >>
>> >> > -------Original Message-------
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > From: Mark Van De Vyver
>> >>
>> >> >
>> >>
>> >> > Date: 02/03/2007 03:26:40
>> >>
>> >> >
>> >>
>> >> > To: [email protected]
>> >>
>> >> >
>> >>
>> >> > Subject: [Pvfs2-users] PVFS 2.6.2 intermittent cmp/diff failure
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > Hi,
>> >>
>> >> >
>> >>
>> >> > This is a follow up on an earlier email where I reported that
>> PVFS
>> >>
>> >> >
>> >>
>> >> > 1.5.1 failed copy binary files from several DVD's.
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > I'm running a 3 node Rocks 4.2.1 Cluster, CentOS4.4, x86_64,
>> nodes are
>> >>
>> >> >
>> >>
>> >> > Connected via an unmanaged switch.
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > I have reinstalled the Rocks Cluster (all nodes), including the
>> >> PVFS2 Roll
>> >>
>> >>
>> >> >
>> >>
>> >> > The cluster is set up with the frontend as the metadaat
>> server and the
>> >>
>> >> >
>> >>
>> >> > Other two nodes are PVFS2 I/O servers and clients. The /
>> mnt.pvfs2
>> >>
>> >> >
>> >>
>> >> > Area is on a 3 disk RAID 0 partition formatted as ext3.
>> >>
>> >> >
>> >>
>> >> > After installing I ran the test steps in the "PVFS2 Quick Start
>> >>
>> >> >
>> >>
>> >> > Guide". The test steps ran without error.
>> >>
>> >> >
>> >>
>> >> > I upgraded to PVFS 2.6.2 on all nodes and re-ran the test
>> steps, again
>> >>
>> >> >
>> >>
>> >> > No errors or problems.
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > I build PVFS 2.6.2 with the following:
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > ./configure --with-kernel=</path/to/kernel26/>
>> >>
>> >> >
>> >>
>> >> > --enable-kernel-sendfile --prefix=/usr/local/pvfs2/
>> >>
>> >> >
>> >>
>> >> > Then type
>> >>
>> >> >
>> >>
>> >> > Make all
>> >>
>> >> >
>> >>
>> >> > Make kmod_install
>> >>
>> >> >
>> >>
>> >> > Make install
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > On each node I have a script that lists the files on the DVD
>> disc
>> >>
>> >> >
>> >>
>> >> > Loaded on that node.
>> >>
>> >> >
>> >>
>> >> > Each file is copied if it does not exist on the HDD (PVFS
>> area) and
>> >>
>> >> >
>> >>
>> >> > The copy is immediately verified:
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > Cp /dvd/file1 /mnt/pvfs2/file1
>> >>
>> >> >
>> >>
>> >> > Cmp /dvd/file1 /mnt/pvfs2/file1
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > `cmp` does not report any error.
>> >>
>> >> >
>> >>
>> >> > This has been done for 60-70 DVD.
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > If I insert a DVD that has previously been copied my script
>> finds that
>> >>
>> >> >
>> >>
>> >> > A file exists in the PVFS area and does a `cmp` with the DVD
>> file, if
>> >>
>> >> >
>> >>
>> >> > The file fails this comparison the file is deleted, copied,
>> verified
>> >>
>> >> >
>> >>
>> >> > (cmp).
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > I notice that frequently and randomly the previously copied
>> files will
>> >>
>> >> >
>> >>
>> >> > Fail the _initial_ `cmp` check if more than one node is
>> 'active', I.e.
>> >>
>> >> >
>> >>
>> >> > Processing a DVD.
>> >>
>> >> >
>> >>
>> >> > Once deleted and copied the second `cmp` check is passed.
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > Some details:
>> >>
>> >> >
>> >>
>> >> > The files do not fail the `cmp` check immediately after being
>> copied -
>> >>
>> >> >
>> >>
>> >> > Only when checking a previously copied file.
>> >>
>> >> >
>> >>
>> >> > The `cmp` result indicates a different byte at which the
>> files differ.
>> >>
>> >> >
>> >>
>> >> > Re-inserting the same dvd several times results if different
>> files
>> >>
>> >> >
>> >>
>> >> > Failing the first `cmp` check.
>> >>
>> >> >
>> >>
>> >> > The second check (immediately after the copy is finished) is
>> always
>> >> passed
>> >>
>> >>
>> >> >
>> >>
>> >> > This occurs rarely, if at all (I.e. I haven't noticed it),
>> when only
>> >>
>> >> >
>> >>
>> >> > One node is processing a DVD.
>> >>
>> >> >
>> >>
>> >> > This only occurs with binary files - which are relatively large
>> >> 200MB - 2
>> >> GB
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > This never occurs with text files - which are also small 100'sKB
>> >>
>> >> >
>> >>
>> >> > The pvfs2-client.log file is empty on each node.
>> >>
>> >> >
>> >>
>> >> > I have tried using diff and experience the same results.
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > This is similar to an error I was seeing in PVFS 1.5.1 -
>> hence the
>> >>
>> >> >
>> >>
>> >> > Upgrade. I've also changed my previous script which `dd`
>> copied the
>> >>
>> >> >
>> >>
>> >> > DVD to memory (approx 8GB), then wrote this ISO file to the
>> PVFS2 area
>> >>
>> >> >
>> >>
>> >> > - this worked fine for initial copies, but failed for re-
>> copies. At
>> >>
>> >> >
>> >>
>> >> > That time I wasn't verifiying the copy, so it was the copy to
>> the
>> >>
>> >> >
>> >>
>> >> > PVFS2 area that failed.....
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > Finally, on one occasion when manually running `cmp` on a file I
>> >>
>> >> >
>> >>
>> >> > Noticed the following sequence.
>> >>
>> >> >
>> >>
>> >> > Cmp file1 file2 (pass)
>> >>
>> >> >
>> >>
>> >> > Cmp file1 file2 (pass)
>> >>
>> >> >
>> >>
>> >> > Difffile1 file2 (fail)
>> >>
>> >> >
>> >>
>> >> > Cmp file1 file2 (fail)
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > Is this known behavior with a known workaround/configuration
>> setting?
>> >>
>> >> >
>> >>
>> >> > The behavior I see made me guess a caching or network issue
>> (there are
>> >>
>> >> >
>> >>
>> >> > No other machines on the cluster network).
>> >>
>> >> >
>> >>
>> >> > Can anyone suggest PVFS configuration settings that will make
>> PVFS more
>> >>
>> >> > robust.
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > I'm not a programmer or linux guru - I just spent this summer
>> >>
>> >> >
>> >>
>> >> > Converting from winxp...
>> >>
>> >> >
>> >>
>> >> > I'm happy to explore some possible fixes, but don't assume
>> too much :)
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > Thanks in advance
>> >>
>> >> >
>> >>
>> >> > Mark
>> >>
>> >> >
>> >>
>> >> > _______________________________________________
>> >>
>> >> >
>> >>
>> >> > Pvfs2-users mailing list
>> >>
>> >> >
>> >>
>> >> > [email protected]
>> >>
>> >> >
>> >>
>> >> > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >>
>> >>
>> > _______________________________________________
>> > Pvfs2-users mailing list
>> > [email protected]
>> > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>> >
>>
>> <copy-taq-dvd-monitor.sh>
>> <copy-taq-dvd.sh>
>> <pvfs2-client.log.frontend>
>> <pvfs2-client.log.compute-0-0>
>> <pvfs2-client.log.compute-0-1>
> _______________________________________________
> Pvfs2-users mailing list
> [email protected]
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
pvfs2-fs.conf
Description: Binary data
pvfs2-server.conf-frontend
Description: Binary data
pvfs2-server.conf-pvfs2-compute-0-0
Description: Binary data
pvfs2-server.conf-pvfs2-compute-0-1
Description: Binary data
_______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
