Knew I was forgetting something, what is the value of TroveSyncData? Michael
On Wed, Sep 28, 2011 at 7:45 PM, Michael Moore <[email protected]> wrote: > No doubt something is awry. Offhand I'm suspecting the network. A couple > things that might help give a direction: > 1) Do an end-to-end TCP test between client/server. Something like iperf or > nuttcp should do the trick. > 2) Check server and client ethernet ports on the switch for high error > counts (not familiar with that switch, not sure if it's managed or not). > Hardware (port/cable) errors should show up in the above test. > 3) Can you mount the PVFS2 file system on the server and run some I/O tests > (single datafile per file) to see if the network is in fact in play. > 4) What are the number of datafiles (by default) each file you're writing > to is using? 3? > 5) When you watch network bandwidth and see 10 MB/s where is that? On the > server? > 6) What backend are you using for I/O, direct or alt-aio. Nothing really > wrong either way, just wondering. > > It sounds like based on the dd output the disks are capable of more than > you're seeing, just need to narrow down where the performance is getting > squelched. > > Michael > > > On Wed, Sep 28, 2011 at 6:10 PM, Jim Kusznir <[email protected]> wrote: > >> Hi all: >> >> I've got a pvfs2 install on my cluster. I never felt it was >> performing up to snuff, but lately it seems that things have gone way, >> way down in total throughput and overall usability. To the tune that >> jobs writing out 900MB will take an extra 1-2 hours to complete due to >> disk I/O waits. A 2-hr job that would write about 30GB over the >> course of the run (normally about 2hrs long) takes up to 20hrs. Once >> the disk I/O is cut out, it completes in 1.5-2hrs. I've noticed >> personally that there's up to a 5 sec lag time when I cd into >> /mnt/pvfs2 and do an ls. Note that all of our operations are using >> the kernel module / mount point. Our problems and code base do not >> support the use of other tools (such as the pvfs2-* or the native MPI >> libraries); its all done through the kernel module / filesystem >> mountpoint. >> >> My configuration is this: 3 pvfs2 servers (Dell PowerEdge 1950's with >> 1.6Ghz quad-core CPUs, 4GB ram, raid-0 for metadata+os on perc5i >> card), Dell Perc6e card with hardware raid6 in two volumes: one on a >> bunch of 750GB sata drives, and the other on its second SAS connector >> to about 12 2tb WD drives. The two raid volumes are lvm'ed together >> in the OS and mounted as the pvfs2 data store. Each server is >> connected via ethernet to a stack of LG-errison gig-e switches >> (stack==2 switches with 40Gbit stacking cables installed). PVFS 2.8.2 >> used throughout the cluster on Rocks (using site-compiled pvfs, not >> the rocks-supplied pvfs). OSes are CentOS5-x-based (both clients and >> servers). >> >> As I said, I always felt something wasn't quite right, but a few >> months back, I performed a series of upgrades and reconfigurations on >> the infrastructure and hardware. Specifically, I upgraded to the >> lg-errison switches and replaced a full 12-bay drive shelf with a >> 24-bay one (moving all the disks through) and adding some additional >> disks. All three pvfs2 servers are identical in this. At some point >> prior to these changes, my users were able to get acceptable >> performance from pvfs2; now they are not. I don't have any evidence >> pointing to the switch or to the disks. >> >> I can run dd if=/dev/zero of=testfile bs=1024k count=10000 and get >> 380+MB/s locally on the pvfs server, writing to the partition on the >> hardware raid6 card. From a compute node, doing that for 100MB file, >> I get 47.7MB/s to my RAID-5 NFS server on the head node, and 36.5MB/s >> to my pvfs2 mounted share. When I watch the network >> bandwidth/throughput using bwm-ng, I rarely see more than 10MB/s, and >> often its around 4MB/s with a 12-node IO-bound job running. >> >> I originally had the pvfs2 servers connected to the switch with dual >> gig-e connections and using bonding (ALB) to make it more able to >> serve multiple nodes. I never saw anywhere close to the throughput I >> should. In any case, to test of that was the problem, I removed the >> bonding and am running through a single gig-e pipe now, but >> performance hasn't improved at all. >> >> I'm not sure how to troubleshoot this problem further. Presently, the >> cluster isn't usable for large I/O jobs, so I really have to fix this. >> >> --Jim >> _______________________________________________ >> Pvfs2-users mailing list >> [email protected] >> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users >> > >
_______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
