If you have a single-node setup.. what happens if you mount pvfs2 to /tmp/pvfs2-storage ? This should immediately eliminate a lot of potential issues with disks at least.
Kyle Schochenmaier On Wed, Jul 1, 2009 at 5:57 PM, David Bonnie<[email protected]> wrote: > Sam - > > All of the nodes checked out fine with netpipe, still no errors on any of > the adapters. > > - Dave > > On Wed, Jul 1, 2009 at 4:47 PM, Sam Lang <[email protected]> wrote: >> >> On Jul 1, 2009, at 5:45 PM, David Bonnie wrote: >> >> I'll run it on each node and let you know if anything is out of place. I >> believe the above results are fine for GigE, yes? >> >> They certainly don't match with the numbers you're getting from PVFS. >> -sam >> >> - Dave >> >> On Wed, Jul 1, 2009 at 4:20 PM, Sam Lang <[email protected]> wrote: >>> >>> David, >>> It sounds like your initial thought (that there is a network >>> problem) could be correct. I would probably explore that first. What sort >>> of numbers do you get from netpipe runs (or even bmi_pingpong) between >>> client and server? >>> -sam >>> On Jul 1, 2009, at 5:15 PM, David Bonnie wrote: >>> >>> Sorry for not being clear. >>> >>> The hardware and software is unchanged. Runs from a few months ago (on >>> 2.8.0) performed as expected. Current runs (on both 2.8.0 and 2.8.1) are >>> slow. >>> >>> The nodes are sitting there with very low CPU usage even when running the >>> benchmark. I'm the only one running any jobs and there aren't any processes >>> running (the system load is < .02 and the cpu usage is pretty much 0%). >>> >>> The local disks haven't changed and are empty except for the pvfs2 >>> storage space; performance is bad even when I put the PVFS2 file system >>> storage onto a very fast (>300 MB/s local bandwidth) Atrato vlun connected >>> over fiber channel. >>> >>> My initial thought is that some hardware along the line died but I can't >>> seem to pinpoint it. All of the network interfaces show 0 errors and 0 >>> dropped packets. >>> >>> - Dave >>> >>> On Wed, Jul 1, 2009 at 4:10 PM, Rob Ross <[email protected]> wrote: >>>> >>>> Hi David, >>>> >>>> I still don't get it: when was the performance good? Same software and >>>> hardware, just some time in the past? Or is there a software change? >>>> >>>> The nodes aren't being used for anything else, there are no rogue >>>> processes, and the local file systems are otherwise empty? >>>> >>>> Thanks, >>>> >>>> Rob >>>> >>>> On Jul 1, 2009, at 5:05 PM, David Bonnie wrote: >>>> >>>>> Rob - >>>>> >>>>> Performance is down across all PVFS2 installations. The benchmark >>>>> simply creates files of a random size (between 1 and 25 MB) in a single >>>>> folder on the mounted PVFS2 partition, 16 KB at a time. It's not anywhere >>>>> near ideal, but it's the workload I'm working with. >>>>> >>>>> Prior to this problem we were getting ~22 MB/s write throughput and >>>>> we're down to about 2.5 MB/s for no apparent reason. Reads are down from >>>>> about 55 MB/s to 30 MB/s. No hardware has changed and as far as I can >>>>> tell >>>>> no hardware has died either. >>>>> >>>>> - Dave >>>>> >>>>> >>>>> On Wed, Jul 1, 2009 at 4:00 PM, Rob Ross <[email protected]> wrote: >>>>> Do you mean that 2.8.0 is fast and 2.8.1 is slow? Can you describe the >>>>> benchmark and how you are doing your measurements? >>>>> >>>>> Rob >>>>> >>>>> >>>>> On Jul 1, 2009, at 4:43 PM, David Bonnie wrote: >>>>> >>>>> Hello all - >>>>> >>>>> I'm having trouble figuring out a problem with performance depredation >>>>> on a simple 10 node cluster. Prior runs on the cluster (before this >>>>> problem >>>>> manifested itself) resulted in bandwidth and IOPS about 10 times higher >>>>> on a >>>>> small file creation workload. Each node is running as a metadata server >>>>> and >>>>> a data server. >>>>> >>>>> The problem is persistent between versions and installations of PVFS2 >>>>> 2.8.0 and 2.8.1. Rebooting all of the nodes didn't improve anything. The >>>>> network connections (simple GigE) showed no errors or dropped packets. >>>>> Using different physical disks (both SAS and FC) didn't improve things. >>>>> The kernel logs didn't show anything out of place nor did the pvfs2 >>>>> server >>>>> or client logs. It seems like a network issue but I can't seem to find >>>>> anything wrong with any of the connections. >>>>> >>>>> Has anyone seen this kind of problem before? I seem to remember >>>>> something on the list before about performance suddenly dropping but I >>>>> can't >>>>> find the message now (of course). Any insight would be appreciated! >>>>> >>>>> Thanks, >>>>> >>>>> - Dave >>>>> _______________________________________________ >>>>> Pvfs2-developers mailing list >>>>> [email protected] >>>>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers >>>>> >>>>> >>>> >>> >>> _______________________________________________ >>> Pvfs2-developers mailing list >>> [email protected] >>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers >>> >>> >>> _______________________________________________ >>> Pvfs2-developers mailing list >>> [email protected] >>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers >>> >> >> >> >> _______________________________________________ >> Pvfs2-developers mailing list >> [email protected] >> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers >> > > > _______________________________________________ > Pvfs2-developers mailing list > [email protected] > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers > > _______________________________________________ Pvfs2-developers mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
