One thing that I've seen in the past is that pvfs2-cp does not often - especially with IB - use large enough buffers to push the numbers up really high, by adjusting the buffer settings in pvfs2-cp with `pvfs2 -t -b <4MB+>` I've seen 1GByte+ bandwidth numbers.
But the mpi-io tests are less easy to fool.. Can you check to see how much cpu usage is going on with the native IB interfaces? If something is spinning, or even using more than 50% we should look at doing some profiling to see what the problem areas are. Kyle On 8/15/07, xiangyong ouyang <[EMAIL PROTECTED]> wrote: > > I setup pvfs-2.6.3 in a cluster with IB support(ofed-1.2) and linux kernel > 2.6.20.5. I compiled pvfs2 using the following flags: > > ./configure --prefix=/tmp --with-db=/tmp/BerkeleyDB > --with-openib=/usr/local/ofed --with-openib-li > bs=/usr/local/ofed/lib64 --with-openib-includes=/usr/local/ofed/include > --with-mpi=/home/ouyangx/m > pich2-install --with-kernel=/usr/src/linux-2.6.20.5 > > make; make install > > > Then I start pvfs2 with IB and IPoIB as transport layer respectively. 4 > machines are IO server, 1 machine serves as metadata server. 1 machine is > pvfs2-client and mounts up the pvfs2 file system. > > After that, I tested pvfs2 performance with "pvfs2-cp -t" to copy a local > file to pvfs2 file system. At all different file size, pvfs2/IPoIB yields a > better performance than pvfs2/IB. Here are the numbers (MB/sec). > > file-sieze IpoIB IB > 100MB 402 300 > 400MB 405 332 > 800MB 411 334 > > > > I also used MPI-IO(mpich2-1.0) to test pvfs2 performance. Again IPoIB > outperforms IB. > > I believe something must be wrong with my configuration. Any > comments/suggestions? Thanks! > > > > xiangyong ouyang > > > > > > > _______________________________________________ > Pvfs2-users mailing list > [email protected] > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users > -- Kyle Schochenmaier
_______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
