It would be interesting to know what the numbers are for copying pvfs2->local_memory and then back, local_memory->pvfs2 and compare that to doing pvfs2->pvfs2. If I were wanting to further look at these results and try to discern what the slow parts are, some important things to note are cpu utilization on the servers and clients.
You should also ideally have a difference in cpu utilization when you do 'cp -r' vs. the pvfs2-cp which doesnt go through the kernel module. What is the cpu load on the client, I/O, MD servers? If you find that the MD server is getting hit heavily, you might be able to startup a second MD server, maybe even on the same box with multiple-cores. I'm also assuming you are using very small files. PVFS2 does not particularly shine with large numbers of small files in general, especially when compared to systems that do caching (nfs). >From what I understand about whats going on in the servers during these operations, these numbers kind of fall in line with what I would have expected. Just some random thoughts, good luck! Kyle On Thu, Sep 4, 2008 at 4:13 PM, hawaii <[EMAIL PROTECTED]> wrote: > A pvfs2 I/O test > (for comments) > =================== > > pvfs version: 2.7.1 > configuration: 1 meta server + 3 data servre. Each server has 2 cpus > reserved > > Tests: within pvfs2; from pvfs2 to local; nfs > Use pvfs2-cp-r, pvfs2-cp-r.pl, and cp > (thanks Mark Bartelt for providing pvfs2-cp-r and pvfs2-cp-r.pl) > > Source direcotry: WRF > df -k WRF > lots of subdirctories > 511868 WRF > > (1) pvfs2 -> pvfs2 > > time pvfs2-cp-r WRF tmp > real 4m16.414s > user 0m38.385s > sys 0m31.874s > I/O 2 MB/s > > time pvfs2-cp-r.pl WRF tmp > Copying files from "WRF" to "tmp/WRF" using pvfs2-cp > real 4m21.573s > user 0m38.741s > sys 0m32.293s > I/O 1.96 MB/s > > time cp -rp WRF tmp > Lots of 0 files (unreliable) > real 7m21.473s > user 0m0.100s > sys 0m5.792s > I/O 1.16 MB/s > > (2) pvfs2 -> local hard drive > > time pvfs2-cp-r.pl /pvfs2-dir/WRF /home/user > Copying files from "/pvfs2-dir/WRF" to "/home/user/WRF" using pvfs2-cp > real 1m19.311s > user 0m19.008s > sys 0m29.327s > I/O 6.4 MB/s > > time pvfs2-cp-r /pvfs2-dir/WRF /home/user > real 1m10.739s > user 0m20.609s > sys 0m30.021s > I/O 7.25 MB/s > > time cp -rp /pvfs2-dir/WRF /home/user > Lots of 0 files (unreliable) > real 1m34.071s > user 0m0.113s > sys 0m3.906s > I/O 5.45 MB/s > > (3) cp an nfs file to local disk > 2057207616 t.dat > time cp -p /nfs-dir/t.dat /home/usr/. > real 2m55.894s > user 0m0.257s > sys 0m6.946s > Read 11.7 MB/s > > Conclusion from this test: > (1) pvfs2-cp-r is the fastest and cp -r is the slowest > (2) it is slower to copy directory inside pvfs2 than to local disk > (3) I/O pvfs2 is much slower than nfs !!! > Any comments on the results? > _______________________________________________ > Pvfs2-users mailing list > [email protected] > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users > > -- Kyle Schochenmaier _______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
