Milo I would first start with examining the potential for network issues here. -Try netpipe or something? -About your first test, are you copying FROM your home directory, that could be an obbvious source of slowness. -Iozone will almost always give bad numbers for pvfs2. - any reason for the 4mb strip size for a single server test? It shouldn't technically affect performance here. - if you are just doing single server tests what does a bone-stock pvfs config file produce? -- I'm still leaning towards network issues, anything in the server logs ?
Kyle Kyle Schochenmaier On Jul 23, 2009 12:00 PM, "Milo" <[email protected]> wrote: Hi, guys. I'm getting surprisingly poor performance on my PVFS2 cluster. Here's the setup: *) 13 nodes running PVFS2 2.8.1 with Linux Kernel 2.6.28-13 server, each with a 15 drive RAID-5 array. *) The RAID-5 array gets 60 MB/s local write speeds with XFS according to iozone (writing in 4 MB records) I'd like to get at least 50 MB/s/server from the cluster and I've been testing this with a single PVFS2 server and client with the client running either on the same node or a node on the same switch (it doesn't seem to make a lot of difference). The server is configured with Trove syncing off, a 4 MB strip size simple_strip distribution, and a 1 MB FlowBufferSizeBytes. Results have been as follow: With TroveMethod alt-aio or default, I'm getting around 15 MB/s when transferring a 3 GB file through pvfs2-cp: r...@ss239:~# pvfs2-cp -t ./file.3g /mnt/pvfs2/out Wrote 2867200000 bytes in 192.811478 seconds. 14.181599 MB/seconds dd'ing a similar file through pvfs2fuse gets about a third of that performance, 5 MB/s: r...@ss239:~# dd if=/dev/zero of=/mnt/pvfs2fuse/out bs=1024K count=1024 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 206.964 s, 5.2 MB/s I get similar results using iozone writing through the fuse client. If I switch the method to null-aio, things speed up a lot, but it's still suspiciously slow: r...@ss239:~# pvfs2-cp -t ./file.out /mnt/pvfs2fuse/out7-nullaio Wrote 2867200000 bytes in 60.815127 seconds. 44.962086 MB/seconds r...@ss239:~# dd if=/dev/zero of=/mnt/pvfs2fuse/out-nullaio bs=1024K count=1024 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 21.0201 s, 51.1 MB/s I suspect there's some network bottleneck. I'm going to try to adjust the MTU as Jim just did. But are there any other configuration options I should look into? Thanks. ~Milo _______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
_______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
