Which version of PVFS are you using? Your setup will work better if each of your 16 servers are both meta and I/O servers. Your current configuration causes a bottleneck at the metadata server.
BEcky -- Becky Ligon HPC Admin Staff PVFS/OrangeFS Developer Clemson University/Omnibond.com 864-650-4065 > Hi guys, > > I am a PVFS2 newbie and made some performance tests using IOZone, but the > results puzzle me. I have 16 machines. One is meta data server, and other > 15 machines are both PVFS2 IO servers and clients. Each client machine > runs one IOZone process, so the aggregate performance is measured. Those > machines are configured as follows: one Intel i7-860 processor, 16GB DDR3 > memory and 1TB SATA hard disk. They are connected through a gigabit > Ethernet switch. The OS is Debian Lenny (2.6.26 kernel). The PVFS2 is > 2.8.2 with default configuration. > > The IOZone command used is: ./iozone -i 0 -i 1 -i 2 -r 4m -s 32g -t 15 -+m > pvfs_client_list. Since the memory capacity for each machine is 16GB, so I > set the test file size to 32GB to exercise the PVFS2 heavily. The result > is listed below: > > Record Size 4096 KB > File size set to 33554432 KB > Network distribution mode enabled. > Command line used: ./iozone -i 0 -i 1 -i 2 -r 4m -s 32g -t 15 -+m > pvfs_client_list > Output is in Kbytes/sec > Time Resolution = 0.000001 seconds. > Processor cache size set to 1024 Kbytes. > Processor cache line size set to 32 bytes. > File stride size set to 17 * record size. > Throughput test with 15 processes > Each process writes a 33554432 Kbyte file in 4096 Kbyte records > > Test running: > Children see throughput for 15 initial writers = 785775.56 > KB/sec > Min throughput per process = 50273.01 KB/sec > Max throughput per process = 53785.79 KB/sec > Avg throughput per process = 52385.04 KB/sec > Min xfer = 31375360.00 KB > > Test running: > Children see throughput for 15 rewriters = 612876.38 KB/sec > Min throughput per process = 39466.78 KB/sec > Max throughput per process = 41843.63 KB/sec > Avg throughput per process = 40858.43 KB/sec > Min xfer = 31649792.00 KB > > Test running: > Children see throughput for 15 readers = 366397.27 KB/sec > Min throughput per process = 9371.45 KB/sec > Max throughput per process = 29229.74 KB/sec > Avg throughput per process = 24426.48 KB/sec > Min xfer = 10760192.00 KB > > Test running: > Children see throughput for 15 re-readers = 370985.14 KB/sec > Min throughput per process = 9850.98 KB/sec > Max throughput per process = 29660.86 KB/sec > Avg throughput per process = 24732.34 KB/sec > Min xfer = 11145216.00 KB > > Test running: > Children see throughput for 15 random readers = 257970.32 KB/sec > Min throughput per process = 8147.65 KB/sec > Max throughput per process = 20084.32 KB/sec > Avg throughput per process = 17198.02 KB/sec > Min xfer = 13615104.00 KB > > Test running: > Children see throughput for 15 random writers = 376059.73 KB/sec > Min throughput per process = 24060.38 KB/sec > Max throughput per process = 26446.96 KB/sec > Avg throughput per process = 25070.65 KB/sec > Min xfer = 30527488.00 KB > > I have three questions: > 1. Why does write outperforms rewrite significantly? According to > IOZone's document, rewrite is supposed to perform better, since it > writes to a file which already exists, and the metadata is already > there. > 2. Why is write/random-write faster than read/random-read so much? This > result is really unexpected. I feel that read is supposed to be faster. > Is there anything wrong in my result numbers? > 3. Observing the max and min throughput per process in each test item, > you can find that in write/re-write/random-write, the difference between > max and min is acceptable; while in read/re-read/random-read, the max > throughput is about two or three times of the min number. How can I > explain this result? Is it normal? > > These results are out of my expectation. Is it possible that they are > caused by faulty hardware (network or disk) or configuration? > > Any advice is appreciated. > > Sincerely, > Wantao_______________________________________________ > Pvfs2-users mailing list > [email protected] > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users > _______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
