I believe each iozone process is writing its own file... Im curious to see if pvfs2.8.2 addresses any of this, would it be possible to try it out ?
As a general note with IB hardware - and it probably wont help with this specific problem - you may also want to set FlowBufferSizeBytes in your config file up to 1MB from 64k. Best, Kyle Schochenmaier On Fri, Apr 30, 2010 at 7:39 PM, neutron <[email protected]> wrote: > Thanks Kyle! > > I disabled the "TroveSyncMeta" to "no", but it doesn't make a difference. > The poster you mentioned said the problem occurs with serial programs, > and parallel program wont' see such a hit. > what's serial/parallel here mean? I let 8 iozone processes run on 8 > nodes and aggregate the bandwidth. Isn't it a parallel program? > > http://www.beowulf-underground.org/pipermail/pvfs2-users/2009-April/002770.html > > > > -Neutron > > > On Fri, Apr 30, 2010 at 8:17 PM, Kyle Schochenmaier <[email protected]> > wrote: >> Hi Neutron - >> >> Check out a very similar thread from about a year ago : >> http://www.beowulf-underground.org/pipermail/pvfs2-users/2009-April/002770.html >> >> Im not sure if we found out what was actually causing the performance >> differences.. but its probably a start to look into. >> >> Best, >> Kyle Schochenmaier >> >> >> >> On Fri, Apr 30, 2010 at 7:08 PM, neutron <[email protected]> wrote: >>> Hi all, >>> >>> I have a pvfs2-2.8.1 with 4 DS. "BMIModules" is set to "bmi_ib". I >>> use simple-tripe mode with stipe-size =1MB. >>> >>> 8 client nodes mount pvfs2, and I use iozone to measure the write >>> throughput, with each client node runs one iozone process. >>> Each process writes 1GB file with block size = 4MB. >>> >>> I got the following results: >>> >>> Test running: >>> >>> Children see throughput for 8 initial writers = 98513.53 KB/sec >>> Min throughput per process = 11406.64 KB/sec >>> Max throughput per process = 12642.13 KB/sec >>> Avg throughput per process = 12314.19 KB/sec >>> Min xfer = 946176.00 KB >>> CPU Utilization: Wall time 86.492 CPU time 6.355 CPU >>> utilization 7.35 % >>> >>> >>> Test running: >>> Children see throughput for 8 rewriters = 2499467.09 KB/sec >>> Min throughput per process = 289479.62 KB/sec >>> Max throughput per process = 321507.38 KB/sec >>> Avg throughput per process = 312433.39 KB/sec >>> Min xfer = 946176.00 KB >>> CPU utilization: Wall time 3.272 CPU time 5.990 CPU >>> utilization 183.08 % >>> >>> >>> We can see the "initial write" gets very low throughput, while the >>> "rewrite" gives very high bandwidth. >>> My questions are: >>> 1. Why 8 client nodes as a whole get low throughput? >>> 2. Why at "rewrite" phase, the write bandwidth boosts ? >>> >>> >>> >>> -Neutron >>> _______________________________________________ >>> Pvfs2-users mailing list >>> [email protected] >>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users >>> >> > _______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
