The "FlowBufferSizeBytes" by default is 262144. I set it to 1M, and doesn't see a difference.
Thanks a lot Kyle! -Neutron On Fri, Apr 30, 2010 at 8:58 PM, Kyle Schochenmaier <[email protected]> wrote: > I believe each iozone process is writing its own file... > > Im curious to see if pvfs2.8.2 addresses any of this, would it be > possible to try it out ? > > As a general note with IB hardware - and it probably wont help with > this specific problem - you may also want to set FlowBufferSizeBytes > in your config file up to 1MB from 64k. > > Best, > > Kyle Schochenmaier > > > > On Fri, Apr 30, 2010 at 7:39 PM, neutron <[email protected]> wrote: >> Thanks Kyle! >> >> I disabled the "TroveSyncMeta" to "no", but it doesn't make a difference. >> The poster you mentioned said the problem occurs with serial programs, >> and parallel program wont' see such a hit. >> what's serial/parallel here mean? I let 8 iozone processes run on 8 >> nodes and aggregate the bandwidth. Isn't it a parallel program? >> >> http://www.beowulf-underground.org/pipermail/pvfs2-users/2009-April/002770.html >> >> >> >> -Neutron >> >> >> On Fri, Apr 30, 2010 at 8:17 PM, Kyle Schochenmaier <[email protected]> >> wrote: >>> Hi Neutron - >>> >>> Check out a very similar thread from about a year ago : >>> http://www.beowulf-underground.org/pipermail/pvfs2-users/2009-April/002770.html >>> >>> Im not sure if we found out what was actually causing the performance >>> differences.. but its probably a start to look into. >>> >>> Best, >>> Kyle Schochenmaier >>> >>> >>> >>> On Fri, Apr 30, 2010 at 7:08 PM, neutron <[email protected]> wrote: >>>> Hi all, >>>> >>>> I have a pvfs2-2.8.1 with 4 DS. "BMIModules" is set to "bmi_ib". I >>>> use simple-tripe mode with stipe-size =1MB. >>>> >>>> 8 client nodes mount pvfs2, and I use iozone to measure the write >>>> throughput, with each client node runs one iozone process. >>>> Each process writes 1GB file with block size = 4MB. >>>> >>>> I got the following results: >>>> >>>> Test running: >>>> >>>> Children see throughput for 8 initial writers = 98513.53 KB/sec >>>> Min throughput per process = 11406.64 KB/sec >>>> Max throughput per process = 12642.13 KB/sec >>>> Avg throughput per process = 12314.19 KB/sec >>>> Min xfer = 946176.00 KB >>>> CPU Utilization: Wall time 86.492 CPU time 6.355 CPU >>>> utilization 7.35 % >>>> >>>> >>>> Test running: >>>> Children see throughput for 8 rewriters = 2499467.09 KB/sec >>>> Min throughput per process = 289479.62 KB/sec >>>> Max throughput per process = 321507.38 KB/sec >>>> Avg throughput per process = 312433.39 KB/sec >>>> Min xfer = 946176.00 KB >>>> CPU utilization: Wall time 3.272 CPU time 5.990 CPU >>>> utilization 183.08 % >>>> >>>> >>>> We can see the "initial write" gets very low throughput, while the >>>> "rewrite" gives very high bandwidth. >>>> My questions are: >>>> 1. Why 8 client nodes as a whole get low throughput? >>>> 2. Why at "rewrite" phase, the write bandwidth boosts ? >>>> >>>> >>>> >>>> -Neutron >>>> _______________________________________________ >>>> Pvfs2-users mailing list >>>> [email protected] >>>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users >>>> >>> >> > _______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
