On Mon, Nov 26, 2007 at 11:59:32AM -0700, Andreas Dilger wrote: >On Nov 26, 2007 18:16 +0100, Andrei Maslennikov wrote: >> Confirmed: 1.6.3 striped write performance sux. >> >> With 1.6.2, I see this: >> >> [EMAIL PROTECTED] ~]$ lfs setstripe /lustre/162 0 0 3 >> [EMAIL PROTECTED] ~]$ lmdd.linux of=/lustre/162 bs=1024k time=180 fsync=1 >> 157705.8304 MB in 180.0225 secs, 876.0341 MB/sec >> >> I.e. 1.6.2 had nicely joined the aggregate bw of three OSTs of 300 MB/sec >> each >> into the almost 900 MB/sec. > >Can you verify that you disabled data checksumming: > echo 0 > /proc/fs/lustre/llite/*/checksum_pages
those checksums were off in my runs (they were off by default?). so I don't think any of the checksums are making a difference. >Note that there are 2 kinds of checksumming that Lustre does. The first one >is checksumming of data in client memory, and the second one is checksumming >of data over the network. Setting $LPROC/llite/*/checksum_pages turns on/off >both in-memory and wire checksums. Setting $LPROC/osc/*/checksums turns on/off >the network checksums only. good to know. thanks. all those are new in 1.6.3? >If checksums are disabled, can you please report if the CPU usage on the >client is consuming all of the CPU, or possibly all of a single CPU on 1.6.3 >and on 1.6.2? with checksums disabled, a 1.6.3+ client looks like: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7437 root 15 0 0 0 0 R 57 0.0 7:31.77 ldlm_poold 18547 rjh900 15 0 5820 504 412 S 3 0.1 0:34.52 dd which is interesting. ldlm_poold is using an awful lot of cpu. a 'top' on a 1.6.2 client shows only dd using significant cpu (plus the usual small percentages for ptlrpcd, kswapd0, pdflush, kiblnd_sd_*) cheers, robin >> On Nov 26, 2007 4:58 PM, Andrei Maslennikov >> <[EMAIL PROTECTED]> wrote: >> > On Nov 26, 2007 3:32 PM, Robin Humble <[EMAIL PROTECTED]> wrote: >> > >> > > >> I'm seeing what can only be described as dismal striped write >> > > >> performance from lustre 1.6.3 clients :-/ >> > > >> 1.6.2 and 1.6.1 clients are fine. 1.6.4rc3 clients (from cvs a couple >> > > >> of days ago) are also terrible. >> > >> > I have 3 OSTs capable to deliver 300+ MB/sec each for large streaming >> > writes >> > with 1M blocksize. On one client, with one OST I can see almost all >> > this bandwidth over Infiniband. If I run three processes in parallel on >> > this very client, >> > each writing into a separate OST, I arrive to 520 MB/sec aggregate (3 >> > streams at >> > approx 170+ MB/sec each). >> > >> > If I try to stripe over these three OSTs on this client, performance of one >> > stream drops to 60+ MB/sec. Changing stripesize to a smaller one (1/3 MB) >> > makes things worse. Writing with larger block sizes (9M, 30M) does not >> > improve >> > things. Increasing the stripesize to 25 MB allows to approach the speed >> > of a single OST, as one would expect (blocks are round robined over all >> > three >> > OSTs). But never more. Zeroing checksums on the client does not help. >> > >> > Will now be downgrading the client to 1.6.2 to see if this helps. > >Cheers, Andreas >-- >Andreas Dilger >Sr. Staff Engineer, Lustre Group >Sun Microsystems of Canada, Inc. _______________________________________________ Lustre-discuss mailing list [email protected] https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
