On Jun 21, 2007 10:20 -0300, Balagopal Pillai wrote: > I am using Lustre 1.6.0.1 with one OST and 20 clients in an > HPC cluster. > The OST/MDT/MGS has a 16 channel 3ware 9650 using raid6. > > *Now the problem - Lustre performance dips a lot when it comes to small > files. Please see the following fileop -f 5 test comparing NFS and Lustre -
We have had reports of read performance problems with 3ware in the past. > Lustre - > > Fileop: File size is 1, Output is in Ops/sec. (A=Avg, B=Best, W=Worst) > . mkdir rmdir create read write close stat access chmod > readdir link unlink delete Total_files > A 5 1654 691 132 14228 719 4874 1987 32737 1718 > 2506 1262 1340 1608 125 > > > NFS - > Fileop: File size is 1, Output is in Ops/sec. (A=Avg, B=Best, W=Worst) > . mkdir rmdir create read write close stat access chmod > readdir link unlink delete Total_files > A 5 177 594 459 380747 137392 2282 1219 444312 502 > 1274 306 513 464 125 > Could you please recommend any tunables to get a bit more > performance out of Lustre with lots of small files? Lots of small files > was bad in GFS too, but it was better than NFS though. Try setting the debug on the clients to zero "sysctl -w lnet.debug=0". > *Also the read performance of Lustre seems to be a little behind NFS. I > had /opt which has all the software for users moved to Lustre in the new > setup. But > software like Matlab, Splus etc takes almost a minute to come up. The > second time is very fast though, maybe due to caching. So i am thinking > of putting /opt > back to NFS. Is it possible to boost the read performance of Lustre a bit? Turn off debug is the easiest. > *One last question. In the older Lustre setup (version 1.4.5), i have 5 > scsi drives one each as an OST for a single volume. The volume became > full. But df still reported that there is 27GB free. There doesn't seem > to be an lfs df option in that version of Lustre. So i couldn't see the > individual utilization of each of the 5 OST. Is this a striping problem? In the absence of "lfs df" use /proc to find per-OST usage on the client: grep '[0-9]' /proc/fs/lustre/osc/*/kbytes* Lustre 1.6 should be better at balancing the OST usage than 1.4. Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc. _______________________________________________ Lustre-discuss mailing list [email protected] https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
