On Jun 21, 2007 19:00 +0200, Andrei Maslennikov wrote: > We are in the process of testing of a would-be low cost Lustre head > (a black box disk server with an Infiniband outlet). The box contains > 3 standalone RAID-6 controllers capable to deliver 300 MB/sec each. > The box has 4 cores at 3GHz, so 3 parallel dd processes are delivering > 3x300=900 MB/sec aggregate without any problem.
Is this locally, or from the lustre client? > This configuration perfectly works, but we are only able to achieve max > 336 MB/sec for a striped file on a stanadlone IB client. Is that a single-threaded test. > Our further actions > will be to play with the ost_num_threads and/or mds_num_threads, > cache segment sizes, maxcmds etc. Before doing that, I however would > seek for a guru's comment on the following: not that we will *never* be > able to detach from the performance of a single controller due to the fact > that our MDT and MGS are using their areas served by only one of the > three controllers? Could you rephrase the question? > If the answer is "yes", then the better bet would probably be to come back > to an LVM-based solution which we have previously discarded as it was > starting only at 750 MB/sec. And to place MDT, MGS and 1 OST on 3 > separate logical volumes each striped over the 3 controllers. You hardly need to have a separate LV/controller for just the MGS. It would be better to have 2 OSTs and put the MGS on a small LV on the same controller with the MDS. > PS We had the "lnet" option set as "networks=o2ib". Is there any > chance that we were using IPoIB in the place of RDMA? This > could explain low performance, as well. A. No, that would happen only if you had "networks=tcp". Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc. _______________________________________________ Lustre-discuss mailing list [email protected] https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
