We are in the process of testing of a would-be low cost Lustre head
(a black box disk server with an Infiniband outlet). The box contains
3 standalone RAID-6 controllers capable to deliver 300 MB/sec each.
The box has 4 cores at 3GHz, so 3 parallel dd processes are delivering
3x300=900 MB/sec aggregate without any problem.
Following the advice obtained on this list, we are hence configuring one
single OSS with 3 OSTs inside (one OST per controller), and then stripe
over these 3 OSTs to get the best performance on IB.
This configuration perfectly works, but we are only able to achieve max
336 MB/sec for a striped file on a stanadlone IB client. Our further actions
will be to play with the ost_num_threads and/or mds_num_threads,
cache segment sizes, maxcmds etc. Before doing that, I however would
seek for a guru's comment on the following: not that we will *never* be
able to detach from the performance of a single controller due to the fact
that our MDT and MGS are using their areas served by only one of the
three controllers?
If the answer is "yes", then the better bet would probably be to come back
to an LVM-based solution which we have previously discarded as it was
starting only at 750 MB/sec. And to place MDT, MGS and 1 OST on 3
separate logical volumes each striped over the 3 controllers.
Thanks ahead to anyone who would comment on this.
Andrei.
PS We had the "lnet" option set as "networks=o2ib". Is there any
chance that we were using IPoIB in the place of RDMA? This
could explain low performance, as well. A.
_______________________________________________
Lustre-discuss mailing list
[email protected]
https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
