On 6/22/07, Andreas Dilger <[EMAIL PROTECTED]> wrote:
On Jun 21, 2007 19:00 +0200, Andrei Maslennikov wrote: > We are in the process of testing of a would-be low cost Lustre head > (a black box disk server with an Infiniband outlet). The box contains > 3 standalone RAID-6 controllers capable to deliver 300 MB/sec each. > The box has 4 cores at 3GHz, so 3 parallel dd processes are delivering > 3x300=900 MB/sec aggregate without any problem. Is this locally, or from the lustre client?
Locally on the box, before it was configured as an OSS.
This configuration perfectly works, but we are only able to achieve max > 336 MB/sec for a striped file on a stanadlone IB client. Is that a single-threaded test.
Yes, it is a thingle-threaded test. The goal is to provide a highest possible peak throughput for just one process on the client. The OSS has 3 OSTs, and the file is being striped via IB over these three OSTs, i.e. all 3 controllers are being used simultaneously.
Our further actions > will be to play with the ost_num_threads and/or mds_num_threads, > cache segment sizes, maxcmds etc. Before doing that, I however would > seek for a guru's comment on the following: not that we will *never* be > able to detach from the performance of a single controller due to the fact > that our MDT and MGS are using their areas served by only one of the > three controllers? Could you rephrase the question?
In our case MDT and MGS are using the bandwidth of only one controller (300 MB/sec) to store their data, while they are serving three OSTs spread over three such controllers. I suspect that in this configuration one cannot get an aggregate of 3x300 out of all OSTs together, because the MDT and MGS cannot operate at more than 300x1. So probably the only remedy is to use LVM for MDT and MGS, and spread their data over all controllers.
If the answer is "yes", then the better bet would probably be to come back > to an LVM-based solution which we have previously discarded as it was > starting only at 750 MB/sec. And to place MDT, MGS and 1 OST on 3 > separate logical volumes each striped over the 3 controllers. You hardly need to have a separate LV/controller for just the MGS. It would be better to have 2 OSTs and put the MGS on a small LV on the same controller with the MDS.
I tried this, but there was no improvement... Andrei.
_______________________________________________ Lustre-discuss mailing list [email protected] https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
