On Feb 24, 2011, at 10:45 AM, Jeremy Filizetti wrote: > As Chris mentioned your talking about two very different methods. I think > you can use netem with IPoIB but I have never tried it. If you use connected > mode I think your still technically doing RDMA but the maximum size (MTU) is > around 64k which isn't sufficient for higher latencies. In the first few > slides of my LUG presentation last year I have some graphs that show how RDMA > performance is affected by latency and need to be increased to compensate for > the bandwidth delay product (BDP). If you do what to use IPoIB you can add a > line similar the following in your /etc/modprobe.conf or a fille in > /etc/modprobe.d directory: > > options lnet networks=tcp(ib0) > > If you want to use RC QPs as ko2iblnd does, we use the following kernel > parameters: > > options lnet networks=o2ib(ib0) > options ko2iblnd map_on_demand=2 peer_credits=128 credits=256 > concurrent_sends=256 ntx=512 fmr_pool_size=2048 fmr_flush_trigger=512 > fmr_cache=1
If you have peer_credits=128, then I would suggest increase credits=1024 and ntx=2048, otherwise a couple of clients could consume all NI credits, concurrent_sends is not necessary here because o2iblnd will estimate proper value for it. Thanks Liang _______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
