On Thu, Jan 17, 2013 at 7:00 PM, Atchley, Scott <[email protected]> wrote: > On Jan 17, 2013, at 9:48 AM, Gandalf Corvotempesta > <[email protected]> wrote: > >> 2013/1/17 Atchley, Scott <[email protected]>: >>> IB DDR should get you close to 2 GB/s with IPoIB. I have gotten our IB QDR >>> PCI-E Gen. 2 up to 2.8 GB/s measured via netperf with lots of tuning. Since >>> it uses the traditional socket stack through the kernel, CPU usage will be >>> as high (or higher if QDR) than 10GbE. >> >> Which kind of tuning? Do you have a paper about this? > > No, I followed the Mellanox tuning guide and modified their interrupt > affinity scripts.
Did you tried to bind interrupts only to core to which QPI link belongs in reality and measure difference with spread-over-all-cores binding? > >> But, actually, is possible to use ceph with IPoIB in a stable way or >> is this experimental ? > > IPoIB appears as a traditional Ethernet device to Linux and can be used as > such. Not exactly, this summer kernel added additional driver for fully featured L2(ib ethernet driver), before that it was quite painful to do any possible failover using ipoib. > >> I don't know if i support for rsocket that is experimental/untested >> and IPoIB is a stable workaroud or what else. > > IPoIB is much more used and pretty stable, while rsockets is new with limited > testing. That said, more people using it will help Sean improve it. > > Ideally, we would like support for zero-copy and reduced CPU usage (via > OS-bypass) and with more interconnects than just InfiniBand. :-) > >> And is a dual controller needed on each OSD node? Ceph is able to >> handle OSD network failures? This is really important to know. It >> change the whole network topology. > > I will let others answer this. > > Scott-- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to [email protected] > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
