Re: [OMPI devel] open ib btl and xrc

2008-01-20 Thread Gleb Natapov
On Fri, Jan 18, 2008 at 11:43:03AM -0500, Jeff Squyres wrote: > I think the main savings is that mellanox hardware works better when > fewer qp's are open. I.e., it's a resource issue on the HCA, not > necessarily a savings in posting buffers to the qp. Interesting. I hear this justification

Re: [OMPI devel] open ib btl and xrc

2008-01-20 Thread Pavel Shamis (Pasha)
It easy to see the benefit of fewer qps (per node instead of per peer) and less consumption of resources the better but I am curious about the actual percentage of memory footprint decrease. I am thinking that the largest portion of the footprint comes from the fragments. BTW here is link to

Re: [OMPI devel] open ib btl and xrc

2008-01-18 Thread Jeff Squyres
I think the main savings is that mellanox hardware works better when fewer qp's are open. I.e., it's a resource issue on the HCA, not necessarily a savings in posting buffers to the qp. But it's quite a complicated issue. :-) Gleb has some reservations about XRC; I'll let him expound on

Re: [OMPI devel] open ib btl and xrc

2008-01-18 Thread Don Kerr
Those pointers were perfect thanks. It easy to see the benefit of fewer qps (per node instead of per peer) and less consumption of resources the better but I am curious about the actual percentage of memory footprint decrease. I am thinking that the largest portion of the footprint comes from

Re: [OMPI devel] open ib btl and xrc

2008-01-17 Thread Pavel Shamis (Pasha)
Here is paper from openib http://www.openib.org/archives/nov2007sc/XRC.pdf and here is mvapich presentation http://mvapich.cse.ohio-state.edu/publications/ofa_nov07-mvapich-xrc.pdf Button line: XRC decrease number of QPs that ompi opens and as result decrease ompi's memory footprint. In the

[OMPI devel] open ib btl and xrc

2008-01-17 Thread Don Kerr
Hi, After searching, about the only thing I can find on xrc is what it stands for, can someone explain the benefits of open mpi's use of xrc, maybe point me to a paper, or both? TIA -DON