On Fri, Jan 18, 2008 at 11:43:03AM -0500, Jeff Squyres wrote: > I think the main savings is that mellanox hardware works better when > fewer qp's are open. I.e., it's a resource issue on the HCA, not > necessarily a savings in posting buffers to the qp. Interesting. I hear this justification of XRC for the first time. It's always was about decreasing memory consumption. As far as I know from the tests we ran here QP cache on Mellanox HCA is small 10-12 QPs, so I doubt XRC will help here, but maybe there is another threshold after which performance drops even more.
> > But it's quite a complicated issue. :-) > > Gleb has some reservations about XRC; I'll let him expound on them... My current "reservations" are not about XRC per se, but about how OFED became to be a platform for Mellanox to push things to the world without any serious reviews. I don't really care about 10000 things that goes into OFED without going into the linux kernel first as long as they are not change/define interfaces. Upcoming OFED 1.3 will include XRC interface without any feedback from linux kernel developers. What if interface will have to be changed in order to be included into the linux kernel? Do you remember why PLPA exists? Because some distribution hurried to include process affinity before interface was finalized. Same thing are happening here. But this discussion is not for this list :) > > > > On Jan 18, 2008, at 1:06 AM, Don Kerr wrote: > > > Those pointers were perfect thanks. > > > > It easy to see the benefit of fewer qps (per node instead of per peer) > > and less consumption of resources the better but I am curious about > > the > > actual percentage of memory footprint decrease. I am thinking that the > > largest portion of the footprint comes from the fragments. Do you have > > any numbers showing the actual memory footprint savings when using > > xrc? > > Just to be clear, I am not asking for you or anyone else to generate > > these numbers, but if you had them already I would be curious to know > > the over all savings. > > > > -DON > > > > Pavel Shamis (Pasha) wrote: > >> Here is paper from openib http://www.openib.org/archives/nov2007sc/XRC.pdf > >> and here is mvapich presentation > >> http://mvapich.cse.ohio-state.edu/publications/ofa_nov07-mvapich-xrc.pdf > >> > >> Button line: XRC decrease number of QPs that ompi opens and as result > >> decrease ompi's memory footprint. > >> In the openib paper you may see more details about XRC. If you need > >> more > >> details about XRC implemention > >> in openib blt , please let me know. > >> > >> > >> Instead > >> Don Kerr wrote: > >> > >>> Hi, > >>> > >>> After searching, about the only thing I can find on xrc is what it > >>> stands for, can someone explain the benefits of open mpi's use of > >>> xrc, > >>> maybe point me to a paper, or both? > >>> > >>> TIA > >>> -DON > >>> > >>> _______________________________________________ > >>> devel mailing list > >>> de...@open-mpi.org > >>> http://www.open-mpi.org/mailman/listinfo.cgi/devel > >>> > >>> > >>> > >> > >> > >> > > _______________________________________________ > > devel mailing list > > de...@open-mpi.org > > http://www.open-mpi.org/mailman/listinfo.cgi/devel > > > -- > Jeff Squyres > Cisco Systems > > _______________________________________________ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel -- Gleb.
