Re: [OMPI devel] open ib btl and xrc
On Fri, Jan 18, 2008 at 11:43:03AM -0500, Jeff Squyres wrote: > I think the main savings is that mellanox hardware works better when > fewer qp's are open. I.e., it's a resource issue on the HCA, not > necessarily a savings in posting buffers to the qp. Interesting. I hear this justification of XRC for the first time. It's always was about decreasing memory consumption. As far as I know from the tests we ran here QP cache on Mellanox HCA is small 10-12 QPs, so I doubt XRC will help here, but maybe there is another threshold after which performance drops even more. > > But it's quite a complicated issue. :-) > > Gleb has some reservations about XRC; I'll let him expound on them... My current "reservations" are not about XRC per se, but about how OFED became to be a platform for Mellanox to push things to the world without any serious reviews. I don't really care about 1 things that goes into OFED without going into the linux kernel first as long as they are not change/define interfaces. Upcoming OFED 1.3 will include XRC interface without any feedback from linux kernel developers. What if interface will have to be changed in order to be included into the linux kernel? Do you remember why PLPA exists? Because some distribution hurried to include process affinity before interface was finalized. Same thing are happening here. But this discussion is not for this list :) > > > > On Jan 18, 2008, at 1:06 AM, Don Kerr wrote: > > > Those pointers were perfect thanks. > > > > It easy to see the benefit of fewer qps (per node instead of per peer) > > and less consumption of resources the better but I am curious about > > the > > actual percentage of memory footprint decrease. I am thinking that the > > largest portion of the footprint comes from the fragments. Do you have > > any numbers showing the actual memory footprint savings when using > > xrc? > > Just to be clear, I am not asking for you or anyone else to generate > > these numbers, but if you had them already I would be curious to know > > the over all savings. > > > > -DON > > > > Pavel Shamis (Pasha) wrote: > >> Here is paper from openib http://www.openib.org/archives/nov2007sc/XRC.pdf > >> and here is mvapich presentation > >> http://mvapich.cse.ohio-state.edu/publications/ofa_nov07-mvapich-xrc.pdf > >> > >> Button line: XRC decrease number of QPs that ompi opens and as result > >> decrease ompi's memory footprint. > >> In the openib paper you may see more details about XRC. If you need > >> more > >> details about XRC implemention > >> in openib blt , please let me know. > >> > >> > >> Instead > >> Don Kerr wrote: > >> > >>> Hi, > >>> > >>> After searching, about the only thing I can find on xrc is what it > >>> stands for, can someone explain the benefits of open mpi's use of > >>> xrc, > >>> maybe point me to a paper, or both? > >>> > >>> TIA > >>> -DON > >>> > >>> ___ > >>> devel mailing list > >>> de...@open-mpi.org > >>> http://www.open-mpi.org/mailman/listinfo.cgi/devel > >>> > >>> > >>> > >> > >> > >> > > ___ > > devel mailing list > > de...@open-mpi.org > > http://www.open-mpi.org/mailman/listinfo.cgi/devel > > > -- > Jeff Squyres > Cisco Systems > > ___ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel -- Gleb.
Re: [OMPI devel] open ib btl and xrc
It easy to see the benefit of fewer qps (per node instead of per peer) and less consumption of resources the better but I am curious about the actual percentage of memory footprint decrease. I am thinking that the largest portion of the footprint comes from the fragments. BTW here is link to another paper http://www.cs.sandia.gov/~rbbrigh/papers/ompi-ib-pvmmpi07.pdf that talks about more efficient usage of receive buffers. Do you have any numbers showing the actual memory footprint savings when using xrc? I don't have. Pasha. -DON Pavel Shamis (Pasha) wrote: Here is paper from openib http://www.openib.org/archives/nov2007sc/XRC.pdf and here is mvapich presentation http://mvapich.cse.ohio-state.edu/publications/ofa_nov07-mvapich-xrc.pdf Button line: XRC decrease number of QPs that ompi opens and as result decrease ompi's memory footprint. In the openib paper you may see more details about XRC. If you need more details about XRC implemention in openib blt , please let me know. Instead Don Kerr wrote: Hi, After searching, about the only thing I can find on xrc is what it stands for, can someone explain the benefits of open mpi's use of xrc, maybe point me to a paper, or both? TIA -DON ___ devel mailing list de...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/devel -- Pavel Shamis (Pasha) Mellanox Technologies
Re: [OMPI devel] open ib btl and xrc
I think the main savings is that mellanox hardware works better when fewer qp's are open. I.e., it's a resource issue on the HCA, not necessarily a savings in posting buffers to the qp. But it's quite a complicated issue. :-) Gleb has some reservations about XRC; I'll let him expound on them... On Jan 18, 2008, at 1:06 AM, Don Kerr wrote: Those pointers were perfect thanks. It easy to see the benefit of fewer qps (per node instead of per peer) and less consumption of resources the better but I am curious about the actual percentage of memory footprint decrease. I am thinking that the largest portion of the footprint comes from the fragments. Do you have any numbers showing the actual memory footprint savings when using xrc? Just to be clear, I am not asking for you or anyone else to generate these numbers, but if you had them already I would be curious to know the over all savings. -DON Pavel Shamis (Pasha) wrote: Here is paper from openib http://www.openib.org/archives/nov2007sc/XRC.pdf and here is mvapich presentation http://mvapich.cse.ohio-state.edu/publications/ofa_nov07-mvapich-xrc.pdf Button line: XRC decrease number of QPs that ompi opens and as result decrease ompi's memory footprint. In the openib paper you may see more details about XRC. If you need more details about XRC implemention in openib blt , please let me know. Instead Don Kerr wrote: Hi, After searching, about the only thing I can find on xrc is what it stands for, can someone explain the benefits of open mpi's use of xrc, maybe point me to a paper, or both? TIA -DON ___ devel mailing list de...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/devel ___ devel mailing list de...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/devel -- Jeff Squyres Cisco Systems
Re: [OMPI devel] open ib btl and xrc
Those pointers were perfect thanks. It easy to see the benefit of fewer qps (per node instead of per peer) and less consumption of resources the better but I am curious about the actual percentage of memory footprint decrease. I am thinking that the largest portion of the footprint comes from the fragments. Do you have any numbers showing the actual memory footprint savings when using xrc? Just to be clear, I am not asking for you or anyone else to generate these numbers, but if you had them already I would be curious to know the over all savings. -DON Pavel Shamis (Pasha) wrote: Here is paper from openib http://www.openib.org/archives/nov2007sc/XRC.pdf and here is mvapich presentation http://mvapich.cse.ohio-state.edu/publications/ofa_nov07-mvapich-xrc.pdf Button line: XRC decrease number of QPs that ompi opens and as result decrease ompi's memory footprint. In the openib paper you may see more details about XRC. If you need more details about XRC implemention in openib blt , please let me know. Instead Don Kerr wrote: Hi, After searching, about the only thing I can find on xrc is what it stands for, can someone explain the benefits of open mpi's use of xrc, maybe point me to a paper, or both? TIA -DON ___ devel mailing list de...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/devel
Re: [OMPI devel] open ib btl and xrc
Here is paper from openib http://www.openib.org/archives/nov2007sc/XRC.pdf and here is mvapich presentation http://mvapich.cse.ohio-state.edu/publications/ofa_nov07-mvapich-xrc.pdf Button line: XRC decrease number of QPs that ompi opens and as result decrease ompi's memory footprint. In the openib paper you may see more details about XRC. If you need more details about XRC implemention in openib blt , please let me know. Instead Don Kerr wrote: Hi, After searching, about the only thing I can find on xrc is what it stands for, can someone explain the benefits of open mpi's use of xrc, maybe point me to a paper, or both? TIA -DON ___ devel mailing list de...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/devel -- Pavel Shamis (Pasha) Mellanox Technologies
[OMPI devel] open ib btl and xrc
Hi, After searching, about the only thing I can find on xrc is what it stands for, can someone explain the benefits of open mpi's use of xrc, maybe point me to a paper, or both? TIA -DON
