On Oct 20, 2015, at 5:26 PM, Paul Hargrove <phhargr...@lbl.gov> wrote: > > I think heterogeneous multirail is still pretty uncommon. It might still be > ok to force users (or better yet, their admins -- via the global > mca-params.conf file) to use level 3 to precisely specify which network / > OMPI API to use (e.g., BTL, MTL, ...etc.). > > I think a reasonable fraction of IB-connected clusters also have an Ethernet > network plus have IPoIB enabled (thus two IP networks). > So, I don't agree that heterogenous multirail is "pretty uncommon".
(just addressing this one point) Sure, but in that case, it's pretty clear that OMPI should exclusively use some kinda of "native" IB API to use the IB -- not the Ethernet network, and not IPoIB. That's what OMPI has been doing for years. Perhaps I should clarify: the "heterogeneous multirail" I was referring to is when you have multiple high-speed/HPC-oriented networks. I'm not counting your scenario because 1G Ethernet is not generally used for MPI traffic unless that's your only network. IPoIB is not generally used for MPI traffic because it just adds latency and decreases bandwidth compared to native IB transports. -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
