On Mar 27, 2014, at 11:15 AM, "Wang,Yanfei(SYS)" <wangyanfe...@baidu.com> wrote:
> Normally we use rdma-cm to build rdma connection ,then create Qpairs to do > rdma data transmit ion, so what is the consideration for separating rdma-cm > connection built and data transmit ion at design stage? There's some history here... Waaaay back in the day, the only way to make RC verbs connections over IB was to send QP numbers (and other info) out-of-band to a peer (e.g., via TCP sockets). OMPI implemented this method in the openib BTL. This had some scalability issues, though, so we eventually started experimenting with some other mechanisms for making RC QPs. For example, we tried using the IB connection manager for a while (IBCM), but that ultimately got dropped. The RDMA Connection Manager was always an option (RDMA CM), but we never bothered to implement it in OMPI until other technologies came along that *required* the use of the RDMA CM, namely: iWARP and RoCE. Meaning: you *can't* make RC QPs over iWARP and RoCE over the OOB method, nor can you use the IB CM -- you *have* to use the RDMA CM. RDMA CM has its own limitations, though. So for IB RC QPs -- where you don't *have* to use the RDMA CM -- we recently implemented the UDCM, which basically does the same thing as the initial OOB method, but in a more scalable and efficient fashion (I'm leaving out the details; let me know if you want to hear them). So at different times, we've had different numbers of mechanisms in OMPI for making these connections. In the v1.7/v1.8 tree, I believe that the only 2 left are the RDMA CM and the UDCM. I also believe that for iWARP and RoCE, the RDMA CM will be chosen automatically, and UD CM will be automatically chosen for IB. So after all that: I think you shouldn't need to specify the connection manager MCA parameter at all; the openib BTL should choose the Right one for you. -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
