>Hi all, > I'm using libibverbs to build a cluster memory pool, and >using TCP/IP >handshake to exchange memory information and establish the connection >before the IB communication. While I found this process costed a lot >of time, 100ms in 1GEth LAN, so I want to use the rdma_cm or ib_ucm to >handle the establishment. But I dont't find sample code or API >document, is there anything I missed? > BTW, how to establish communication in current OFED? Any >comparision >or suggestion is appreciated, that will help me a lot. >
What scale are you targeting? Your single connection number seems high. For a connection (socket connect, exchanging QP info, private data, qp modify) using uDAPL socket cm versus rdma_cm I get: socket_cm on 1Ge == ~900us socket_cm on IPoIB (mlx4 ddr) == ~400us rdma_cm on IB (mlx4 ddr) == ~2200us As you can see, the path record queries via rdma_cm add a substantial penalty. With larger scale clusters this really starts to hurt. You can look at uDAPL (dapl/openib_cma and dapl/openib_scm) source for examples of a socket cm implementation vs rdma_cm. With the socket cm version we ran up to 14,400 cores with no problems using Intel MPI. However, with rdma_cm we had problems reaching 1000 cores due to IPoIB ARP storms and SA path record query issues. If someone would step up and provide a scalable SA caching solution in OFED then rdma_cm could possibly work for us again. Any takers? :^) -arlin _______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
