Hi Aprea, MVAPICH2 should support multiple IB interfaces as default. As I don't have experience on single image systems (ScaleMP?), I am not really sure will MVAPHICH2 works as normal.
Otherwise, you might need to explicitly change your code to take advantage of multiple physical IB connections. BTW: as 8b-10b encoding method is used for PCI-e 2.0, you would only be able to see 16Gb per IB connection, and if you use 4 connections at same time, you would not be able to see 4*16Gb bandwidth due to hardware and software contentions. You might be interested in this: http://ieeexplore.ieee.org/search/srchabstract.jsp?tp=&arnumber=5328503&queryText%3D.QT.Non-threaded+and+Threaded+Approaches+to+Multirail+Communication+with+uDAPL.QT.%26openedRefinements%3D*%26searchField%3DSearch+All Hope this helps. Kind Regards, Jie -- http://cs.anu.edu.au/~Jie.Cai Giuseppe Aprea wrote:
Hi list, I am dealing with a set of 4 nodes (ibm x3850 ) binded together using ScaleXpander chip. Each node is equipped with a 20 Gb DDR infiniband network card so the new single image multi-node has 4 cards. We are using mellanox ofed to handle infiniband connectivity. Our problem now is how to exploit all 4 cards together. It seems that ofed "ib-bonding" can only be used for a kind of fail-over configuration of the cards and we wonder if it is possible to set up automatic bandwidths sum, i.e. if this node is included in a hostfile for an mpi-job together with other nodes it should be able to use the whole of its potential 20*4 Mb bandwidth, (possibly) without changing job submission parameters. I googled around unsuccessfully quite a lot but I may have missed the right keywords to find what I need. Any help will be much appreciated. Thanks in advance. Dr G. Aprea -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
-- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
