Dear Open MPI users, in my institution a cluster with dual-rail IB has recently deployed. Each compute node has two physical single-port Mellanox Connect-IB MT27600 card (mlx5_0, mlx5_1). By running bandwidth tests (OSU 4.2 benchmark) using MVAPICH2, I can achieve from one node to another (1 MPI process per node) up to 12 GB/s using the rail in sharing by brokering small messages across both HCA devices. This is good.
The I switched to Open MPI (1.7.3 and 1.7.4rc1). I tried to use both HCAs together but it seems to me that only one is used (because there is only one process per node?). In Open MPI it seems more complicated to set up such a test. This is what I did... mpirun --mca coll_fca_enable 0 --mca btl_openib_verbose 1 -host HOST1,HOST2 --mca btl_openib_if_include mlx5_0,mlx5_1 -np 1 ./osu-bin/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw : -np 1 --mca coll_fca_enable 0 --mca btl_openib_verbose 1 --mca btl_openib_if_include mlx5_0,mlx5_1 ./osu-bin/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw Max measured bandwidth is around 6.5 GB/s, basically the same if I use a single HCA. What I do wrong? Is this the correct way to exploit a multi-rail system? Many thanks in advance, Regards -- Mr. Filippo SPIGA, M.Sc. http://www.linkedin.com/in/filippospiga ~ skype: filippo.spiga «Nobody will drive us out of Cantor's paradise.» ~ David Hilbert
