On Apr 20, 2018, at 1:03 PM, Marshall2, John (SSC/SPC)
<john.marsha...@canada.ca> wrote:
>
> I am trying to verify/determine what the proper setting is for
> btl_openib_ib_include.
I think you mean btl_openib_if_include ("if" = "interface").
> Some background:
> * openmpi 2.1.1 (and 1.6.5 - yes it is old)
> * lxc containers
> * SRIOV (virtual functions) being used
> * dedicated IB interface (e.g., ib2) per container
>
> Should the mlx4_X:1 correspond to a specific ibY interface? E.g., for ib26, I
> find
> mlx4_13:1 by:
> $ ls /sys/class/net/ib26/device/infiniband
> mlx4_13
>
> Does the mlx4_X have to be determined at each location where an mpi task
> would run? I suppose it would because the ibY is likely to be different.
Open MPI basically probes its environment at run time. In your case, it will
find all available IB interfaces (per MPI process), filter them through
if_include / if_exclude, and then use whatever is left.
> On some tests, I have found that the setting:
> export OMPI_MCA_btl_openib_if_include=mlx4_0:1
>
> provides better performance than not specifying a value or letting
> mpirun/orted
> figure it out at runtime.
That's a little surprising.
Do you have more than 1 IB interface? If not, then Open MPI should likely be
independently coming to the same conclusion (i.e., "mlx4_0:1"). If it's not,
that's weird.
--
Jeff Squyres
jsquy...@cisco.com
_______________________________________________
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users
