Re: [OMPI users] btl_openib_if_include

2018-04-23 Thread Marshall2, John (SSC/SPC) 
Hi,

That gives me an avenue to pursue.

Thanks,
John

On Mon, 2018-04-23 at 15:12 +, Jeff Squyres (jsquyres) wrote:

On Apr 23, 2018, at 11:00 AM, Marshall2, John (SSC/SPC) 
<john.marsha...@canada.ca<mailto:john.marsha...@canada.ca>> wrote:



Only one ib interface shows up via ifconfig and at /sys/class/net/ibX.

But, under /sys/class/infiniband and /sys/class/infiniband_cm, all the mlx4_Y 
do show
up. E.g.,
mlx4_0  mlx4_10  mlx4_12  mlx4_14  mlx4_16  mlx4_3  mlx4_5  mlx4_7  mlx4_9
mlx4_1  mlx4_11  mlx4_13  mlx4_15  mlx4_2   mlx4_4  mlx4_6  mlx4_8

I'm not sure if this can be avoided.

So, where is openmpi looking for the available mlx4_Y? Under one of those two 
directories
or whatever is at /sys/class/net/ibX/device/infiniband/mlx4_Y?



It will use whatever devices libibverbs reports back.

It's been quite a while since I've looked in the libibverbs code, but it 
*might* return all the devices...?  What does ibv_devinfo(1) return inside one 
of your containers?  That's probably the same information that is returned to 
Open MPI programmatically via the libibverbs API.

If libibverbs is returning all devices vs. just the one that is actually 
available in your container, then that might explain the performance disparity.


___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] btl_openib_if_include

2018-04-23 Thread Marshall2, John (SSC/SPC) 
On Mon, 2018-04-23 at 14:44 +, Jeff Squyres (jsquyres) wrote:

On Apr 20, 2018, at 1:03 PM, Marshall2, John (SSC/SPC) 
<john.marsha...@canada.ca<mailto:john.marsha...@canada.ca>> wrote:



I am trying to verify/determine what the proper setting is for 
btl_openib_ib_include.



I think you mean btl_openib_if_include ("if" = "interface").


Yes.





Some background:
* openmpi 2.1.1 (and 1.6.5 - yes it is old)
* lxc containers
* SRIOV (virtual functions) being used
* dedicated IB interface (e.g., ib2) per container

Should the mlx4_X:1 correspond to a specific ibY interface? E.g., for ib26, I 
find
mlx4_13:1 by:
$ ls /sys/class/net/ib26/device/infiniband
mlx4_13

Does the mlx4_X have to be determined at each location where an mpi task
would run? I suppose it would because the ibY is likely to be different.



Open MPI basically probes its environment at run time.  In your case, it will 
find all available IB interfaces (per MPI process), filter them through 
if_include / if_exclude, and then use whatever is left.



On some tests, I have found that the setting:
export OMPI_MCA_btl_openib_if_include=mlx4_0:1

provides better performance than not specifying a value or letting mpirun/orted
figure it out at runtime.



That's a little surprising.

Do you have more than 1 IB interface?  If not, then Open MPI should likely be 
independently coming to the same conclusion (i.e., "mlx4_0:1").  If it's not, 
that's weird.


Only one ib interface shows up via ifconfig and at /sys/class/net/ibX.

But, under /sys/class/infiniband and /sys/class/infiniband_cm, all the mlx4_Y 
do show
up. E.g.,

mlx4_0  mlx4_10  mlx4_12  mlx4_14  mlx4_16  mlx4_3  mlx4_5  mlx4_7  mlx4_9

mlx4_1  mlx4_11  mlx4_13  mlx4_15  mlx4_2   mlx4_4  mlx4_6  mlx4_8

I'm not sure if this can be avoided.

So, where is openmpi looking for the available mlx4_Y? Under one of those two 
directories
or whatever is at /sys/class/net/ibX/device/infiniband/mlx4_Y?

Thanks,
John






___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

[OMPI users] btl_openib_if_include

2018-04-20 Thread Marshall2, John (SSC/SPC) 
Hi

I am trying to verify/determine what the proper setting is for 
btl_openib_ib_include.

Some background:
* openmpi 2.1.1 (and 1.6.5 - yes it is old)
* lxc containers
* SRIOV (virtual functions) being used
* dedicated IB interface (e.g., ib2) per container

Should the mlx4_X:1 correspond to a specific ibY interface? E.g., for ib26, I 
find
mlx4_13:1 by:
$ ls /sys/class/net/ib26/device/infiniband
mlx4_13

Does the mlx4_X have to be determined at each location where an mpi task
would run? I suppose it would because the ibY is likely to be different.

On some tests, I have found that the setting:
export OMPI_MCA_btl_openib_if_include=mlx4_0:1

provides better performance than not specifying a value or letting mpirun/orted
figure it out at runtime.

Thanks,
John
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] pmix, lxc, hpcx

2017-05-29 Thread Marshall2, John (SSC/SPC) 
Hi,

Building with --disable-pmix-dstore, my program now works fine inside
and outside the containers. When/if I have the time, I will investigate
the "shared memory segments" aspects.

Thanks,
John

On Fri, 2017-05-26 at 15:23 -0700, r...@open-mpi.org wrote:
You can also get around it by configuring OMPI with “--disable-pmix-dstore”


On May 26, 2017, at 3:02 PM, Howard Pritchard 
> wrote:
Hi John,

In the 2.1.x release stream a shared memory capability was introduced into the 
PMIx component.

I know nothing about LXC containers, but it looks to me like there's some issue 
when PMIx tries
to create these shared memory segments.  I'd check to see if there's something 
about your
container configuration that is preventing the creation of shared memory 
segments.

Howard


2017-05-26 15:18 GMT-06:00 John Marshall 
>:
Hi,

I have built openmpi 2.1.1 with hpcx-1.8 and tried to run some mpi code under
ubuntu 14.04 and LXC (1.x) but I get the following:

[ib7-bc2oo42-be10p16.science.gc.ca:16035]
 PMIX ERROR: OUT-OF-RESOURCE in file src/dstore/pmix_esh.c at line 1651
[ib7-bc2oo42-be10p16.science.gc.ca:16035]
 PMIX ERROR: OUT-OF-RESOURCE in file src/dstore/pmix_esh.c at line 1751
[ib7-bc2oo42-be10p16.science.gc.ca:16035]
 PMIX ERROR: OUT-OF-RESOURCE in file src/dstore/pmix_esh.c at line 1114
[ib7-bc2oo42-be10p16.science.gc.ca:16035]
 PMIX ERROR: OUT-OF-RESOURCE in file src/common/pmix_jobdata.c at line 93
[ib7-bc2oo42-be10p16.science.gc.ca:16035]
 PMIX ERROR: OUT-OF-RESOURCE in file src/common/pmix_jobdata.c at line 333
[ib7-bc2oo42-be10p16.science.gc.ca:16035]
 PMIX ERROR: OUT-OF-RESOURCE in file src/server/pmix_server.c at line 606

I do not get the same outside of the LXC container and my code runs fine.

I've looked for more info on these messages but could not find anything
helpful. Are these messages indicative of something missing in, or some
incompatibility with, the container?

When I build using 2.0.2, I do not have a problem running inside or outside of
the container.

Thanks,
John

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users


___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users