Re: [OMPI users] Problems with mpirun in openmpi-1.8.1 and -2.0.0

Juan A. Cordero Varelaq Mon, 22 Aug 2016 04:49:18 -0700

Hi Gilles,

adding *,usnic* made it work :) --mca pml ob1 would not be then needed.

Does it render mpi very slow if infiniband is disabled (what does --mcapml pb1?)?

Regarding the version mismatch, everything seems to be right. When onlyone version is loaded, I see the PATH and the LD_LIBRARY_PATH for onlyone version, and with strings, everything seems to reference the rightversion.


Thanks a lot for the quick answers!
On 22/08/16 13:09, Gilles Gouaillardet wrote:

Juan,

can you try to
mpirun --mca btl ^openib,usnic --mca pml ob1 ...

note this simply disable native infiniband. from a performance pointof view, you should have your sysadmin fix the infiniband fabric.


about the version mismatch, please double check your environment

(e.g. $PATH and $LD_LIBRARY_PATH), it is likely v2.0 is in yourenvironment when you are using v1.8, or the other way around)

also, make sure orted are launched with the right environment.
if you are using ssh, then
ssh node env
should not contain reference to the version you are not using.

(that typically occurs when the environment is set in the .bashrc,directly or via modules)


last but not least, you can
strings /.../bin/orted
strings /.../lib/libmpi.so
and check they do not reference the wrong version
(that can happen if a library was built and then moved)


Cheers,

Gilles

On Monday, August 22, 2016, Juan A. Cordero Varelaq<bioinformatica-i...@us.es <mailto:bioinformatica-i...@us.es>> wrote:


    Dear Ralph,

    The existence of the two versions does not seem to be the source
    of problems, since they are in different locations. I uninstalled
    the most recent version and try again with no luck, getting the
    same warnings/errors. However, after a deep search I found a
    couple of hints, and executed this:

    mpirun *-mca btl ^openib -mca btl_sm_use_knem 0* -np 5 myscript

    and got only a fraction of the previous errors (before I had run
    the same but without the arguments in bold), related to OpenFabrics:

    Open MPI failed to open an OpenFabrics device.  This is an unusual
    error; the system reported the OpenFabrics device as being present,
    but then later failed to access it successfully. This usually
    indicates either a misconfiguration or a failed OpenFabrics hardware
    device.

    All OpenFabrics support has been disabled in this MPI process; your
    job may or may not continue.

      Hostname:    MYMACHINE
      Device name: mlx4_0
      Errror (22): Invalid argument
    --------------------------------------------------------------------------
    --------------------------------------------------------------------------
    [[52062,1],0]: A high-performance Open MPI point-to-point
    messaging module
    was unable to find any relevant network interfaces:

    Module: usNIC
      Host: MYMACHINE

    Another transport will be used instead, although this may result in
    lower performance.
    --------------------------------------------------------------------------

    Do you guess why it could happen?


    Thanks a lot

    On 19/08/16 17:11, r...@open-mpi.org
    <javascript:_e(%7B%7D,'cvml','r...@open-mpi.org');> wrote:

    The rdma error sounds like something isn’t right with your
    machine’s Infiniband installation.

    The cross-version problem sounds like you installed both OMPI
    versions into the same location - did you do that?? If so, then
    that might be the root cause of both problems. You need to
    install them in totally different locations. Then you need to
    _prefix_ your PATH and LD_LIBRARY_PATH with the location of the
    version you want to use.

    HTH
    Ralph

    On Aug 19, 2016, at 12:53 AM, Juan A. Cordero Varelaq
    <bioinformatica-i...@us.es
    <javascript:_e(%7B%7D,'cvml','bioinformatica-i...@us.es');>> wrote:

    Dear users,

    I am totally stuck using openmpi. I have two versions on my
    machine: 1.8.1 and 2.0.0, and none of them work. When use the
    mpirun *1.8.1 version*, I get the following error:

    librdmacm: Fatal: unable to open RDMA device
    librdmacm: Fatal: unable to open RDMA device
    librdmacm: Fatal: unable to open RDMA device
    librdmacm: Fatal: unable to open RDMA device
    librdmacm: Fatal: unable to open RDMA device
    --------------------------------------------------------------------------
    Open MPI failed to open the /dev/knem device due to a local error.
    Please check with your system administrator to get the problem
    fixed,
    or set the btl_sm_use_knem MCA parameter to 0 to run without
    /dev/knem
    support.

      Local host: MYMACHINE
      Errno:      2 (No such file or directory)
    --------------------------------------------------------------------------
    --------------------------------------------------------------------------
    Open MPI failed to open an OpenFabrics device.  This is an unusual
    error; the system reported the OpenFabrics device as being present,
    but then later failed to access it successfully.  This usually
    indicates either a misconfiguration or a failed OpenFabrics hardware
    device.

    All OpenFabrics support has been disabled in this MPI process; your
    job may or may not continue.

      Hostname:    MYMACHINE
      Device name: mlx4_0
      Errror (22): Invalid argument
    --------------------------------------------------------------------------
    --------------------------------------------------------------------------
    [[60527,1],4]: A high-performance Open MPI point-to-point
    messaging module
    was unable to find any relevant network interfaces:

    Module: usNIC
      Host: MYMACHINE

    When I use the *2.0.0 version*, I get something strange, it
    seems openmpi-2.0.0 looks for openmpi-1.8.1 libraries?:

    A requested component was not found, or was unable to be
    opened.  This
    means that this component is either not installed or is unable to be
    used on your system (e.g., sometimes this means that shared
    libraries