Dani,
We have had to specify the path to the external PMIx explicitly when
compiling both Slurm and OpenMPI; e.g.,
--with-pmix=/opt/pmix/3.1.2
That insures the both are referring to the same version.
-- bennet
On Sun, Mar 3, 2019 at 8:56 AM Daniel Letai <d...@letai.org.il> wrote:
>
> Hello,
>
>
> I have built the following stack :
>
> centos 7.5 (gcc 4.8.5-28, libevent 2.0.21-4)
> MLNX_OFED_LINUX-4.5-1.0.1.0-rhel7.5-x86_64.tgz built with --all
> --without-32bit (this includes ucx 1.5.0)
> hwloc from centos 7.5 : 1.11.8-4.el7
> pmix 3.1.2
> slurm 18.08.5-2 built --with-ucx --with-pmix
> openmpi 4.0.0 : configure --with-slurm --with-pmix=external --with-pmi
> --with-libevent=external --with-hwloc=external
> --with-knem=/opt/knem-1.1.3.90mlnx1 --with-hcoll=/opt/mellanox/hcoll
>
> The configure part succeeds, however 'make' errors out with:
>
> ext3x.c: In function 'ext3x_value_unload':
>
> ext3x.c:1109:10: error: 'PMIX_MODEX' undeclared (first use in this function)
>
>
> And same for 'PMIX_INFO_ARRAY'
>
>
> However, both are declared in the
> opal/mca/pmix/pmix3x/pmix/include/pmix_common.h file.
>
> opal/mca/pmix/ext3x/ext3x.c does include pmix_common.h but as a system
> include #include <pmix_common> , while ext3x.h includes it as a local include
> #include "pmix_common". Neither seem to pull from the correct path.
>
>
> Regards,
>
> Dani_L.
>
>
> On 2/24/19 3:09 AM, Gilles Gouaillardet wrote:
>
> Passant,
>
> you have to manually download and apply
> https://github.com/pmix/pmix/commit/2e2f4445b45eac5a3fcbd409c81efe318876e659.patch
> to PMIx 2.2.1
> that should likely fix your problem.
>
> As a side note, it is a bad practice to configure --with-FOO=/usr
> since it might have some unexpected side effects.
> Instead, you can replace
>
> configure --with-slurm --with-pmix=/usr --with-pmi=/usr --with-libevent=/usr
>
> with
>
> configure --with-slurm --with-pmix=external --with-pmi
> --with-libevent=external
>
> to be on the safe side I also invite you to pass --with-hwloc=external
> to the configure command line
>
>
> Cheers,
>
> Gilles
>
> On Sun, Feb 24, 2019 at 1:54 AM Passant A. Hafez
> <passant.ha...@kaust.edu.sa> wrote:
>
> Hello Gilles,
>
> Here are some details:
>
> Slurm 18.08.4
>
> PMIx 2.2.1 (as shown in /usr/include/pmix_version.h)
>
> Libevent 2.0.21
>
> srun --mpi=list
> srun: MPI types are...
> srun: none
> srun: openmpi
> srun: pmi2
> srun: pmix
> srun: pmix_v2
>
> Open MPI versions tested: 4.0.0 and 3.1.2
>
>
> For each installation to be mentioned a different MPI Hello World program was
> compiled.
> Jobs were submitted by sbatch, 2 node * 2 tasks per node then srun --mpi=pmix
> program
>
> File 400ext_2x2.out (attached) is for OMPI 4.0.0 installation with configure
> options:
> --with-slurm --with-pmix=/usr --with-pmi=/usr --with-libevent=/usr
> and configure log:
> Libevent support: external
> PMIx support: External (2x)
>
> File 400int_2x2.out (attached) is for OMPI 4.0.0 installation with configure
> options:
> --with-slurm --with-pmix
> and configure log:
> Libevent support: internal (external libevent version is less that internal
> version 2.0.22)
> PMIx support: Internal
>
> Tested also different installations for 3.1.2 and got errors similar to
> 400ext_2x2.out
> (NOT-SUPPORTED in file event/pmix_event_registration.c at line 101)
>
>
>
>
>
> All the best,
> --
> Passant A. Hafez | HPC Applications Specialist
> KAUST Supercomputing Core Laboratory (KSL)
> King Abdullah University of Science and Technology
> Building 1, Al-Khawarizmi, Room 0123
> Mobile : +966 (0) 55-247-9568
> Mobile : +20 (0) 106-146-9644
> Office : +966 (0) 12-808-0367
>
> ________________________________________
> From: users <users-boun...@lists.open-mpi.org> on behalf of Gilles
> Gouaillardet <gilles.gouaillar...@gmail.com>
> Sent: Saturday, February 23, 2019 5:17 PM
> To: Open MPI Users
> Subject: Re: [OMPI users] Building PMIx and Slurm support
>
> Hi,
>
> PMIx has cross-version compatibility, so as long as the PMIx library
> used by SLURM is compatible with the one (internal or external) used
> by Open MPI, you should be fine.
> If you want to minimize the risk of cross-version incompatibility,
> then I encourage you to use the same (and hence external) PMIx that
> was used to build SLURM with Open MPI.
>
> Can you tell a bit more than "it didn't work" ?
> (Open MPI version, PMIx version used by SLURM, PMIx version used by
> Open MPI, error message, ...)
>
> Cheers,
>
> Gilles
>
> On Sat, Feb 23, 2019 at 9:46 PM Passant A. Hafez
> <passant.ha...@kaust.edu.sa> wrote:
>
> Good day everyone,
>
> I've trying to build and use the PMIx support for Open MPI but I tried many
> things that I can list if needed, but with no luck.
> I was able to test the PMIx client but when I used OMPI specifying srun
> --mpi=pmix it didn't work.
>
> So if you please advise me with the versions of each PMIx and Open MPI that
> should be working well with Slurm 18.08, it'd be great.
>
> Also, what is the difference between using internal vs external PMIx
> installations?
>
>
>
> All the best,
>
> --
>
> Passant A. Hafez | HPC Applications Specialist
> KAUST Supercomputing Core Laboratory (KSL)
> King Abdullah University of Science and Technology
> Building 1, Al-Khawarizmi, Room 0123
> Mobile : +966 (0) 55-247-9568
> Mobile : +20 (0) 106-146-9644
> Office : +966 (0) 12-808-0367
> _______________________________________________
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
>
> _______________________________________________
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
> _______________________________________________
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
>
> _______________________________________________
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
>
> _______________________________________________
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
_______________________________________________
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users
