Daniel, PMIX_MODEX and PMIX_INFO_ARRAY have been removed from PMIx 3.1.2, and Open MPI 4.0.0 was not ready for this.
You can either use the internal PMIx (3.0.2), or try 4.0.1rc1 (with the external PMIx 3.1.2) that was published a few days ago. FWIW, you are right using --with-pmix=external (and not using --with-pmix=/usr) Cheers, Gilles On Sun, Mar 3, 2019 at 10:57 PM Daniel Letai <d...@letai.org.il> wrote: > > Hello, > > > I have built the following stack : > > centos 7.5 (gcc 4.8.5-28, libevent 2.0.21-4) > MLNX_OFED_LINUX-4.5-1.0.1.0-rhel7.5-x86_64.tgz built with --all > --without-32bit (this includes ucx 1.5.0) > hwloc from centos 7.5 : 1.11.8-4.el7 > pmix 3.1.2 > slurm 18.08.5-2 built --with-ucx --with-pmix > openmpi 4.0.0 : configure --with-slurm --with-pmix=external --with-pmi > --with-libevent=external --with-hwloc=external > --with-knem=/opt/knem-1.1.3.90mlnx1 --with-hcoll=/opt/mellanox/hcoll > > The configure part succeeds, however 'make' errors out with: > > ext3x.c: In function 'ext3x_value_unload': > > ext3x.c:1109:10: error: 'PMIX_MODEX' undeclared (first use in this function) > > > And same for 'PMIX_INFO_ARRAY' > > > However, both are declared in the > opal/mca/pmix/pmix3x/pmix/include/pmix_common.h file. > > opal/mca/pmix/ext3x/ext3x.c does include pmix_common.h but as a system > include #include <pmix_common> , while ext3x.h includes it as a local include > #include "pmix_common". Neither seem to pull from the correct path. > > > Regards, > > Dani_L. > > > On 2/24/19 3:09 AM, Gilles Gouaillardet wrote: > > Passant, > > you have to manually download and apply > https://github.com/pmix/pmix/commit/2e2f4445b45eac5a3fcbd409c81efe318876e659.patch > to PMIx 2.2.1 > that should likely fix your problem. > > As a side note, it is a bad practice to configure --with-FOO=/usr > since it might have some unexpected side effects. > Instead, you can replace > > configure --with-slurm --with-pmix=/usr --with-pmi=/usr --with-libevent=/usr > > with > > configure --with-slurm --with-pmix=external --with-pmi > --with-libevent=external > > to be on the safe side I also invite you to pass --with-hwloc=external > to the configure command line > > > Cheers, > > Gilles > > On Sun, Feb 24, 2019 at 1:54 AM Passant A. Hafez > <passant.ha...@kaust.edu.sa> wrote: > > Hello Gilles, > > Here are some details: > > Slurm 18.08.4 > > PMIx 2.2.1 (as shown in /usr/include/pmix_version.h) > > Libevent 2.0.21 > > srun --mpi=list > srun: MPI types are... > srun: none > srun: openmpi > srun: pmi2 > srun: pmix > srun: pmix_v2 > > Open MPI versions tested: 4.0.0 and 3.1.2 > > > For each installation to be mentioned a different MPI Hello World program was > compiled. > Jobs were submitted by sbatch, 2 node * 2 tasks per node then srun --mpi=pmix > program > > File 400ext_2x2.out (attached) is for OMPI 4.0.0 installation with configure > options: > --with-slurm --with-pmix=/usr --with-pmi=/usr --with-libevent=/usr > and configure log: > Libevent support: external > PMIx support: External (2x) > > File 400int_2x2.out (attached) is for OMPI 4.0.0 installation with configure > options: > --with-slurm --with-pmix > and configure log: > Libevent support: internal (external libevent version is less that internal > version 2.0.22) > PMIx support: Internal > > Tested also different installations for 3.1.2 and got errors similar to > 400ext_2x2.out > (NOT-SUPPORTED in file event/pmix_event_registration.c at line 101) > > > > > > All the best, > -- > Passant A. Hafez | HPC Applications Specialist > KAUST Supercomputing Core Laboratory (KSL) > King Abdullah University of Science and Technology > Building 1, Al-Khawarizmi, Room 0123 > Mobile : +966 (0) 55-247-9568 > Mobile : +20 (0) 106-146-9644 > Office : +966 (0) 12-808-0367 > > ________________________________________ > From: users <users-boun...@lists.open-mpi.org> on behalf of Gilles > Gouaillardet <gilles.gouaillar...@gmail.com> > Sent: Saturday, February 23, 2019 5:17 PM > To: Open MPI Users > Subject: Re: [OMPI users] Building PMIx and Slurm support > > Hi, > > PMIx has cross-version compatibility, so as long as the PMIx library > used by SLURM is compatible with the one (internal or external) used > by Open MPI, you should be fine. > If you want to minimize the risk of cross-version incompatibility, > then I encourage you to use the same (and hence external) PMIx that > was used to build SLURM with Open MPI. > > Can you tell a bit more than "it didn't work" ? > (Open MPI version, PMIx version used by SLURM, PMIx version used by > Open MPI, error message, ...) > > Cheers, > > Gilles > > On Sat, Feb 23, 2019 at 9:46 PM Passant A. Hafez > <passant.ha...@kaust.edu.sa> wrote: > > Good day everyone, > > I've trying to build and use the PMIx support for Open MPI but I tried many > things that I can list if needed, but with no luck. > I was able to test the PMIx client but when I used OMPI specifying srun > --mpi=pmix it didn't work. > > So if you please advise me with the versions of each PMIx and Open MPI that > should be working well with Slurm 18.08, it'd be great. > > Also, what is the difference between using internal vs external PMIx > installations? > > > > All the best, > > -- > > Passant A. Hafez | HPC Applications Specialist > KAUST Supercomputing Core Laboratory (KSL) > King Abdullah University of Science and Technology > Building 1, Al-Khawarizmi, Room 0123 > Mobile : +966 (0) 55-247-9568 > Mobile : +20 (0) 106-146-9644 > Office : +966 (0) 12-808-0367 > _______________________________________________ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users > > _______________________________________________ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users > _______________________________________________ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users > > _______________________________________________ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users > > _______________________________________________ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users _______________________________________________ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
