The following (from what you posted earlier): $ srun --mpi=list srun: MPI types are... srun: none srun: pmix_v3 srun: pmi2 srun: openmpi srun: pmix
would indicate that Slurm was built against a PMIx v3.x release. Using OMPI v4.0.3 with pmix=internal should be just fine so long as you set --mpi=pmix_v3 I'm somewhat at a loss as to what might be wrong. Try adding "OMPI_MCA_pmix_base_verbose=5 PMIX_MCA_pmix_client_get_verbose=5" to your environment and see what it says. You also should build OMPI with --enable-debug to ensure you get all the available debug output. On May 6, 2020, at 1:26 PM, Prentice Bisbal via users <users@lists.open-mpi.org <mailto:users@lists.open-mpi.org> > wrote: No, and I fear that may be the problem. When we built OpenMPI, we did --with-pmix=internal. Not sure how Slurm was built, since my coworker built it. Prentice On 4/28/20 2:07 AM, Daniel Letai via users wrote: I know it's not supposed to matter, but have you tried building both ompi and slurm against same pmix? That is - first build pmix, than build slurm with-pmix, and than ompi with both slurm and pmix=external ? On 23/04/2020 17:00, Prentice Bisbal via users wrote: <snip> $ ompi_info | grep slurm Configure command line: '--prefix=/usr/pppl/intel/2019-pkgs/openmpi-4.0.3' '--disable-silent-rules' '--enable-shared' '--with-pmix=internal' '--with-slurm' '--with-psm' MCA ess: slurm (MCA v2.1.0, API v3.0.0, Component v4.0.3) MCA plm: slurm (MCA v2.1.0, API v2.0.0, Component v4.0.3) MCA ras: slurm (MCA v2.1.0, API v2.0.0, Component v4.0.3) MCA schizo: slurm (MCA v2.1.0, API v1.0.0, Component v4.0.3) Any ideas what could be wrong? Do you need any additional information? Prentice -- Prentice Bisbal Lead Software Engineer Research Computing Princeton Plasma Physics Laboratory http://www.pppl.gov
