Yes, mpirun adds that to the environment to ensure we don’t pickup the wrong ess component. Try adding “OMPI_MCA_ess_base_verbose=10” to your environment and just srun one copy of hello_world - let’s ensure it picked up the right ess component.
I can try to replicate here, but it will take me a little while to get to it > On Dec 16, 2015, at 8:50 AM, Michael Di Domenico <[email protected]> > wrote: > > > Yes, i have PMI support included into openmpi > > --with-slurm --with-psm --with-pmi=/opt/slurm > > checking through the config.log it does appear the PMI tests build > successfully. > > though checking with ompi_info i'm not sure i can say with 100% > certainty it's in there. > > ompi_info --parseable | grep pmi does return > > mca:db:pmi > mca:ess:pmi > mca:grpcomm:pmi > mca:pubsub:pmi > > interestingly enough when i run 'mpirun env' (no slurm), i see ^pmi in > the OMPI environment variables, but i'm not sure if that's supposed to > do that or not > > > > On 12/16/15, Ralph Castain <[email protected]> wrote: >> >> Hey Michael >> >> Check ompi_info and ensure that the PMI support built - you have to >> explicitly ask for it and provide the path to pmi.h >> >> >>> On Dec 16, 2015, at 6:48 AM, Michael Di Domenico <[email protected]> >>> wrote: >>> >>> >>> i just compiled and installed Slurm 14.11.4 and Openmpi 1.10.0. but i >>> seem to have an srun oddity i've not seen before and i'm not exactly >>> sure how to debug it >>> >>> srun -n 4 hello_world >>> - does not run, hangs in MPI_INIT >>> >>> srun -n 4 -N1 hello_world >>> - does not run, hangs in MPI_INIT >>> >>> srun -n 4 -N 4 >>> - runs one task per node >>> >>> sbatch and salloc seem to work okay launching using mpirun inside, and >>> mpirun works without issue outside of slurm >>> >>> i disabled all the gres and cgroup controls and all that >>> >>> has anyone seen this before? >>
