I see in the output mca:base:select(ess) selected component [pmi]
the last line in the output reads MCW rank 0 is not bound (or bound to all available processors) (sorry i can't cut and paste from the cluster to here) i recompiled slurm/openmpi with enable-debug, so i might be able to track down something with that if it helps On 12/16/15, Ralph Castain <[email protected]> wrote: > > Yes, mpirun adds that to the environment to ensure we don’t pickup the wrong > ess component. Try adding “OMPI_MCA_ess_base_verbose=10” to your environment > and just srun one copy of hello_world - let’s ensure it picked up the right > ess component. > > I can try to replicate here, but it will take me a little while to get to > it > >> On Dec 16, 2015, at 8:50 AM, Michael Di Domenico <[email protected]> >> wrote: >> >> >> Yes, i have PMI support included into openmpi >> >> --with-slurm --with-psm --with-pmi=/opt/slurm >> >> checking through the config.log it does appear the PMI tests build >> successfully. >> >> though checking with ompi_info i'm not sure i can say with 100% >> certainty it's in there. >> >> ompi_info --parseable | grep pmi does return >> >> mca:db:pmi >> mca:ess:pmi >> mca:grpcomm:pmi >> mca:pubsub:pmi >> >> interestingly enough when i run 'mpirun env' (no slurm), i see ^pmi in >> the OMPI environment variables, but i'm not sure if that's supposed to >> do that or not >> >> >> >> On 12/16/15, Ralph Castain <[email protected]> wrote: >>> >>> Hey Michael >>> >>> Check ompi_info and ensure that the PMI support built - you have to >>> explicitly ask for it and provide the path to pmi.h >>> >>> >>>> On Dec 16, 2015, at 6:48 AM, Michael Di Domenico >>>> <[email protected]> >>>> wrote: >>>> >>>> >>>> i just compiled and installed Slurm 14.11.4 and Openmpi 1.10.0. but i >>>> seem to have an srun oddity i've not seen before and i'm not exactly >>>> sure how to debug it >>>> >>>> srun -n 4 hello_world >>>> - does not run, hangs in MPI_INIT >>>> >>>> srun -n 4 -N1 hello_world >>>> - does not run, hangs in MPI_INIT >>>> >>>> srun -n 4 -N 4 >>>> - runs one task per node >>>> >>>> sbatch and salloc seem to work okay launching using mpirun inside, and >>>> mpirun works without issue outside of slurm >>>> >>>> i disabled all the gres and cgroup controls and all that >>>> >>>> has anyone seen this before? >>> >
