to add some additional info, i let it sit for a long time and finally got PSM returned unhandled/unknown connect error: Operation timed out PSM EP connect error (uknown connect error)
so perhaps my old friend psm and srun aren't getting along again... On 12/16/15, Michael Di Domenico <[email protected]> wrote: > I see in the output > > mca:base:select(ess) selected component [pmi] > > the last line in the output reads > > MCW rank 0 is not bound (or bound to all available processors) > > (sorry i can't cut and paste from the cluster to here) > > i recompiled slurm/openmpi with enable-debug, so i might be able to > track down something with that if it helps > > > > On 12/16/15, Ralph Castain <[email protected]> wrote: >> >> Yes, mpirun adds that to the environment to ensure we don’t pickup the >> wrong >> ess component. Try adding “OMPI_MCA_ess_base_verbose=10” to your >> environment >> and just srun one copy of hello_world - let’s ensure it picked up the >> right >> ess component. >> >> I can try to replicate here, but it will take me a little while to get to >> it >> >>> On Dec 16, 2015, at 8:50 AM, Michael Di Domenico >>> <[email protected]> >>> wrote: >>> >>> >>> Yes, i have PMI support included into openmpi >>> >>> --with-slurm --with-psm --with-pmi=/opt/slurm >>> >>> checking through the config.log it does appear the PMI tests build >>> successfully. >>> >>> though checking with ompi_info i'm not sure i can say with 100% >>> certainty it's in there. >>> >>> ompi_info --parseable | grep pmi does return >>> >>> mca:db:pmi >>> mca:ess:pmi >>> mca:grpcomm:pmi >>> mca:pubsub:pmi >>> >>> interestingly enough when i run 'mpirun env' (no slurm), i see ^pmi in >>> the OMPI environment variables, but i'm not sure if that's supposed to >>> do that or not >>> >>> >>> >>> On 12/16/15, Ralph Castain <[email protected]> wrote: >>>> >>>> Hey Michael >>>> >>>> Check ompi_info and ensure that the PMI support built - you have to >>>> explicitly ask for it and provide the path to pmi.h >>>> >>>> >>>>> On Dec 16, 2015, at 6:48 AM, Michael Di Domenico >>>>> <[email protected]> >>>>> wrote: >>>>> >>>>> >>>>> i just compiled and installed Slurm 14.11.4 and Openmpi 1.10.0. but i >>>>> seem to have an srun oddity i've not seen before and i'm not exactly >>>>> sure how to debug it >>>>> >>>>> srun -n 4 hello_world >>>>> - does not run, hangs in MPI_INIT >>>>> >>>>> srun -n 4 -N1 hello_world >>>>> - does not run, hangs in MPI_INIT >>>>> >>>>> srun -n 4 -N 4 >>>>> - runs one task per node >>>>> >>>>> sbatch and salloc seem to work okay launching using mpirun inside, and >>>>> mpirun works without issue outside of slurm >>>>> >>>>> i disabled all the gres and cgroup controls and all that >>>>> >>>>> has anyone seen this before? >>>> >> >
