Adding OMPI_MCA_mtl=^psm
to my environment and re-running the 'srun -n4 hello_world' seems to fix the issue so i guess we've isolated the problem to slurm/srun and psm, but now the question is what's broke On 12/16/15, Michael Di Domenico <[email protected]> wrote: > > to add some additional info, i let it sit for a long time and finally got > > PSM returned unhandled/unknown connect error: Operation timed out > PSM EP connect error (uknown connect error) > > so perhaps my old friend psm and srun aren't getting along again... > > > > On 12/16/15, Michael Di Domenico <[email protected]> wrote: >> I see in the output >> >> mca:base:select(ess) selected component [pmi] >> >> the last line in the output reads >> >> MCW rank 0 is not bound (or bound to all available processors) >> >> (sorry i can't cut and paste from the cluster to here) >> >> i recompiled slurm/openmpi with enable-debug, so i might be able to >> track down something with that if it helps >> >> >> >> On 12/16/15, Ralph Castain <[email protected]> wrote: >>> >>> Yes, mpirun adds that to the environment to ensure we don’t pickup the >>> wrong >>> ess component. Try adding “OMPI_MCA_ess_base_verbose=10” to your >>> environment >>> and just srun one copy of hello_world - let’s ensure it picked up the >>> right >>> ess component. >>> >>> I can try to replicate here, but it will take me a little while to get >>> to >>> it >>> >>>> On Dec 16, 2015, at 8:50 AM, Michael Di Domenico >>>> <[email protected]> >>>> wrote: >>>> >>>> >>>> Yes, i have PMI support included into openmpi >>>> >>>> --with-slurm --with-psm --with-pmi=/opt/slurm >>>> >>>> checking through the config.log it does appear the PMI tests build >>>> successfully. >>>> >>>> though checking with ompi_info i'm not sure i can say with 100% >>>> certainty it's in there. >>>> >>>> ompi_info --parseable | grep pmi does return >>>> >>>> mca:db:pmi >>>> mca:ess:pmi >>>> mca:grpcomm:pmi >>>> mca:pubsub:pmi >>>> >>>> interestingly enough when i run 'mpirun env' (no slurm), i see ^pmi in >>>> the OMPI environment variables, but i'm not sure if that's supposed to >>>> do that or not >>>> >>>> >>>> >>>> On 12/16/15, Ralph Castain <[email protected]> wrote: >>>>> >>>>> Hey Michael >>>>> >>>>> Check ompi_info and ensure that the PMI support built - you have to >>>>> explicitly ask for it and provide the path to pmi.h >>>>> >>>>> >>>>>> On Dec 16, 2015, at 6:48 AM, Michael Di Domenico >>>>>> <[email protected]> >>>>>> wrote: >>>>>> >>>>>> >>>>>> i just compiled and installed Slurm 14.11.4 and Openmpi 1.10.0. but >>>>>> i >>>>>> seem to have an srun oddity i've not seen before and i'm not exactly >>>>>> sure how to debug it >>>>>> >>>>>> srun -n 4 hello_world >>>>>> - does not run, hangs in MPI_INIT >>>>>> >>>>>> srun -n 4 -N1 hello_world >>>>>> - does not run, hangs in MPI_INIT >>>>>> >>>>>> srun -n 4 -N 4 >>>>>> - runs one task per node >>>>>> >>>>>> sbatch and salloc seem to work okay launching using mpirun inside, >>>>>> and >>>>>> mpirun works without issue outside of slurm >>>>>> >>>>>> i disabled all the gres and cgroup controls and all that >>>>>> >>>>>> has anyone seen this before? >>>>> >>> >>
