Thanks for the reply (and for a great piece of software in SLURM). I
will check the debug info and see if I can learn anything. After some
additional info from the MPICH development team it seems likely (still don't
know for sure) that the root cause is somewhere in the interconnectivity of the
nodes and most likely not in the SLURM interface. I'll keep investigating and
report any relevant findings.
Thanks,
~Mike C.
-----Original Message-----
From: Hongjia Cao [mailto:[email protected]]
Sent: Saturday, January 12, 2013 5:30 AM
To: slurm-dev
Subject: [slurm-dev] RE: Fatal error in PMPI_Reduce
Seems that the error is not caused by the SLURM PMI implementation. You can set
the environment variable "PMI_DEBUG=1" before running the program to check the
PMI call sequences(remember to set the "-l" option of srun). If the version of
SLURM is higher than 2.4, you can also try building MPICH2 without the
configuration option "--with-pmi=slurm", and then re-compile the program and
run it by command "srun -n 32
--mpi=pmi2 /usr/local/apps/cxxcpi".
在 2013-01-11五的 16:09 -0700，Michael Colonno写道：
> Hi All ~
>
>
>
> I've compiled MPICH2 3.0 with the Intel compiler (v. 13)
> on a CentOS 6.3 x64 system using SLURM as the process manager. My
> configure was simply:
>
>
>
> ./configure --with-pmi=slurm --with-pm=no
> --prefix=/usr/local/apps/MPICH2
>
>
>
> No errors during build or install. When I compile and run the example
> program cxxcpi I get (truncated):
>
>
>
> $ srun -n32 /usr/local/apps/cxxcpi
>
> Fatal error in PMPI_Reduce: A process has failed, error stack:
>
> PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff4ad18120,
> rbuf=0x7fff4ad18128, count=1, MPI_DOUBLE, MPI_SUM, root=0,
> MPI_COMM_WORLD) failed
>
> MPIR_Reduce_impl(1029)..........:
>
> MPIR_Reduce_intra(779)..........:
>
> MPIR_Reduce_impl(1029)..........:
>
> MPIR_Reduce_intra(835)..........:
>
> MPIR_Reduce_binomial(144).......:
>
> MPIDI_CH3U_Recvq_FDU_or_AEP(612): Communication error with rank 16
>
> MPIR_Reduce_intra(799)..........:
>
> MPIR_Reduce_impl(1029)..........:
>
> MPIR_Reduce_intra(835)..........:
>
> MPIR_Reduce_binomial(206).......: Failure during collective
>
> srun: error: task 0: Exited with exit code 1
>
>
>
> This error is experienced with many of my MPI programs. A
> different application yields:
>
>
>
> PMPI_Bcast(1525)......: MPI_Bcast(buf=0x7fff545be5fc, count=1,
> MPI_INT, root=0, MPI_COMM_WORLD) failed
>
> MPIR_Bcast_impl(1369).:
>
> MPIR_Bcast_intra(1160):
>
> MPIR_SMP_Bcast(1077)..: Failure during collective
>
>
>
> I have tried this with several different versions of
> MPICH2 with the same results so my guess is it lies somewhere in
> SLURM’s interaction with MPI. Can anyone point me in the right
> direction?
>
>
>
> Thanks,
>
> ~Mike C.
>
>
>
>
>
