Hi Yingdi,
Perhaps you can use Slurm Epilog to clean up the user processes once the
job is completed or aborted. You can add something like this to your epilog:
if [ "$SLURM_JOB_USER" != "root" ]
then
/usr/bin/pkill -9 -u $SLURM_JOB_USER
fi
Hope that helps.
Thanks,
Sourav
On Sat, May 28, 2016 at 4:00 AM, 10000 <[email protected]> wrote:
> Hi,
> I meet a weird problem. I am "salloc mpirun" to start a mpi task on
> Slurm cluster. And then, if I cancel the job by running "scancel <jobid>"
> and after waiting about 30 seconds, there prints "srun: error:
> HK-IDC1-10-1-72-101: task 0: Killed". But I find that it only kill the
> first process that mpi creates and doesn't kill the other processes
> including the "mpirun" and "salloc". Then all the process are just like
> been frozen but the status providing by "ps" has show that these processes
> are in status "Rsl". Then I decide to kill one of the rest mpi processes to
> trigger the mpi failure for cleanning up the rest mpi processes, I find
> that is not worked (That should be work since our process are always doing
> ALL_REDUCE.). Then if I send Ctrl-C to (twice) the shell running "salloc",
> i will kill "salloc" and "mpirun" but leave the rest processes that mpirun
> created. So I have to kill them one by one at every node where I just
> running it. However, if I cancel the job by send Ctrl-C at the shell
> running "salloc" at first it do work and clean up all processes.
> Here are the command I run salloc:
> salloc -n12 --ntasks-per-node=4 /usr/local/mvapich2-2.2rc1/bin/mpirun
> ./aa
>
> The code of aa are following:
>
> #include <stdio.h>
> #include <unistd.h>
> #include <mpi.h>
> int main(int argc,char **argv)
> {
> int rank,size;
> double in=1.0,out;
> MPI_Init(&argc, &argv);
> MPI_Comm_rank(MPI_COMM_WORLD, &rank);
> MPI_Comm_size(MPI_COMM_WORLD, &size);
> while (1)
> {
> MPI_Barrier(MPI_COMM_WORLD);
> MPI_Allreduce(&in, &out, 1, MPI_DOUBLE, MPI_SUM, MPI_COMM_WORLD);
> printf("Running! %lf\n",out);
> sleep(1);
> }
> MPI_Finalize();
> return 0;
> }
>
> The MPI we using are MVAPICH2-2.2rc1 which is the latest version of
> mvapich.
> The SLURM we using are slurm-15.08.11.tar.bz2 which is also the
> latest version of slurm15
>
> Any one have an idea?
>
> Thanks
> Yingdi Guo
