Thanks for the reply. I'm not 100% clear on the below so let me be more
specific. I'm launching the code via srun (for example). The code launches,
runs a few different executables in order, and eventually launches a few MPI
processes though its own MPI implementation. I have no control over the source
code nor what syntax is used to launch the sub-processes. srun launches these
processes and then reports the job completed; this is the only tool that
behaves this way (others seem to track processes even if not launched through
SLURM). Is the conclusion that if the sub-processes are not launched explicitly
via SLURM (but are child processes of a SLURM-launched process) there is
nothing that can be done at the SLURM level to prevent SLURM from relinquishing
the resources before the job is completed?
Thanks,
~Mike C.
-----Original Message-----
From: Moe Jette [mailto:[email protected]]
Sent: Thursday, February 21, 2013 11:00 AM
To: slurm-dev; Michael Colonno
Subject: Re: [slurm-dev] untracked processes
Slurm only tracks the processes that it's daemons launch (most MPI
implementations can launch their tasks using slurm). Anything launched outside
of Slurm can be killed as part of a job prolog, but accounting and job step
management are outside of Slurm's control.
Quoting Michael Colonno <[email protected]>:
>
> SLURM gurus ~
>
> I'm trying to configure a commercial MPI code to run through SLURM.
> I can launch this code through either srun or sbatch without any
> issues (the good) but the processes manage to run completely
> disconnected from SLURM's notice (the bad). i.e. the job is running
> just fine but SLURM thinks it's completed and hence does not report
> anything running. I'm guessing this is due to the fact that this tool
> runs a pre-processing-type executable and then launches sub-processes
> to solve (MPI on a local system) without connecting the process IDs(?)
> In any event, I'm guessing I'm not the first person to run into this.
> Is there a recommended solution to configure SLURM to track codes like
> this?
>
> Thanks,
> ~Mike C.
>
>
