Hello,
we’ve noticed than on clusters configured with cgroups it’s not possible to
launch multiple backgrounded srun instances.
The test job is
#!/bin/bash
#SBATCH --nodes 1
#SBATCH --ntasks 2
#SBATCH --cpus-per-task 1
echo starting at `date`
srun -n 1 sleep 60 &
srun -n 1 sleep 60 &
wait
echo ending at `date`
On clusters without cgroups this task takes 60 seconds as both steps run in
parallel.
On a cgroup enabled cluster we see that the tasks run sequentially so take 120
seconds.
The relevant configuration is
/etc/slurm/slurm.conf:
SelectType=select/cons_res
/etc/slurm/cgroup.conf:
ConstrainCores=yes
ConstrainRAMSpace=yes
ConstrainSwapSpace=yes
TaskAffinity=yes
From a system view we see
slurmstepd: [296900]
\_ /bin/bash /var/spool/slurmd/job296900/slurm_script
\_ srun -n 1 sleep 60
| \_ srun -n 1 sleep 60
\_ srun -n 1 sleep 60
slurmstepd: [296900.0]
\_ /bin/sleep 60
If we look at the cgroup structure of such a run we see
[root@node ~]# cat /cgroup/cpuset/slurm/uid_141666/job_296900/cpuset.cpus
0,8
[root@node ~]# cat /cgroup/cpuset/slurm/uid_141666/job_296900/step_0/cpuset.cpus
0,8
The step_1 directory is only created once step_0 has finished.
Our suspicion is that because the step initiated by srun takes both cpus (the
cpuset of the parent) this blocks the second one from running. Interactively
(salloc) we observe the same behaviour along with the following message
"srun: Job step creation temporarily disabled, retrying"
This looks to us like a bug as the cpuset set for the step should, ideally,
reflect the resources requested by srun.
Thanks
Ewan Roche
Ecole Polytechnique Fédérale de Lausanne
