Hi Ewan, My bet is that one of the job resources is entirely consumed by the first step, so the 2nd one waits in queue. It's likely memory, maybe you have a DefMemPerCpu setting in your slurm.conf? You can try to request say 4G for your whole job and then 2G for each srun steps, they should both run simultaneously.
Cheers, -- Kilian On Wed, Dec 16, 2015 at 8:47 AM, Roche Ewan <[email protected]> wrote: > Hello, > we’ve noticed than on clusters configured with cgroups it’s not possible to > launch multiple backgrounded srun instances. > > The test job is > > #!/bin/bash > #SBATCH --nodes 1 > #SBATCH --ntasks 2 > #SBATCH --cpus-per-task 1 > > echo starting at `date` > srun -n 1 sleep 60 & > srun -n 1 sleep 60 & > wait > echo ending at `date` > > On clusters without cgroups this task takes 60 seconds as both steps run in > parallel. > > On a cgroup enabled cluster we see that the tasks run sequentially so take > 120 seconds. > > The relevant configuration is > > /etc/slurm/slurm.conf: > > SelectType=select/cons_res > > /etc/slurm/cgroup.conf: > > ConstrainCores=yes > ConstrainRAMSpace=yes > ConstrainSwapSpace=yes > TaskAffinity=yes > > > From a system view we see > > slurmstepd: [296900] > \_ /bin/bash /var/spool/slurmd/job296900/slurm_script > \_ srun -n 1 sleep 60 > | \_ srun -n 1 sleep 60 > \_ srun -n 1 sleep 60 > slurmstepd: [296900.0] > \_ /bin/sleep 60 > > > If we look at the cgroup structure of such a run we see > > [root@node ~]# cat /cgroup/cpuset/slurm/uid_141666/job_296900/cpuset.cpus > 0,8 > [root@node ~]# cat > /cgroup/cpuset/slurm/uid_141666/job_296900/step_0/cpuset.cpus > 0,8 > > The step_1 directory is only created once step_0 has finished. > > > Our suspicion is that because the step initiated by srun takes both cpus > (the cpuset of the parent) this blocks the second one from running. > Interactively (salloc) we observe the same behaviour along with the > following message > > "srun: Job step creation temporarily disabled, retrying" > > This looks to us like a bug as the cpuset set for the step should, ideally, > reflect the resources requested by srun. > > > > Thanks > > > Ewan Roche > > Ecole Polytechnique Fédérale de Lausanne > > -- Kilian
