Hi Kilian, good thinking! It really is nothing to do with groups but rather the SelectType=select/cons_res
By default the steps get the same RSS as the parent and this is what prevents the second task being started as SLURM thinks that there is no more memory available. As you suggested #SBATCH --mem 16000 srun -n 1 --mem 1024 task & srun -n 1 --mem 1024 task & allows both to run in parallel. In the logs we see [329507] Set task rss(16000 MB) .. [329507.0] Set task rss(1024 MB) .. [329507.1] Set task rss(1024 MB) Without the mem option the rss for the step is also 16000 MB Thanks Ewan Roche Ecole Polytechnique Fédérale de Lausanne On 16 déc. 2015, at 21:37, Kilian Cavalotti <[email protected]<mailto:[email protected]>> wrote: Hi Ewan, My bet is that one of the job resources is entirely consumed by the first step, so the 2nd one waits in queue. It's likely memory, maybe you have a DefMemPerCpu setting in your slurm.conf? You can try to request say 4G for your whole job and then 2G for each srun steps, they should both run simultaneously. Cheers, -- Kilian On Wed, Dec 16, 2015 at 8:47 AM, Roche Ewan <[email protected]<mailto:[email protected]>> wrote: Hello, we’ve noticed than on clusters configured with cgroups it’s not possible to launch multiple backgrounded srun instances. The test job is #!/bin/bash #SBATCH --nodes 1 #SBATCH --ntasks 2 #SBATCH --cpus-per-task 1 echo starting at `date` srun -n 1 sleep 60 & srun -n 1 sleep 60 & wait echo ending at `date` On clusters without cgroups this task takes 60 seconds as both steps run in parallel. On a cgroup enabled cluster we see that the tasks run sequentially so take 120 seconds. The relevant configuration is /etc/slurm/slurm.conf: SelectType=select/cons_res /etc/slurm/cgroup.conf: ConstrainCores=yes ConstrainRAMSpace=yes ConstrainSwapSpace=yes TaskAffinity=yes From a system view we see slurmstepd: [296900] \_ /bin/bash /var/spool/slurmd/job296900/slurm_script \_ srun -n 1 sleep 60 | \_ srun -n 1 sleep 60 \_ srun -n 1 sleep 60 slurmstepd: [296900.0] \_ /bin/sleep 60 If we look at the cgroup structure of such a run we see [root@node ~]# cat /cgroup/cpuset/slurm/uid_141666/job_296900/cpuset.cpus 0,8 [root@node ~]# cat /cgroup/cpuset/slurm/uid_141666/job_296900/step_0/cpuset.cpus 0,8 The step_1 directory is only created once step_0 has finished. Our suspicion is that because the step initiated by srun takes both cpus (the cpuset of the parent) this blocks the second one from running. Interactively (salloc) we observe the same behaviour along with the following message "srun: Job step creation temporarily disabled, retrying" This looks to us like a bug as the cpuset set for the step should, ideally, reflect the resources requested by srun. Thanks Ewan Roche Ecole Polytechnique Fédérale de Lausanne -- Kilian
