Upgrading OpenMPI from version 1.4.4 to version 1.6.0 did solved the problem.
(SLURM version is 2.3.5) Thanks a lot! ________________________________________ From: Moe Jette [[email protected]] Sent: Tuesday, July 10, 2012 6:18 PM To: slurm-dev Subject: [slurm-dev] RE: openmpi, cgroups & slurm (unexpected behaviour) You do not identify what version of OpenMPI and different versions operate very differently (See http://www.schedmd.com/slurmdocs/mpi_guide.html#open_mpi). You might run "scontrol show step" and "scontrol show -d job" to see what CPUs are allocated by SLURM to the job and the job step. The slurm configuration parameter DebugFlags=CPU_Bind can also be helpful there. My best _guess_ is that OpenMPI is telling slurm to allocate and launch one task per node and then launching the other tasks in that one CPU allocation. Quoting Guglielmi Matteo <[email protected]>: > . > > ________________________________ > From: Guglielmi Matteo [[email protected]] > Sent: Wednesday, June 27, 2012 11:55 PM > To: slurm-dev > Subject: [slurm-dev] openmpi, cgroups & slurm (unexpected behaviour) > > I've configured slurm to use cgroup > > ProctrackType=proctrack/cgroup > TaskPlugin=task/cgroup > > and compiled openmpi with the slurm > option: > > --with-slurm > > but when I run any parallel job using > more than one node I see that on the > first node all cores are 100% loaded: > > 17346 michalik 20 0 414m 176m 10m R 100 1.1 1:04.16 NavierStokesFra > 17347 michalik 20 0 414m 176m 9892 R 100 1.1 1:04.16 NavierStokesFra > 17348 michalik 20 0 413m 175m 9m R 100 1.1 1:04.15 NavierStokesFra > 17345 michalik 20 0 411m 173m 9648 R 100 1.1 1:04.00 NavierStokesFra > 17349 michalik 20 0 415m 177m 9540 R 100 1.1 1:04.07 NavierStokesFra > 17350 michalik 20 0 415m 176m 9376 R 100 1.1 1:04.08 NavierStokesFra > 17344 michalik 20 0 412m 173m 9708 R 98 1.1 1:04.12 NavierStokesFra > 17343 michalik 20 0 412m 174m 9748 R 97 1.1 1:04.07 NavierStokesFra > > but on the second node something > unexpected happens: > > 20150 michalik 20 0 413m 174m 10m R 13 1.1 0:18.03 NavierStokesFra > 20153 michalik 20 0 413m 175m 10m R 13 1.1 0:18.03 NavierStokesFra > 20155 michalik 20 0 413m 176m 11m R 13 1.1 0:18.04 NavierStokesFra > 20156 michalik 20 0 413m 175m 10m R 13 1.1 0:18.03 NavierStokesFra > 20157 michalik 20 0 413m 176m 10m R 13 1.1 0:18.04 NavierStokesFra > 20151 michalik 20 0 412m 174m 10m R 12 1.1 0:18.03 NavierStokesFra > 20152 michalik 20 0 412m 175m 10m R 12 1.1 0:18.03 NavierStokesFra > 20154 michalik 20 0 413m 176m 11m R 12 1.1 0:18.03 NavierStokesFra > > please note that "13%" is basically equal > to "(100/8)%" where 8 is the number of > cores per node. > > These commands added to the job file: > > srun -l hostname | sort > > srun -l cat /proc/self/status | grep Cpus_allowed_list | sort -n > > do report what follow: > > 00: cmcs01 > 01: cmcs01 > 02: cmcs01 > 03: cmcs01 > 04: cmcs01 > 05: cmcs01 > 06: cmcs01 > 07: cmcs01 > 08: cmcs02 > 09: cmcs02 > 10: cmcs02 > 11: cmcs02 > 12: cmcs02 > 13: cmcs02 > 14: cmcs02 > 15: cmcs02 > 00: Cpus_allowed_list: 0 > 01: Cpus_allowed_list: 2 > 02: Cpus_allowed_list: 4 > 03: Cpus_allowed_list: 6 > 04: Cpus_allowed_list: 1 > 05: Cpus_allowed_list: 3 > 06: Cpus_allowed_list: 5 > 07: Cpus_allowed_list: 7 > 08: Cpus_allowed_list: 0 > 09: Cpus_allowed_list: 2 > 10: Cpus_allowed_list: 4 > 11: Cpus_allowed_list: 6 > 12: Cpus_allowed_list: 1 > 13: Cpus_allowed_list: 3 > 14: Cpus_allowed_list: 5 > 15: Cpus_allowed_list: 7 > > Running the very same job "manually" > i.e. without using slurm: > > mpirun -machinefile ./hostfile -np 16 ./mpibinary > > I get 100% on all 16 cores as expected. > > Any suggestion is more than welcome. > (I think this has to do with the cgroup plugin) > > Thanks, > > --matt > >
