. ________________________________ From: Guglielmi Matteo [[email protected]] Sent: Wednesday, June 27, 2012 11:55 PM To: slurm-dev Subject: [slurm-dev] openmpi, cgroups & slurm (unexpected behaviour)
I've configured slurm to use cgroup ProctrackType=proctrack/cgroup TaskPlugin=task/cgroup and compiled openmpi with the slurm option: --with-slurm but when I run any parallel job using more than one node I see that on the first node all cores are 100% loaded: 17346 michalik 20 0 414m 176m 10m R 100 1.1 1:04.16 NavierStokesFra 17347 michalik 20 0 414m 176m 9892 R 100 1.1 1:04.16 NavierStokesFra 17348 michalik 20 0 413m 175m 9m R 100 1.1 1:04.15 NavierStokesFra 17345 michalik 20 0 411m 173m 9648 R 100 1.1 1:04.00 NavierStokesFra 17349 michalik 20 0 415m 177m 9540 R 100 1.1 1:04.07 NavierStokesFra 17350 michalik 20 0 415m 176m 9376 R 100 1.1 1:04.08 NavierStokesFra 17344 michalik 20 0 412m 173m 9708 R 98 1.1 1:04.12 NavierStokesFra 17343 michalik 20 0 412m 174m 9748 R 97 1.1 1:04.07 NavierStokesFra but on the second node something unexpected happens: 20150 michalik 20 0 413m 174m 10m R 13 1.1 0:18.03 NavierStokesFra 20153 michalik 20 0 413m 175m 10m R 13 1.1 0:18.03 NavierStokesFra 20155 michalik 20 0 413m 176m 11m R 13 1.1 0:18.04 NavierStokesFra 20156 michalik 20 0 413m 175m 10m R 13 1.1 0:18.03 NavierStokesFra 20157 michalik 20 0 413m 176m 10m R 13 1.1 0:18.04 NavierStokesFra 20151 michalik 20 0 412m 174m 10m R 12 1.1 0:18.03 NavierStokesFra 20152 michalik 20 0 412m 175m 10m R 12 1.1 0:18.03 NavierStokesFra 20154 michalik 20 0 413m 176m 11m R 12 1.1 0:18.03 NavierStokesFra please note that "13%" is basically equal to "(100/8)%" where 8 is the number of cores per node. These commands added to the job file: srun -l hostname | sort srun -l cat /proc/self/status | grep Cpus_allowed_list | sort -n do report what follow: 00: cmcs01 01: cmcs01 02: cmcs01 03: cmcs01 04: cmcs01 05: cmcs01 06: cmcs01 07: cmcs01 08: cmcs02 09: cmcs02 10: cmcs02 11: cmcs02 12: cmcs02 13: cmcs02 14: cmcs02 15: cmcs02 00: Cpus_allowed_list: 0 01: Cpus_allowed_list: 2 02: Cpus_allowed_list: 4 03: Cpus_allowed_list: 6 04: Cpus_allowed_list: 1 05: Cpus_allowed_list: 3 06: Cpus_allowed_list: 5 07: Cpus_allowed_list: 7 08: Cpus_allowed_list: 0 09: Cpus_allowed_list: 2 10: Cpus_allowed_list: 4 11: Cpus_allowed_list: 6 12: Cpus_allowed_list: 1 13: Cpus_allowed_list: 3 14: Cpus_allowed_list: 5 15: Cpus_allowed_list: 7 Running the very same job "manually" i.e. without using slurm: mpirun -machinefile ./hostfile -np 16 ./mpibinary I get 100% on all 16 cores as expected. Any suggestion is more than welcome. (I think this has to do with the cgroup plugin) Thanks, --matt
