Hi Goran, I have been able to replicate this problem on similar hardware and have a fix that will be in Slurm v2.5.2. If you want the patch now, see
https://github.com/SchedMD/slurm/commit/a6bfae71d9fc5af45a5b3eb601d5ddb54b46a3ad.patch And another patch to better document CPU index values: https://github.com/SchedMD/slurm/commit/54ef617265d9a2689ff796b83d68dae395b85213.patch Moe Quoting "Pocina, Goran" <[email protected]>: > We're not sure how to interpret the 2.5.0 SBATCH_CPU_BIND_LIST > values we see when enabling task/affinity scheduling. Or, we're > using the wrong slurm internal CPU numbering in gres.conf. We have > the following enabled: > >> TaskPluginParam=sched,verbose >> TaskPlugin=task/affinity >> DebugFlags=Gres,CPU_BIND,Steps >> GresTypes=gpu,diskio >> NodeName=drdgpu0006 Procs=16 Sockets=2 CoresPerSocket=4 >> ThreadsPerCore=2 RealMemory=20000 TmpDisk=1024 >> Gres="diskio:0,gpu:8" State=UNKNOWN Weight=1 Feature="gtx580,class2" >> PartitionName=DEFAULT Default=NO DefaultTime=10 MaxTime=INFINITE >> AllowGroups=dradadm State=Up Priority=100 PreemptMode=OFF >> DefMemPerCPU=0 >> PartitionName=swdev-gpu Nodes=drdgpu0006,drdws0121 AllowGroups=ALL >> State=Up PreemptMode=OFF Shared=NO DefMemPerCPU=0 > > > The gres.conf for drdgpu0006 follows. The node is a two socket, 4 > core per socket, 2 thread per core node with 8 GPUs. 4 GPUs are on > the same PLX as the first socket, and 4 are closer to the 2nd. Are > goal is always to use combinations of GPUs and CPUs on the same PLX. > > Our understanding is that the CPU number in gres.conf should be > slurm internal CPU numbering, which has adjacent threads on adjacent > CPUs, and not Linux numbering (adjacent threads on CPUs 0,8, etc.) > and, so we've defined the following gres.conf: > > ################################################################## > # SLURM's Generic Resource (GRES) configuration file > ################################################################## > Name=diskio Count=0 > Name=gpu File=/dev/nvidia0 CPUs=0,1,2,3,4,5,6,7 > Name=gpu File=/dev/nvidia1 CPUs=0,1,2,3,4,5,6,7 > Name=gpu File=/dev/nvidia2 CPUs=0,1,2,3,4,5,6,7 > Name=gpu File=/dev/nvidia3 CPUs=0,1,2,3,4,5,6,7 > Name=gpu File=/dev/nvidia4 CPUs=8,9,10,11,12,13,14,15 > Name=gpu File=/dev/nvidia5 CPUs=8,9,10,11,12,13,14,15 > Name=gpu File=/dev/nvidia6 CPUs=8,9,10,11,12,13,14,15 > Name=gpu File=/dev/nvidia7 CPUs=8,9,10,11,12,13,14,15 > > > When we run 4 sbatch commands similar to the following: sbatch > -p swdev-gpu -n2 --gres=gpu:2 --wrap="env; sleep 10" > > we get the following recommendations for GPU/CPU combinations: > > slurm-75635.out:SBATCH_CPU_BIND_LIST=0x0101 > CUDA_VISIBLE_DEVICES=0,1 SLURM_JOB_NODELIST=drdgpu0006 > slurm-75632.out:SBATCH_CPU_BIND_LIST=0x1010 > CUDA_VISIBLE_DEVICES=2,3 SLURM_JOB_NODELIST=drdgpu0006 > slurm-75633.out:SBATCH_CPU_BIND_LIST=0x0202 > CUDA_VISIBLE_DEVICES=4,5 SLURM_JOB_NODELIST=drdgpu0006 > slurm-75634.out:SBATCH_CPU_BIND_LIST=0x2020 > CUDA_VISIBLE_DEVICES=6,7 SLURM_JOB_NODELIST=drdgpu0006 > > If we interpret the CPU_BIND_LIST as a binary bitmask indicating > what CPUs to use, starting at zero, we get: > > 0x0101 = 00000100000001 = CPUs 0,8 CUDA_VISIBLE_DEVICES=0,1 > 0x1010 = 01000000010000 = CPUs 4,12 CUDA_VISIBLE_DEVICES=2,3 > 0x0202 = 00001000000010 = CPUs 1,9 CUDA_VISIBLE_DEVICES=4,5 > 0x2020 = 01000000010000 = CPUs 5,13 CUDA_VISIBLE_DEVICES=6,7 > > Whether we interpret these CPU numbers as Linux or Slurm CPUs, we > don't seem to have the desired mapping of GPUs to sockets, so very > likely we're using the wrong slurm CPU mapping for gres.conf. BTW. > Here's a detailed table of what we thought the mapping was: > > Slurm Linux Socket/Core/Thread > 0 0 0/0/0 > 1 8 0/0/1 > 2 1 0/1/0 > 3 9 0/1/1 > 4 2 0/2/0 > 5 10 0/2/1 > 6 3 0/3/0 > 7 11 0/3/1 > 8 4 1/0/0 > 9 12 1/0/1 > 10 5 1/1/0 > 11 13 1/1/1 > 12 6 1/2/0 > 13 14 1/2/1 > 14 7 1/3/0 > 15 15 1/3/1 > > > Can anyone see where we left the road? > > Thanks, > > Goran > > > > >
