Hi Moe, Sorry it took me so long to reply. (It took me a while to figure out how to get patches into our build system, and test.)
Looks good for jobs requesting 1, 2 or 3 GPUs. However we've had 2 segfaults during testing: slurmctld[25748]: segfault at 0000000000000000 rip 0000000000513a80 rsp 0000000042085570 error 4 slurmctld[1069]: segfault at 0000000000000000 rip 0000000000513a80 rsp 0000000040ffc570 error 4 The core dumps do seem to seem to be timed exactly when a sbatch is run that causes the in use GPUs to transition from one socket to the next. Once it happened when requesting the 5th, single GPU sbatch, another time when requesting 2 GPUs followed by 3 GPUs. Also, if 4 GPUs are requested, the job pends waiting on resources indefinitely. I'll see if I can get a core file and stack trace and send them on. Thanks, Goran -----Original Message----- From: Moe Jette [mailto:[email protected]] Sent: Wednesday, January 09, 2013 6:54 PM To: slurm-dev Subject: [slurm-dev] Re: Mapping slurm CPU IDs to Linux CPUs with task/affinity Hi Goran, I have been able to replicate this problem on similar hardware and have a fix that will be in Slurm v2.5.2. If you want the patch now, see https://github.com/SchedMD/slurm/commit/a6bfae71d9fc5af45a5b3eb601d5ddb54b46a3ad.patch And another patch to better document CPU index values: https://github.com/SchedMD/slurm/commit/54ef617265d9a2689ff796b83d68dae395b85213.patch Moe Quoting "Pocina, Goran" <[email protected]>: > We're not sure how to interpret the 2.5.0 SBATCH_CPU_BIND_LIST > values we see when enabling task/affinity scheduling. Or, we're > using the wrong slurm internal CPU numbering in gres.conf. We have > the following enabled: > >> TaskPluginParam=sched,verbose >> TaskPlugin=task/affinity >> DebugFlags=Gres,CPU_BIND,Steps >> GresTypes=gpu,diskio >> NodeName=drdgpu0006 Procs=16 Sockets=2 CoresPerSocket=4 >> ThreadsPerCore=2 RealMemory=20000 TmpDisk=1024 >> Gres="diskio:0,gpu:8" State=UNKNOWN Weight=1 Feature="gtx580,class2" >> PartitionName=DEFAULT Default=NO DefaultTime=10 MaxTime=INFINITE >> AllowGroups=dradadm State=Up Priority=100 PreemptMode=OFF >> DefMemPerCPU=0 >> PartitionName=swdev-gpu Nodes=drdgpu0006,drdws0121 AllowGroups=ALL >> State=Up PreemptMode=OFF Shared=NO DefMemPerCPU=0 > > > The gres.conf for drdgpu0006 follows. The node is a two socket, 4 > core per socket, 2 thread per core node with 8 GPUs. 4 GPUs are on > the same PLX as the first socket, and 4 are closer to the 2nd. Are > goal is always to use combinations of GPUs and CPUs on the same PLX. > > Our understanding is that the CPU number in gres.conf should be > slurm internal CPU numbering, which has adjacent threads on adjacent > CPUs, and not Linux numbering (adjacent threads on CPUs 0,8, etc.) > and, so we've defined the following gres.conf: > > ################################################################## > # SLURM's Generic Resource (GRES) configuration file > ################################################################## > Name=diskio Count=0 > Name=gpu File=/dev/nvidia0 CPUs=0,1,2,3,4,5,6,7 > Name=gpu File=/dev/nvidia1 CPUs=0,1,2,3,4,5,6,7 > Name=gpu File=/dev/nvidia2 CPUs=0,1,2,3,4,5,6,7 > Name=gpu File=/dev/nvidia3 CPUs=0,1,2,3,4,5,6,7 > Name=gpu File=/dev/nvidia4 CPUs=8,9,10,11,12,13,14,15 > Name=gpu File=/dev/nvidia5 CPUs=8,9,10,11,12,13,14,15 > Name=gpu File=/dev/nvidia6 CPUs=8,9,10,11,12,13,14,15 > Name=gpu File=/dev/nvidia7 CPUs=8,9,10,11,12,13,14,15 > > > When we run 4 sbatch commands similar to the following: sbatch > -p swdev-gpu -n2 --gres=gpu:2 --wrap="env; sleep 10" > > we get the following recommendations for GPU/CPU combinations: > > slurm-75635.out:SBATCH_CPU_BIND_LIST=0x0101 > CUDA_VISIBLE_DEVICES=0,1 SLURM_JOB_NODELIST=drdgpu0006 > slurm-75632.out:SBATCH_CPU_BIND_LIST=0x1010 > CUDA_VISIBLE_DEVICES=2,3 SLURM_JOB_NODELIST=drdgpu0006 > slurm-75633.out:SBATCH_CPU_BIND_LIST=0x0202 > CUDA_VISIBLE_DEVICES=4,5 SLURM_JOB_NODELIST=drdgpu0006 > slurm-75634.out:SBATCH_CPU_BIND_LIST=0x2020 > CUDA_VISIBLE_DEVICES=6,7 SLURM_JOB_NODELIST=drdgpu0006 > > If we interpret the CPU_BIND_LIST as a binary bitmask indicating > what CPUs to use, starting at zero, we get: > > 0x0101 = 00000100000001 = CPUs 0,8 CUDA_VISIBLE_DEVICES=0,1 > 0x1010 = 01000000010000 = CPUs 4,12 CUDA_VISIBLE_DEVICES=2,3 > 0x0202 = 00001000000010 = CPUs 1,9 CUDA_VISIBLE_DEVICES=4,5 > 0x2020 = 01000000010000 = CPUs 5,13 CUDA_VISIBLE_DEVICES=6,7 > > Whether we interpret these CPU numbers as Linux or Slurm CPUs, we > don't seem to have the desired mapping of GPUs to sockets, so very > likely we're using the wrong slurm CPU mapping for gres.conf. BTW. > Here's a detailed table of what we thought the mapping was: > > Slurm Linux Socket/Core/Thread > 0 0 0/0/0 > 1 8 0/0/1 > 2 1 0/1/0 > 3 9 0/1/1 > 4 2 0/2/0 > 5 10 0/2/1 > 6 3 0/3/0 > 7 11 0/3/1 > 8 4 1/0/0 > 9 12 1/0/1 > 10 5 1/1/0 > 11 13 1/1/1 > 12 6 1/2/0 > 13 14 1/2/1 > 14 7 1/3/0 > 15 15 1/3/1 > > > Can anyone see where we left the road? > > Thanks, > > Goran > > > > >
