Backtraces attached. Let me know if you need the core files and binaries as
well.
Thanks.
-----Original Message-----
From: Pocina, Goran
Sent: Friday, January 11, 2013 11:18 AM
To: slurm-dev
Subject: RE: [slurm-dev] Re: Mapping slurm CPU IDs to Linux CPUs with
task/affinity
Hi Moe,
Sorry it took me so long to reply. (It took me a while to figure out how to
get patches into our build system, and test.)
Looks good for jobs requesting 1, 2 or 3 GPUs. However we've had 2 segfaults
during testing:
slurmctld[25748]: segfault at 0000000000000000 rip 0000000000513a80 rsp
0000000042085570 error 4
slurmctld[1069]: segfault at 0000000000000000 rip 0000000000513a80 rsp
0000000040ffc570 error 4
The core dumps do seem to seem to be timed exactly when a sbatch is run that
causes the in use GPUs to transition from one socket to the next. Once it
happened when requesting the 5th, single GPU sbatch, another time when
requesting 2 GPUs followed by 3 GPUs.
Also, if 4 GPUs are requested, the job pends waiting on resources indefinitely.
I'll see if I can get a core file and stack trace and send them on.
Thanks,
Goran
-----Original Message-----
From: Moe Jette [mailto:[email protected]]
Sent: Wednesday, January 09, 2013 6:54 PM
To: slurm-dev
Subject: [slurm-dev] Re: Mapping slurm CPU IDs to Linux CPUs with task/affinity
Hi Goran,
I have been able to replicate this problem on similar hardware and
have a fix that will be in Slurm v2.5.2. If you want the patch now, see
https://github.com/SchedMD/slurm/commit/a6bfae71d9fc5af45a5b3eb601d5ddb54b46a3ad.patch
And another patch to better document CPU index values:
https://github.com/SchedMD/slurm/commit/54ef617265d9a2689ff796b83d68dae395b85213.patch
Moe
Quoting "Pocina, Goran" <[email protected]>:
> We're not sure how to interpret the 2.5.0 SBATCH_CPU_BIND_LIST
> values we see when enabling task/affinity scheduling. Or, we're
> using the wrong slurm internal CPU numbering in gres.conf. We have
> the following enabled:
>
>> TaskPluginParam=sched,verbose
>> TaskPlugin=task/affinity
>> DebugFlags=Gres,CPU_BIND,Steps
>> GresTypes=gpu,diskio
>> NodeName=drdgpu0006 Procs=16 Sockets=2 CoresPerSocket=4
>> ThreadsPerCore=2 RealMemory=20000 TmpDisk=1024
>> Gres="diskio:0,gpu:8" State=UNKNOWN Weight=1 Feature="gtx580,class2"
>> PartitionName=DEFAULT Default=NO DefaultTime=10 MaxTime=INFINITE
>> AllowGroups=dradadm State=Up Priority=100 PreemptMode=OFF
>> DefMemPerCPU=0
>> PartitionName=swdev-gpu Nodes=drdgpu0006,drdws0121 AllowGroups=ALL
>> State=Up PreemptMode=OFF Shared=NO DefMemPerCPU=0
>
>
> The gres.conf for drdgpu0006 follows. The node is a two socket, 4
> core per socket, 2 thread per core node with 8 GPUs. 4 GPUs are on
> the same PLX as the first socket, and 4 are closer to the 2nd. Are
> goal is always to use combinations of GPUs and CPUs on the same PLX.
>
> Our understanding is that the CPU number in gres.conf should be
> slurm internal CPU numbering, which has adjacent threads on adjacent
> CPUs, and not Linux numbering (adjacent threads on CPUs 0,8, etc.)
> and, so we've defined the following gres.conf:
>
> ##################################################################
> # SLURM's Generic Resource (GRES) configuration file
> ##################################################################
> Name=diskio Count=0
> Name=gpu File=/dev/nvidia0 CPUs=0,1,2,3,4,5,6,7
> Name=gpu File=/dev/nvidia1 CPUs=0,1,2,3,4,5,6,7
> Name=gpu File=/dev/nvidia2 CPUs=0,1,2,3,4,5,6,7
> Name=gpu File=/dev/nvidia3 CPUs=0,1,2,3,4,5,6,7
> Name=gpu File=/dev/nvidia4 CPUs=8,9,10,11,12,13,14,15
> Name=gpu File=/dev/nvidia5 CPUs=8,9,10,11,12,13,14,15
> Name=gpu File=/dev/nvidia6 CPUs=8,9,10,11,12,13,14,15
> Name=gpu File=/dev/nvidia7 CPUs=8,9,10,11,12,13,14,15
>
>
> When we run 4 sbatch commands similar to the following: sbatch
> -p swdev-gpu -n2 --gres=gpu:2 --wrap="env; sleep 10"
>
> we get the following recommendations for GPU/CPU combinations:
>
> slurm-75635.out:SBATCH_CPU_BIND_LIST=0x0101
> CUDA_VISIBLE_DEVICES=0,1 SLURM_JOB_NODELIST=drdgpu0006
> slurm-75632.out:SBATCH_CPU_BIND_LIST=0x1010
> CUDA_VISIBLE_DEVICES=2,3 SLURM_JOB_NODELIST=drdgpu0006
> slurm-75633.out:SBATCH_CPU_BIND_LIST=0x0202
> CUDA_VISIBLE_DEVICES=4,5 SLURM_JOB_NODELIST=drdgpu0006
> slurm-75634.out:SBATCH_CPU_BIND_LIST=0x2020
> CUDA_VISIBLE_DEVICES=6,7 SLURM_JOB_NODELIST=drdgpu0006
>
> If we interpret the CPU_BIND_LIST as a binary bitmask indicating
> what CPUs to use, starting at zero, we get:
>
> 0x0101 = 00000100000001 = CPUs 0,8 CUDA_VISIBLE_DEVICES=0,1
> 0x1010 = 01000000010000 = CPUs 4,12 CUDA_VISIBLE_DEVICES=2,3
> 0x0202 = 00001000000010 = CPUs 1,9 CUDA_VISIBLE_DEVICES=4,5
> 0x2020 = 01000000010000 = CPUs 5,13 CUDA_VISIBLE_DEVICES=6,7
>
> Whether we interpret these CPU numbers as Linux or Slurm CPUs, we
> don't seem to have the desired mapping of GPUs to sockets, so very
> likely we're using the wrong slurm CPU mapping for gres.conf. BTW.
> Here's a detailed table of what we thought the mapping was:
>
> Slurm Linux Socket/Core/Thread
> 0 0 0/0/0
> 1 8 0/0/1
> 2 1 0/1/0
> 3 9 0/1/1
> 4 2 0/2/0
> 5 10 0/2/1
> 6 3 0/3/0
> 7 11 0/3/1
> 8 4 1/0/0
> 9 12 1/0/1
> 10 5 1/1/0
> 11 13 1/1/1
> 12 6 1/2/0
> 13 14 1/2/1
> 14 7 1/3/0
> 15 15 1/3/1
>
>
> Can anyone see where we left the road?
>
> Thanks,
>
> Goran
>
>
>
>
>
There are two traces here:
(gdb) bt
#0 0x0000000000513a80 in _job_alloc (job_gres_data=<value optimized out>,
node_gres_data=<value optimized out>,
node_cnt=<value optimized out>, node_offset=<value optimized out>,
cpu_cnt=<value optimized out>,
gres_name=<value optimized out>, job_id=75658, node_name=0x141ffe18
"drdws0121", core_bitmap=0x1421c690) at gres.c:2838
#1 0x0000000000518e5f in gres_plugin_job_alloc (job_gres_list=<value optimized
out>, node_gres_list=0x141fe408,
node_cnt=1, node_offset=0, cpu_cnt=2, job_id=75658, node_name=0x141ffe18
"drdws0121", core_bitmap=0x1421c690)
at gres.c:2951
#2 0x00002aaaab0ba68e in _add_job_to_res (job_ptr=0x1421f588, action=0) at
select_cons_res.c:821
#3 0x00002aaaab0ba99a in select_p_select_nodeinfo_set (job_ptr=0x1421f588) at
select_cons_res.c:2286
#4 0x0000000000457526 in select_nodes (job_ptr=0x1421f588, test_only=false,
select_node_bitmap=0x0)
at node_scheduler.c:1647
#5 0x000000000044ae48 in schedule (job_limit=1) at job_scheduler.c:934
#6 0x000000000045ff15 in _slurm_rpc_submit_batch_job (msg=0x14217088) at
proc_req.c:2693
#7 0x0000000000463db8 in slurmctld_req (msg=0x14217088) at proc_req.c:293
#8 0x00000000004303db in _service_connection (arg=0x2aaab0001ce8) at
controller.c:1022
#9 0x000000329aa0673d in start_thread () from /lib64/libpthread.so.0
#10 0x0000003299ed44bd in clone () from /lib64/libc.so.6
warning: no loadable sections found in added symbol-file system-supplied DSO at
0x7fff187c6000
Core was generated by `/opt/slurmcl2/sbin/slurmctld -n -18'.
Program terminated with signal 11, Segmentation fault.
#0 0x0000000000513a80 in _job_alloc (job_gres_data=<value optimized out>,
node_gres_data=<value optimized out>,
node_cnt=<value optimized out>, node_offset=<value optimized out>,
cpu_cnt=<value optimized out>,
gres_name=<value optimized out>, job_id=75649, node_name=0x6928e28
"drdws0121", core_bitmap=0x693c410) at gres.c:2838
2838 gres.c: No such file or directory.
in gres.c
(gdb) bt
#0 0x0000000000513a80 in _job_alloc (job_gres_data=<value optimized out>,
node_gres_data=<value optimized out>,
node_cnt=<value optimized out>, node_offset=<value optimized out>,
cpu_cnt=<value optimized out>,
gres_name=<value optimized out>, job_id=75649, node_name=0x6928e28
"drdws0121", core_bitmap=0x693c410) at gres.c:2838
#1 0x0000000000518e5f in gres_plugin_job_alloc (job_gres_list=<value optimized
out>, node_gres_list=0x6927468,
node_cnt=1, node_offset=0, cpu_cnt=2, job_id=75649, node_name=0x6928e28
"drdws0121", core_bitmap=0x693c410)
at gres.c:2951
#2 0x00002aaaab0ba68e in _add_job_to_res (job_ptr=0x6949458, action=0) at
select_cons_res.c:821
#3 0x00002aaaab0ba99a in select_p_select_nodeinfo_set (job_ptr=0x6949458) at
select_cons_res.c:2286
#4 0x0000000000457526 in select_nodes (job_ptr=0x6949458, test_only=false,
select_node_bitmap=0x0)
at node_scheduler.c:1647
#5 0x000000000044ae48 in schedule (job_limit=1) at job_scheduler.c:934
#6 0x000000000045ff15 in _slurm_rpc_submit_batch_job (msg=0x691b1e8) at
proc_req.c:2693
#7 0x0000000000463db8 in slurmctld_req (msg=0x691b1e8) at proc_req.c:293
#8 0x00000000004303db in _service_connection (arg=0x693d928) at
controller.c:1022
#9 0x000000329aa0673d in start_thread () from /lib64/libpthread.so.0
#10 0x0000003299ed44bd in clone () from /lib64/libc.so.6
