We're not sure how to interpret the 2.5.0 SBATCH_CPU_BIND_LIST values we see
when enabling task/affinity scheduling. Or, we're using the wrong slurm
internal CPU numbering in gres.conf. We have the following enabled:
> TaskPluginParam=sched,verbose
> TaskPlugin=task/affinity
> DebugFlags=Gres,CPU_BIND,Steps
> GresTypes=gpu,diskio
> NodeName=drdgpu0006 Procs=16 Sockets=2 CoresPerSocket=4 ThreadsPerCore=2
> RealMemory=20000 TmpDisk=1024 Gres="diskio:0,gpu:8" State=UNKNOWN Weight=1
> Feature="gtx580,class2"
> PartitionName=DEFAULT Default=NO DefaultTime=10 MaxTime=INFINITE
> AllowGroups=dradadm State=Up Priority=100 PreemptMode=OFF DefMemPerCPU=0
> PartitionName=swdev-gpu Nodes=drdgpu0006,drdws0121 AllowGroups=ALL State=Up
> PreemptMode=OFF Shared=NO DefMemPerCPU=0
The gres.conf for drdgpu0006 follows. The node is a two socket, 4 core per
socket, 2 thread per core node with 8 GPUs. 4 GPUs are on the same PLX as the
first socket, and 4 are closer to the 2nd. Are goal is always to use
combinations of GPUs and CPUs on the same PLX.
Our understanding is that the CPU number in gres.conf should be slurm internal
CPU numbering, which has adjacent threads on adjacent CPUs, and not Linux
numbering (adjacent threads on CPUs 0,8, etc.) and, so we've defined the
following gres.conf:
##################################################################
# SLURM's Generic Resource (GRES) configuration file
##################################################################
Name=diskio Count=0
Name=gpu File=/dev/nvidia0 CPUs=0,1,2,3,4,5,6,7
Name=gpu File=/dev/nvidia1 CPUs=0,1,2,3,4,5,6,7
Name=gpu File=/dev/nvidia2 CPUs=0,1,2,3,4,5,6,7
Name=gpu File=/dev/nvidia3 CPUs=0,1,2,3,4,5,6,7
Name=gpu File=/dev/nvidia4 CPUs=8,9,10,11,12,13,14,15
Name=gpu File=/dev/nvidia5 CPUs=8,9,10,11,12,13,14,15
Name=gpu File=/dev/nvidia6 CPUs=8,9,10,11,12,13,14,15
Name=gpu File=/dev/nvidia7 CPUs=8,9,10,11,12,13,14,15
When we run 4 sbatch commands similar to the following: sbatch -p swdev-gpu
-n2 --gres=gpu:2 --wrap="env; sleep 10"
we get the following recommendations for GPU/CPU combinations:
slurm-75635.out:SBATCH_CPU_BIND_LIST=0x0101 CUDA_VISIBLE_DEVICES=0,1
SLURM_JOB_NODELIST=drdgpu0006
slurm-75632.out:SBATCH_CPU_BIND_LIST=0x1010 CUDA_VISIBLE_DEVICES=2,3
SLURM_JOB_NODELIST=drdgpu0006
slurm-75633.out:SBATCH_CPU_BIND_LIST=0x0202 CUDA_VISIBLE_DEVICES=4,5
SLURM_JOB_NODELIST=drdgpu0006
slurm-75634.out:SBATCH_CPU_BIND_LIST=0x2020 CUDA_VISIBLE_DEVICES=6,7
SLURM_JOB_NODELIST=drdgpu0006
If we interpret the CPU_BIND_LIST as a binary bitmask indicating what CPUs to
use, starting at zero, we get:
0x0101 = 00000100000001 = CPUs 0,8 CUDA_VISIBLE_DEVICES=0,1
0x1010 = 01000000010000 = CPUs 4,12 CUDA_VISIBLE_DEVICES=2,3
0x0202 = 00001000000010 = CPUs 1,9 CUDA_VISIBLE_DEVICES=4,5
0x2020 = 01000000010000 = CPUs 5,13 CUDA_VISIBLE_DEVICES=6,7
Whether we interpret these CPU numbers as Linux or Slurm CPUs, we don't seem to
have the desired mapping of GPUs to sockets, so very likely we're using the
wrong slurm CPU mapping for gres.conf. BTW. Here's a detailed table of what
we thought the mapping was:
Slurm Linux Socket/Core/Thread
0 0 0/0/0
1 8 0/0/1
2 1 0/1/0
3 9 0/1/1
4 2 0/2/0
5 10 0/2/1
6 3 0/3/0
7 11 0/3/1
8 4 1/0/0
9 12 1/0/1
10 5 1/1/0
11 13 1/1/1
12 6 1/2/0
13 14 1/2/1
14 7 1/3/0
15 15 1/3/1
Can anyone see where we left the road?
Thanks,
Goran
Configuration data as of 2013-01-09T12:25:22
AccountingStorageBackupHost = (null)
AccountingStorageEnforce = associations,limits,qos
AccountingStorageHost = drdwesty0001
AccountingStorageLoc = N/A
AccountingStoragePort = 6819
AccountingStorageType = accounting_storage/slurmdbd
AccountingStorageUser = N/A
AccountingStoreJobComment = YES
AcctGatherEnergyType = acct_gather_energy/none
AcctGatherNodeFreq = 0 sec
AuthType = auth/munge
BackupAddr = (null)
BackupController = (null)
BatchStartTimeout = 10 sec
BOOT_TIME = 2013-01-09T11:42:23
CacheGroups = 0
CheckpointType = checkpoint/none
ClusterName = swdev
CompleteWait = 0 sec
ControlAddr = 149.77.53.80
ControlMachine = drdctld0001
CryptoType = crypto/munge
DebugFlags = CPU_Bind,Gres,Steps
DefMemPerNode = UNLIMITED
DisableRootJobs = NO
EnforcePartLimits = YES
Epilog = /opt/slurmcl2/epilog/run-epilog
EpilogMsgTime = 2000 usec
EpilogSlurmctld = (null)
FastSchedule = 1
FirstJobId = 1
GetEnvTimeout = 2 sec
GresTypes = gpu,diskio
GroupUpdateForce = 0
GroupUpdateTime = 600 sec
HASH_VAL = Different Ours=0xb4ca2cc Slurmctld=0x41c9e8e9
HealthCheckInterval = 0 sec
HealthCheckProgram = (null)
InactiveLimit = 0 sec
JobAcctGatherFrequency = 300 sec
JobAcctGatherType = jobacct_gather/linux
JobCheckpointDir = /var/slurm/checkpoint
JobCompHost = localhost
JobCompLoc = /var/log/slurm_jobcomp.log
JobCompPort = 0
JobCompType = jobcomp/none
JobCompUser = root
JobCredentialPrivateKey = (null)
JobCredentialPublicCertificate = (null)
JobFileAppend = 0
JobRequeue = 1
JobSubmitPlugins = (null)
KillOnBadExit = 0
KillWait = 30 sec
LaunchType = launch/slurm
Licenses = (null)
LicensesUsed = (null)
MailProg = /bin/mail
MaxJobCount = 100000
MaxJobId = 4294901760
MaxMemPerNode = UNLIMITED
MaxStepCount = 40000
MaxTasksPerNode = 128
MessageTimeout = 10 sec
MinJobAge = 3 sec
MpiDefault = none
MpiParams = ports=12000-12999
NEXT_JOB_ID = 75639
OverTimeLimit = 0 min
PluginDir = /opt/slurmcl2/lib64/slurm
PlugStackConfig = /opt/slurmcl2/etc/plugstack.conf
PreemptMode = OFF
PreemptType = preempt/none
PriorityDecayHalfLife = 7-00:00:00
PriorityCalcPeriod = 01:00:00
PriorityFavorSmall = 0
PriorityFlags = 0
PriorityMaxAge = 7-00:00:00
PriorityUsageResetPeriod = NONE
PriorityType = priority/multifactor
PriorityWeightAge = 1000000
PriorityWeightFairShare = 10000000
PriorityWeightJobSize = 0
PriorityWeightPartition = 10000000
PriorityWeightQOS = 100000000
PrivateData = none
ProctrackType = proctrack/linuxproc
Prolog = /opt/slurmcl2/prolog/run-prolog
PrologSlurmctld = (null)
PropagatePrioProcess = 1
PropagateResourceLimits = ALL
PropagateResourceLimitsExcept = (null)
RebootProgram = (null)
ReconfigFlags = (null)
ResumeProgram = /opt/slurmcl2/prolog/slurmcl2_resume
ResumeRate = 300 nodes/min
ResumeTimeout = 60 sec
ResvOverRun = 0 min
ReturnToService = 1
SallocDefaultCommand = (null)
SchedulerParameters = (null)
SchedulerPort = 7321
SchedulerRootFilter = 1
SchedulerTimeSlice = 30 sec
SchedulerType = sched/backfill
SelectType = select/cons_res
SelectTypeParameters = CR_CORE_MEMORY
SlurmUser = slurmcl2(406)
SlurmctldDebug = info
SlurmctldLogFile = (null)
SlurmSchedLogFile = (null)
SlurmctldPort = 6930
SlurmctldTimeout = 120 sec
SlurmdDebug = info
SlurmdLogFile = (null)
SlurmdPidFile = /var/run/slurmcl2d.pid
SlurmdPort = 6930
SlurmdSpoolDir = /opt/slurmcl2/tmp/slurmd.%n
SlurmdTimeout = 300 sec
SlurmdUser = root(0)
SlurmSchedLogLevel = 0
SlurmctldPidFile = /var/run/slurmcl2ctld-swdev.pid
SLURM_CONF = /opt/slurmcl2/etc/slurm.conf
SLURM_VERSION = 2.5.0
SrunEpilog = (null)
SrunProlog = (null)
StateSaveLocation = /var/spool/slurmcl2/swdev
SuspendExcNodes = swdev[01-14],drdgpu0006,drdws0121
SuspendExcParts = (null)
SuspendProgram = /opt/slurmcl2/prolog/slurmcl2_suspend
SuspendRate = 60 nodes/min
SuspendTime = 120 sec
SuspendTimeout = 30 sec
SwitchType = switch/none
TaskEpilog = (null)
TaskPlugin = task/affinity
TaskPluginParam = verbose
TaskProlog = (null)
TmpFS = /tmp
TopologyPlugin = topology/none
TrackWCKey = 0
TreeWidth = 50
UsePam = 0
UnkillableStepProgram = (null)
UnkillableStepTimeout = 60 sec
VSizeFactor = 0 percent
WaitTime = 0 sec
Slurmctld(primary/backup) at drdctld0001/(NULL) are UP/DOWN
