I need to do something that I haven't seen described. I'm a little
new here. I'll appreciate any help. Thanks!
Setup info:
* Partitions CPU and GPU share the same nodes.
* Each node contains 32 CPUs and 4 GPUs. Of the 32 CPUs, 4 are needed
for the GPUs ('GPU' partition), and the other 28 are for CPU-only jobs
('CPU' partition).
* I'm splitting the CPUs up between the partitions because it's
simple, and we want each GPU to have a dedicated CPU.
The problem:
I can get multiple CPU jobs on a node *or* multiple GPU jobs on a
node. But, I can't get a mixture of CPU and GPU jobs on a node.
Otherwise, it seems to work just fine, allocating resources just the
way it should.
For example, assume that I have a single node, and I start a 7-cpu job
in the CPU partition and then start two 1-cpu jobs in the GPU
partition. In this example, I have plenty of resources for those
three jobs to run simultaneously. But, the two GPU-partition jobs
will not run until the CPU-partition job is finished. And, if I start
a new 7-cpu, CPU-partition job, it will start running, with the
GPU-partition jobs waiting in queue. The opposite is also true: if I
start a GPU-partition job first, no CPU-partition job will run until
it, and any subsequently-submitted GPU-partition jobs, are finished.
Although this example assumes a single node, the analogous behavior
occurs on multiple nodes.
More details:
The file slurm.conf contains these lines, and I wonder if
"SelectTypeParameters=N/A" below is is a clue to what's wrong:
SelectType=select/cons_res
SelectTypeParameters=CR_CPU_Memory
The results of "scontrol show partition" are below.
* I couldn't get anything to happen without using "Shared=YES", and I
had to set a DefMemPerCPU, too. The number following "YES" seems to
have no effect on the problem behavior.
* If you wonder, the 'default' partition has no nodes because I want
users to have to pick a partition explicitly.
PartitionName=CPU
AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
AllocNodes=ALL Default=NO
DefaultTime=NONE DisableRootJobs=NO GraceTime=0 Hidden=NO
MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=1 LLN=NO MaxCPUsPerNode=28
Nodes=node[001-010]
Priority=1 RootOnly=NO ReqResv=NO Shared=YES:32 PreemptMode=OFF
State=UP TotalCPUs=320 TotalNodes=10 SelectTypeParameters=N/A
DefMemPerCPU=7100 MaxMemPerNode=200000
PartitionName=GPU
AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
AllocNodes=ALL Default=NO
DefaultTime=NONE DisableRootJobs=NO GraceTime=0 Hidden=NO
MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=1 LLN=NO MaxCPUsPerNode=4
Nodes=node[001-010]
Priority=1 RootOnly=NO ReqResv=NO Shared=YES:32 PreemptMode=OFF
State=UP TotalCPUs=320 TotalNodes=10 SelectTypeParameters=N/A
DefMemPerCPU=12500 MaxMemPerNode=50000
--
:-) Lachele
Lachele Foley
CCRC/UGA
Athens, GA USA
