I didn't try gres. I believe it might have worked. But, I got it to work differently. I'm sharing the info here in case others might find it useful.
The issue seems to be that one or more of the relevant partitions had SHARED=Yes set. I wanted it set to No anyway, but had changed to Yes when troubleshooting. This time, I made sure that every relevant partition, even the default one that we don't use, had it set to No, and, poof, it worked. Thanks to AS, a student helping in the lab, for figuring it out! On Sat, Jan 9, 2016 at 11:43 AM, Paul Edmon <[email protected]> wrote: > > Have you tried using GRES? That's what we use and it permits both CPU and > GPU jobs on a node as GPU's become exhaustible resources: > > http://slurm.schedmd.com/gres.html > > -Paul Edmon- > > > On 1/8/2016 10:04 PM, Lachele Foley wrote: >> >> I need to do something that I haven't seen described. I'm a little >> new here. I'll appreciate any help. Thanks! >> >> Setup info: >> >> * Partitions CPU and GPU share the same nodes. >> * Each node contains 32 CPUs and 4 GPUs. Of the 32 CPUs, 4 are needed >> for the GPUs ('GPU' partition), and the other 28 are for CPU-only jobs >> ('CPU' partition). >> * I'm splitting the CPUs up between the partitions because it's >> simple, and we want each GPU to have a dedicated CPU. >> >> The problem: >> >> I can get multiple CPU jobs on a node *or* multiple GPU jobs on a >> node. But, I can't get a mixture of CPU and GPU jobs on a node. >> >> Otherwise, it seems to work just fine, allocating resources just the >> way it should. >> >> For example, assume that I have a single node, and I start a 7-cpu job >> in the CPU partition and then start two 1-cpu jobs in the GPU >> partition. In this example, I have plenty of resources for those >> three jobs to run simultaneously. But, the two GPU-partition jobs >> will not run until the CPU-partition job is finished. And, if I start >> a new 7-cpu, CPU-partition job, it will start running, with the >> GPU-partition jobs waiting in queue. The opposite is also true: if I >> start a GPU-partition job first, no CPU-partition job will run until >> it, and any subsequently-submitted GPU-partition jobs, are finished. >> Although this example assumes a single node, the analogous behavior >> occurs on multiple nodes. >> >> >> More details: >> >> The file slurm.conf contains these lines, and I wonder if >> "SelectTypeParameters=N/A" below is is a clue to what's wrong: >> >> SelectType=select/cons_res >> SelectTypeParameters=CR_CPU_Memory >> >> The results of "scontrol show partition" are below. >> >> * I couldn't get anything to happen without using "Shared=YES", and I >> had to set a DefMemPerCPU, too. The number following "YES" seems to >> have no effect on the problem behavior. >> * If you wonder, the 'default' partition has no nodes because I want >> users to have to pick a partition explicitly. >> >> PartitionName=CPU >> AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL >> AllocNodes=ALL Default=NO >> DefaultTime=NONE DisableRootJobs=NO GraceTime=0 Hidden=NO >> MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=1 LLN=NO >> MaxCPUsPerNode=28 >> Nodes=node[001-010] >> Priority=1 RootOnly=NO ReqResv=NO Shared=YES:32 PreemptMode=OFF >> State=UP TotalCPUs=320 TotalNodes=10 SelectTypeParameters=N/A >> DefMemPerCPU=7100 MaxMemPerNode=200000 >> >> PartitionName=GPU >> AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL >> AllocNodes=ALL Default=NO >> DefaultTime=NONE DisableRootJobs=NO GraceTime=0 Hidden=NO >> MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=1 LLN=NO >> MaxCPUsPerNode=4 >> Nodes=node[001-010] >> Priority=1 RootOnly=NO ReqResv=NO Shared=YES:32 PreemptMode=OFF >> State=UP TotalCPUs=320 TotalNodes=10 SelectTypeParameters=N/A >> DefMemPerCPU=12500 MaxMemPerNode=50000 >> >> > -- :-) Lachele Lachele Foley CCRC/UGA Athens, GA USA
