Paul,
>> To answer John's question: I don't want to limit 1 job per node, which is
>> why I don't want "SHARED=FORCE:1". I want for jobs to be able to share
>> nodes, but not cores -- and I want for users to not be able to override this
>> with "--exclusive".
Ok - we are doing exactly this: sharing nodes but not CPU's (cores), and
there are no issues with over-subscription. However, users can, and do,
request exclusivity by using "--exclusive". A job submit plugin can
strip that request.
Here are the relevant portions of our sanitized configuration which you
could potentially apply and test to negate over-subscription:
PreemptMode = CANCEL
PreemptType = preempt/qos
SchedulerType = sched/backfill
SelectType = select/cons_res
SelectTypeParameters = CR_CPU_MEMORY
NodeName= CPUs=8 CoresPerSocket=4 Sockets=2 RealMemory=16012
NodeName= CPUs=12 CoresPerSocket=6 Sockets=2 RealMemory=24028
NodeName= CPUs=12 CoresPerSocket=6 Sockets=2 RealMemory=24028
NodeName= CPUs=20 CoresPerSocket=10 Sockets=2 RealMemory=516766
NodeName= CPUs=16 CoresPerSocket=4 Sockets=4 RealMemory=129119
NodeName= CPUs=12 CoresPerSocket=6 Sockets=2 RealMemory=24013
NodeName= CPUs=12 CoresPerSocket=6 Sockets=2 RealMemory=24016
NodeName= CPUs=12 CoresPerSocket=6 Sockets=2 RealMemory=48258
NodeName= CPUs=12 CoresPerSocket=6 Sockets=2 RealMemory=48251
NodeName= CPUs=12 CoresPerSocket=6 Sockets=2 RealMemory=19973
NodeName= CPUs=12 CoresPerSocket=6 Sockets=2 RealMemory=24016
NodeName= CPUs=12 CoresPerSocket=6 Sockets=2 RealMemory=23980
NodeName= CPUs=16 CoresPerSocket=8 Sockets=2 RealMemory=32075
NodeName= CPUs=16 CoresPerSocket=8 Sockets=2 RealMemory=32074
NodeName= CPUs=16 CoresPerSocket=8 Sockets=2 RealMemory=32076
NodeName= CPUs=16 CoresPerSocket=8 Sockets=2 RealMemory=32073
PartitionName=blah1 State=UP Shared=NO Default=YES DefaultTime=01:00:00
DefMemPerCPU=512
PartitionName=blah2 Nodes=list1 DefaultTime=01:00:00 MaxTime=168:00:00
AllowQOS=qos1 Shared=NO State=UP DefMemPerCPU=512
PartitionName=blah3 Nodes=list2 State=UP Shared=No DefaultTime=01:00:00
DenyQOS=qos2 Priority=5000 DefMemPerCPU=512
PartitionName=blah4 Nodes=list3 State=UP Shared=No MaxTime=02:00:00
MaxNodes=4 DenyQOS=qos2 DefMemPerCPU=512
PartitionName=blah5 Nodes=list4 State=UP Shared=No DefaultTime=01:00:00
MaxTime=26:00:00 DenyQOS=qos2 DefMemPerCPU=512
Again, this configuration affords us the ability to have nodes run
multiple user jobs without CPU's being shared; if all CPU's are
allocated, the controller will not dispatch jobs to the nodes in question.
I may be incorrect here, but looking at your configuration you've
specified that your preempt type is partition priority, yet your
partition definitions hint to a preemption configured via QOS. Is this
what you wanted? The documentation states that based upon your
preemption settings, the resultant behavior would be a mix of running
and suspended jobs based upon (a) job priorities, and (b) partition
priorities; maybe check the controller logs for preemption notices to
confirm or deny this thought? At any rate, I'd suggest only using
preemption based upon QOS.
HTH,
John DeSantis
On 04/14/2016 07:38 AM, Wiegand, Paul wrote:
> I guess I'm going to have to dig in with a filter or something like that.
> Our setup seems to oversubscribe no matter what I do.
>
> To answer John's question: I don't want to limit 1 job per node, which is
> why I don't want "SHARED=FORCE:1". I want for jobs to be able to share
> nodes, but not cores -- and I want for users to not be able to override this
> with "--exclusive". It seems to be the case that slurm correctly ignores the
> "--exclusive" switch with using FORCE, which is what I want. It's just the
> deploying 50 tasks to a 16-core node that's a problem for us.
>
> Even though it is not what I want, I took everyone's advice and switched to
> SHARED=NO to resolve the oversubscription problems; however, the problem is
> not resolved. Though this appears to happen less frequently, it still
> happens. Currently, we have several user jobs that are alternating between
> suspended and running, and at least one of those nodes is definitely
> oversubscribed. There are 35 tasks from various user jobs deployed to it,
> while it has only 16 cores available.
>
> And I still don't see where my misunderstanding of the shared and cgroups
> parameters are. I've read the documentation repeatedly and asked on several
> email lists what I'm not understanding, and mostly people have been telling
> me "don't do that" ... which is fine as far as that goes, but what I *really*
> want to know is: "What am I misunderstanding?
>
> I've been operating under the assumption that the problem is a failure on my
> part: I've misconfigured something somewhere. This is why I want to
> understand where my reading of the docs is flawed. Or is it possible this a
> bug in slurm?
>
> I admit that I'm pretty frustrated. It is causing a lot of performance
> problems for our users with big jobs. Even though the event is rare, it is
> highly probable that a large, long job will experience some unintended
> oversubscription during the lifetime of its run, and this greatly hinders
> performance (sometimes by orders of magnitudes). Users are very annoyed with
> us, and I honestly have no idea why it is happening. We don't have the the
> resources to be able to manage throughput of jobs with only exclusive nodes,
> so if I cannot resolve this problem, we may have to switch back to
> Torque/{Moab|Maui}.
>
> I'm attaching our slurm.conf. Let me know if you need our topology.conf and
> cgroup.conf file, as well.
>
> Paul.
>
