I guess I'm going to have to dig in with a filter or something like that. Our setup seems to oversubscribe no matter what I do.
To answer John's question: I don't want to limit 1 job per node, which is why
I don't want "SHARED=FORCE:1". I want for jobs to be able to share nodes, but
not cores -- and I want for users to not be able to override this with
"--exclusive". It seems to be the case that slurm correctly ignores the
"--exclusive" switch with using FORCE, which is what I want. It's just the
deploying 50 tasks to a 16-core node that's a problem for us.
Even though it is not what I want, I took everyone's advice and switched to
SHARED=NO to resolve the oversubscription problems; however, the problem is not
resolved. Though this appears to happen less frequently, it still happens.
Currently, we have several user jobs that are alternating between suspended and
running, and at least one of those nodes is definitely oversubscribed. There
are 35 tasks from various user jobs deployed to it, while it has only 16 cores
available.
And I still don't see where my misunderstanding of the shared and cgroups
parameters are. I've read the documentation repeatedly and asked on several
email lists what I'm not understanding, and mostly people have been telling me
"don't do that" ... which is fine as far as that goes, but what I *really* want
to know is: "What am I misunderstanding?
I've been operating under the assumption that the problem is a failure on my
part: I've misconfigured something somewhere. This is why I want to
understand where my reading of the docs is flawed. Or is it possible this a
bug in slurm?
I admit that I'm pretty frustrated. It is causing a lot of performance
problems for our users with big jobs. Even though the event is rare, it is
highly probable that a large, long job will experience some unintended
oversubscription during the lifetime of its run, and this greatly hinders
performance (sometimes by orders of magnitudes). Users are very annoyed with
us, and I honestly have no idea why it is happening. We don't have the the
resources to be able to manage throughput of jobs with only exclusive nodes, so
if I cannot resolve this problem, we may have to switch back to
Torque/{Moab|Maui}.
I'm attaching our slurm.conf. Let me know if you need our topology.conf and
cgroup.conf file, as well.
Paul.
slurm.conf
Description: slurm.conf
> On Apr 11, 2016, at 10:26, Aaron Knister <[email protected]> wrote: > > > Hi Paul, > > There's always the Swiss Army knife that is a submission filter. If a user > specifies exclusive you could literally strip that argument via a submission > filter before the allocation request hits the scheduler. > > -Aaron > > Sent from my iPhone > >> On Apr 8, 2016, at 2:03 PM, Wiegand, Paul <[email protected]> wrote: >> >> >> This is *almost* what I want, but not quite. When I do this, users can >> throw the "--exclusive=YES" flag and get an exclusive node. I guess I'd >> interpreted "Shared=FORCE" to mean that they could not do this, which is >> what I wanted. I'm not sure I understand, based on the documentation, why >> "Shared=FORCE" implies that cores and consumed differently. >> >> Is there no way to disable "exclusive" while also getting what I want? >> >> Thanks, >> Paul. >> >> >>> On Apr 8, 2016, at 13:40, John DeSantis <[email protected]> wrote: >>> >>> >>> Paul, >>> >>> Try changing the Partition "Shared=FORCE" statement to "Shared=NO". >>> >>> We do that on all of our partitions and get the desired behavior. >>> >>> John DeSantis >>> >>>> On 04/08/2016 01:06 PM, Wiegand, Paul wrote: >>>> >>>> Greetings, >>>> >>>> I would like to have our cluster configured (and believed I had done) to >>>> work as follows: >>>> >>>> * User jobs can share a node, but not cores and memory >>>> * Users cannot override the share option using "--exclusive" >>>> * Once assigned cores and memory, jobs are restricted to just those >>>> assigned resources >>>> >>>> >>>> But this is not what is happening. Instead, if jobs request a >>>> sufficiently small amount of memory that they can both consume their >>>> requested amounts without contention, slurm can and does deploy jobs to >>>> the same node ... even if there are an insufficient number of cores. >>>> Instead, the jobs alternate between a "suspended" mode and "running" mode >>>> and, of course, run much, much slower as a result. >>>> >>>> Clearly I've misunderstood my configuration. We are running Slurm >>>> 15.08.3. Here are what I believe are the relevant parameters from our >>>> slurm.conf. Let me know if you want me to provide more. >>>> >>>> >>>> ProctrackType=proctrack/cgroup >>>> TaskPlugin=task/cgroup >>>> >>>> SelectType=select/cons_res >>>> SelectTypeParameters=CR_Core_Memory >>>> >>>> PartitionName=DEFAULT Shared=FORCE State=UP DefaultTime=00:10:00
