Hello,
I've taken a very good look at our cluster, however as yet not made any
significant changes. The one change that I did make was to increase the
"jobsizeweight". That's now our dominant parameter and it does ensure that our
largest jobs (> 20 nodes) are making it to the top of the sprio listing which
is what we want to see.
These large jobs aren't making an progress despite the priority lift. I
additionally decreased the nice value of the job that sparked this discussion.
That is (looking at at sprio) there is a 32 node job with a very high
priority...
JOBID PARTITION USER PRIORITY AGE FAIRSHARE JOBSIZE PARTITION
QOS NICE
280919 batch mep1c10 1275481 400000 59827 415655
0 0 -400000
That job has been sitting in the queue for well over a week and it is
disconcerting that we never see nodes becoming idle in order to service these
large jobs. Nodes do become idle and then get scooped by jobs started by
backfill. Looking at the slurmctld logs I see that the vast majority of jobs
are being started via backfill -- including, for example, a 24 node job. I see
very few jobs allocated by the scheduler. That is, messages like sched:
Allocate JobId=296915 are few and far between and I never see any of the large
jobs being allocated in the batch queue.
Surely, this is not correct, however does anyone have any advice on what to
check, please?
Best regards,
David
________________________________
From: slurm-users <slurm-users-boun...@lists.schedmd.com> on behalf of Killian
Murphy <killian.mur...@york.ac.uk>
Sent: 04 February 2020 10:48
To: Slurm User Community List <slurm-users@lists.schedmd.com>
Subject: Re: [slurm-users] Longer queuing times for larger jobs
Hi David.
I'd love to hear back about the changes that you make and how they affect the
performance of your scheduler.
Any chance you could let us know how things go?
Killian
On Tue, 4 Feb 2020 at 10:43, David Baker
<d.j.ba...@soton.ac.uk<mailto:d.j.ba...@soton.ac.uk>> wrote:
Hello,
Thank you very much again for your comments and the details of your slurm
configuration. All the information is really useful. We are working on our
cluster right now and making some appropriate changes. We'll see how we get on
over the next 24 hours or so.
Best regards,
David
________________________________
From: slurm-users
<slurm-users-boun...@lists.schedmd.com<mailto:slurm-users-boun...@lists.schedmd.com>>
on behalf of Renfro, Michael <ren...@tntech.edu<mailto:ren...@tntech.edu>>
Sent: 31 January 2020 22:08
To: Slurm User Community List
<slurm-users@lists.schedmd.com<mailto:slurm-users@lists.schedmd.com>>
Subject: Re: [slurm-users] Longer queuing times for larger jobs
Slurm 19.05 now, though all these settings were in effect on 17.02 until quite
recently. If I get some detail wrong below, I hope someone will correct me. But
this is our current working state. We’ve been able to schedule 10-20k jobs per
month since late 2017, and we successfully scheduled 320k jobs over December
and January (largely due to one user using some form of automated submission
for very short jobs).
Basic scheduler setup:
As I’d said previously, we prioritize on fairshare almost exclusively. Most of
our jobs (molecular dynamics, CFD) end up in a single batch partition, since
GPU and big-memory jobs have other partitions.
SelectType=select/cons_res
SelectTypeParameters=CR_Core_Memory
PriorityType=priority/multifactor
PriorityDecayHalfLife=14-0
PriorityWeightFairshare=100000
PriorityWeightAge=1000
PriorityWeightPartition=10000
PriorityWeightJobSize=1000
PriorityMaxAge=1-0
TRES limits:
We’ve limited users to 1000 CPU-days with: sacctmgr modify user someuser set
grptresrunmin=cpu=1440000 — there might be a way of doing this at a higher
accounting level, but it works as is.
We also force QoS=gpu in each GPU partition’s definition in slurm.conf, and set
MaxJobsPerUser equal to our total GPU count. That helps prevent users from
queue-stuffing the GPUs even if they stay well below the 1000 CPU-day TRES
limit above.
Backfill:
SchedulerType=sched/backfill
SchedulerParameters=bf_window=43200,bf_resolution=2160,bf_max_job_user=80,bf_continue,default_queue_depth=200
Can’t remember where I found the backfill guidance, but:
- bf_window is set to our maximum job length (30 days) and bf_resolution is set
to 1.5 days. Most of our users’ jobs are well over 1 day.
- We have had users who didn’t use job arrays, and submitted a ton of small
jobs at once, thus bf_max_job_user gives the scheduler a chance to start up to
80 jobs per user each cycle. This also prompted us to increase
default_queue_depth, so the backfill scheduler would examine more jobs each
cycle.
- bf_continue should let the backfill scheduler continue where it left off if
it gets interrupted, instead of having to start from scratch each time.
I can guarantee you that our backfilling was sub-par until we tuned these
parameters (or at least a few users could find a way to submit so many jobs
that the backfill couldn’t keep up, even when we had idle resources for their
very short jobs).
> On Jan 31, 2020, at 3:01 PM, David Baker
> <d.j.ba...@soton.ac.uk<mailto:d.j.ba...@soton.ac.uk>> wrote:
>
> External Email Warning
> This email originated from outside the university. Please use caution when
> opening attachments, clicking links, or responding to requests.
> Hello,
>
> Thank you for your detailed reply. That’s all very useful. I manage to
> mistype our cluster size since there are actually 450 standard compute, 40
> core, compute nodes. What you say is interesting and so it concerns me that
> things are so bad at the moment,
>
> I wondered if you could please give me some more details of how you use TRES
> to throttle user activity. We have applied some limits to throttle users,
> however perhaps not enough or not well enough. So the details of what you do
> would be really appreciated, please.
>
> In addition, we do use backfill, however we rarely see nodes being freed up
> in the cluster to make way for high priority work which again concerns me. If
> you could please share your backfill configuration then that would be
> appreciated, please.
>
> Finally, which version of Slurm are you running? We are using an early
> release of v18.
>
> Best regards,
> David
>
> From: slurm-users
> <slurm-users-boun...@lists.schedmd.com<mailto:slurm-users-boun...@lists.schedmd.com>>
> on behalf of Renfro, Michael <ren...@tntech.edu<mailto:ren...@tntech.edu>>
> Sent: 31 January 2020 17:23:05
> To: Slurm User Community List
> <slurm-users@lists.schedmd.com<mailto:slurm-users@lists.schedmd.com>>
> Subject: Re: [slurm-users] Longer queuing times for larger jobs
>
> I missed reading what size your cluster was at first, but found it on a
> second read. Our cluster and typical maximum job size scales about the same
> way, though (our users’ typical job size is anywhere from a few cores up to
> 10% of our core count).
>
> There are several recommendations to separate your priority weights by an
> order of magnitude or so. Our weights are dominated by fairshare, and we
> effectively ignore all other factors.
>
> We also put TRES limits on by default, so that users can’t queue-stuff beyond
> a certain limit (any jobs totaling under around 1 cluster-day can be in a
> running or queued state, and anything past that is ignored until their
> running jobs burn off some of their time). This allows other users’ jobs to
> have a chance to run if resources are available, even if they were submitted
> well after the heavy users’ blocked jobs.
>
> We also make extensive use of the backfill scheduler to run small, short jobs
> earlier than their queue time might allow, if and only if they don’t delay
> other jobs. If a particularly large job is about to run, we can see the nodes
> gradually empty out, which opens up lots of capacity for very short jobs.
>
> Overall, our average wait times since September 2017 haven’t exceeded 90
> hours for any job size, and I’m pretty sure a *lot* of that wait is due to a
> few heavy users submitting large numbers of jobs far beyond the TRES limit.
> Even our jobs of 5-10% cluster size have average start times of 60 hours or
> less (and we've managed under 48 hours for those size jobs for all but 2
> months of that period), but those larger jobs tend to be run by our lighter
> users, and they get a major improvement to their queue time due to being far
> below their fairshare target.
>
> We’ve been running at >50% capacity since May 2018, and >60% capacity since
> December 2018, and >80% capacity since February 2019. So our wait times
> aren’t due to having a ton of spare capacity for extended periods of time.
>
> Not sure how much of that will help immediately, but it may give you some
> ideas.
>
> > On Jan 31, 2020, at 10:14 AM, David Baker
> > <d.j.ba...@soton.ac.uk<mailto:d.j.ba...@soton.ac.uk>> wrote:
> >
> > External Email Warning
> > This email originated from outside the university. Please use caution when
> > opening attachments, clicking links, or responding to requests.
> > Hello,
> >
> > Thank you for your reply. in answer to Mike's questions...
> >
> > Our serial partition nodes are partially shared by the high memory
> > partition. That is, the partitions overlap partially -- shared nodes move
> > one way or another depending upon demand. Jobs requesting up to and
> > including 20 cores are routed to the serial queue. The serial nodes are
> > shared resources. In other words, jobs from different users can share the
> > nodes. The maximum time for serial jobs is 60 hours.
> >
> > Overtime there hasn't been any particular change in the time that users are
> > requesting. Likewise I'm convinced that the overall job size spread is the
> > same over time. What has changed is the increase in the number of smaller
> > jobs. That is, one node jobs that are exclusive (can't be routed to the
> > serial queue) or that require more then 20 cores, and also jobs requesting
> > up to 10/15 nodes (let's say). The user base has increased dramatically
> > over the last 6 months or so.
> >
> > This over population is leading to the delay in scheduling the larger jobs.
> > Given the size of the cluster we may need to make decisions regarding which
> > types of jobs we allow to "dominate" the system. The larger jobs at the
> > expense of the small fry for example, however that is a difficult decision
> > that means that someone has got to wait longer for results..
> >
> > Best regards,
> > David
> > From: slurm-users
> > <slurm-users-boun...@lists.schedmd.com<mailto:slurm-users-boun...@lists.schedmd.com>>
> > on behalf of Renfro, Michael <ren...@tntech.edu<mailto:ren...@tntech.edu>>
> > Sent: 31 January 2020 13:27
> > To: Slurm User Community List
> > <slurm-users@lists.schedmd.com<mailto:slurm-users@lists.schedmd.com>>
> > Subject: Re: [slurm-users] Longer queuing times for larger jobs
> >
> > Greetings, fellow general university resource administrator.
> >
> > Couple things come to mind from my experience:
> >
> > 1) does your serial partition share nodes with the other non-serial
> > partitions?
> >
> > 2) what’s your maximum job time allowed, for serial (if the previous answer
> > was “yes”) and non-serial partitions? Are your users submitting
> > particularly longer jobs compared to earlier?
> >
> > 3) are you using the backfill scheduler at all?
> >
> > --
> > Mike Renfro, PhD / HPC Systems Administrator, Information Technology
> > Services
> > 931 372-3601 / Tennessee Tech University
> >
> >> On Jan 31, 2020, at 6:23 AM, David Baker
> >> <d.j.ba...@soton.ac.uk<mailto:d.j.ba...@soton.ac.uk>> wrote:
> >>
> >> Hello,
> >>
> >> Our SLURM cluster is relatively small. We have 350 standard compute nodes
> >> each with 40 cores. The largest job that users can run on the partition
> >> is one requesting 32 nodes. Our cluster is a general university research
> >> resource and so there are many different sizes of jobs ranging from single
> >> core jobs, that get routed to a serial partition via the job-submit.lua,
> >> through to jobs requesting 32 nodes. When we first started the service, 32
> >> node jobs were typically taking in the region of 2 days to schedule --
> >> recently queuing times have started to get out of hand. Our setup is
> >> essentially...
> >>
> >> PriorityFavorSmall=NO
> >> FairShareDampeningFactor=5
> >> PriorityFlags=ACCRUE_ALWAYS,FAIR_TREE
> >> PriorityType=priority/multifactor
> >> PriorityDecayHalfLife=7-0
> >>
> >> PriorityWeightAge=400000
> >> PriorityWeightPartition=1000
> >> PriorityWeightJobSize=500000
> >> PriorityWeightQOS=1000000
> >> PriorityMaxAge=7-0
> >>
> >> To try to reduce the queuing times for our bigger jobs should we
> >> potentially increase the PriorityWeightJobSize factor in the first
> >> instance to bump up the priority of such jobs? Or should we potentially
> >> define a set of QOSs which we assign to jobs in our job_submit.lua
> >> depending on the size of the job. In other words, let's say there islarge
> >> QOS that give the largest jobs a higher priority, and also limits how many
> >> of those jobs that a single user can submit?
> >>
> >> Your advice would be appreciated, please. At the moment these large jobs
> >> are not accruing a sufficiently high priority to rise above the other jobs
> >> in the cluster.
> >>
> >> Best regards,
> >> David
>
--
Killian Murphy
Research and High Performance Computing Team Leader
Research Software Engineer
Information Services & Wolfson Atmospheric Chemistry Laboratories
University of York
Heslington
York
YO10 5DD
+44 (0)1904 32 4753
e-mail disclaimer:
http://www.york.ac.uk/docs/disclaimer/email.htm<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.york.ac.uk%2Fdocs%2Fdisclaimer%2Femail.htm&data=01%7C01%7Cd.j.baker%40soton.ac.uk%7C556406aa96c54e0dbb8608d7a95ff956%7C4a5378f929f44d3ebe89669d03ada9d8%7C0&sdata=AfWJItMNASlNf0NMDH9YhOdkMlb3jFbzopwPWf9xGkY%3D&reserved=0>
