Trey This helps a lot. Thank you! I'll be reconfiguring PriorityFavorSmall=NO PriorityFlags=DEPTH_OBLIVIOUS,SMALL_RELATIVE_TO_TIME
shortly, and see about changing MAXJOBS and MAXSUBMIT some more to tweak how he's handled. Gerry On Sun, Aug 30, 2015 at 4:11 PM, Trey Dockendorf <[email protected]> wrote: > You may have to adjust how you prioritize jobs by size and wallclock time. > > PriorityFavorSmall=NO > PriorityFlags=DEPTH_OBLIVIOUS,SMALL_RELATIVE_TO_TIME > > The PriorityFavorSmall and SMALL_RELATIVE_TO_TIME is used by us to favor > both short and large jobs. So if two jobs are equal in size, the shorter > of the two is favored. Also if two jobs are equal in time, the larger is > favored. We use this as a way to get short jobs in and out of the queues > quickly as well as help large jobs (typically MPI) have priority over small > serial jobs. > > How much weight the job size is given for priority will be determined also > by PriorityWeightJobSize. > > A small job getting in before big jobs also has to do with how backfill is > configured. If the small job starting won't impact a higher priority large > job's start time, it should get backfilled. > > You can use 'sshare' to view the fair share values for users. You can use > 'sprio' to inspect a pending job's various priority factors. So if you > inspect this particular user's jobs against that of someone who had to > wait, you may see why SLURM favored those jobs. One thing I did when > planning our FairShare was create a handful of temporary accounts to > represent our various users and then submit batches of jobs that just > executed 'sleep' to see how various scenarios would effect their fairshare > and priorities. > > Using MaxJobs and MaxSubmit in the accounting database is one way to limit > overly aggressive usage. We've done such limitations via a QOS, and then > we set that QOS as the user's default. We also use the preemptable > partitions to allow for aggressive use, at the cost of potentially being > preempted. The partitions that can't be preempted are all restricted on > how much they can monopolize the cluster. The QOS assigned to each > partition is automatically assigned to all jobs in that partition via > job_submit.lua plugin. We enforce some QOS usage this way so that > problematic users are always forced to use a specific QOS, and that QOS > value comes from their DefaultQOS set on their user record in accounting > database. > > In theory, if your FairShare is configured to account for this type of > aggressive usage, the person running those jobs would begin to have a lower > and lower priority. Then at some point those other users who had to wait > would begin to have a higher priority in relation to the aggressive user > and would no longer have to wait. > > Currently we use QOS to set limits but also split up our resources by > stakeholder partitions , non-stakeholder partitions and then the > "background" partitions (that can be preempted). Should testing of SUSPEND > preemption via QOS go well in 15.08 RC, we plan to move away from > stakeholder partitions and handle everything via QOS and FairShare > (FAIR_TREE). > > - Trey > > ============================= > > Trey Dockendorf > Systems Analyst I > Texas A&M University > Academy for Advanced Telecommunications and Learning Technologies > Phone: (979)458-2396 > Email: [email protected] > Jabber: [email protected] > > On Thu, Aug 27, 2015 at 4:55 PM, Gerry Creager - NOAA Affiliate < > [email protected]> wrote: > >> I'm running slurm on a Cray XE6m, and a relatively small one at that... >> with fair-share enabled. Today we had an episode where a single user was >> effectively snagging the entire machine with small, short (wallclock) jobs >> while all other users had to wait. >> >> He's using array control (--array=1-21), requesting 4 nodes, and >> generally a 1-hr wall clock limit. And occupying a significant portion of >> the machine. >> >> I've already restricted him (sacctmgr) to maxjobs=16, maxsubmit=32. If I >> understand the parameters correctly, that should reduce the number of jobs, >> and thus, nodes, he's consuming at a given time. >> >> I'm considering telling him to restrict the number of simultaneous array >> members to n=4, and reducing his maxnodes to 1, which will result in a >> slower run, but should still be managable. >> >> Are there other options/parameters I'm missing here, in configuring slurm >> for fair-share? It doesn't seem like it was a very fair distribution of >> resources... >> >> Thanks >> gerry >> -- >> Gerry Creager >> NSSL/CIMMS >> 405.325.6371 >> ++++++++++++++++++++++ >> “Big whorls have little whorls, >> That feed on their velocity; >> And little whorls have lesser whorls, >> And so on to viscosity.” >> Lewis Fry Richardson (1881-1953) >> > > -- Gerry Creager NSSL/CIMMS 405.325.6371 ++++++++++++++++++++++ “Big whorls have little whorls, That feed on their velocity; And little whorls have lesser whorls, And so on to viscosity.” Lewis Fry Richardson (1881-1953)
