You may have to adjust how you prioritize jobs by size and wallclock time. PriorityFavorSmall=NO PriorityFlags=DEPTH_OBLIVIOUS,SMALL_RELATIVE_TO_TIME
The PriorityFavorSmall and SMALL_RELATIVE_TO_TIME is used by us to favor both short and large jobs. So if two jobs are equal in size, the shorter of the two is favored. Also if two jobs are equal in time, the larger is favored. We use this as a way to get short jobs in and out of the queues quickly as well as help large jobs (typically MPI) have priority over small serial jobs. How much weight the job size is given for priority will be determined also by PriorityWeightJobSize. A small job getting in before big jobs also has to do with how backfill is configured. If the small job starting won't impact a higher priority large job's start time, it should get backfilled. You can use 'sshare' to view the fair share values for users. You can use 'sprio' to inspect a pending job's various priority factors. So if you inspect this particular user's jobs against that of someone who had to wait, you may see why SLURM favored those jobs. One thing I did when planning our FairShare was create a handful of temporary accounts to represent our various users and then submit batches of jobs that just executed 'sleep' to see how various scenarios would effect their fairshare and priorities. Using MaxJobs and MaxSubmit in the accounting database is one way to limit overly aggressive usage. We've done such limitations via a QOS, and then we set that QOS as the user's default. We also use the preemptable partitions to allow for aggressive use, at the cost of potentially being preempted. The partitions that can't be preempted are all restricted on how much they can monopolize the cluster. The QOS assigned to each partition is automatically assigned to all jobs in that partition via job_submit.lua plugin. We enforce some QOS usage this way so that problematic users are always forced to use a specific QOS, and that QOS value comes from their DefaultQOS set on their user record in accounting database. In theory, if your FairShare is configured to account for this type of aggressive usage, the person running those jobs would begin to have a lower and lower priority. Then at some point those other users who had to wait would begin to have a higher priority in relation to the aggressive user and would no longer have to wait. Currently we use QOS to set limits but also split up our resources by stakeholder partitions , non-stakeholder partitions and then the "background" partitions (that can be preempted). Should testing of SUSPEND preemption via QOS go well in 15.08 RC, we plan to move away from stakeholder partitions and handle everything via QOS and FairShare (FAIR_TREE). - Trey ============================= Trey Dockendorf Systems Analyst I Texas A&M University Academy for Advanced Telecommunications and Learning Technologies Phone: (979)458-2396 Email: [email protected] Jabber: [email protected] On Thu, Aug 27, 2015 at 4:55 PM, Gerry Creager - NOAA Affiliate < [email protected]> wrote: > I'm running slurm on a Cray XE6m, and a relatively small one at that... > with fair-share enabled. Today we had an episode where a single user was > effectively snagging the entire machine with small, short (wallclock) jobs > while all other users had to wait. > > He's using array control (--array=1-21), requesting 4 nodes, and generally > a 1-hr wall clock limit. And occupying a significant portion of the machine. > > I've already restricted him (sacctmgr) to maxjobs=16, maxsubmit=32. If I > understand the parameters correctly, that should reduce the number of jobs, > and thus, nodes, he's consuming at a given time. > > I'm considering telling him to restrict the number of simultaneous array > members to n=4, and reducing his maxnodes to 1, which will result in a > slower run, but should still be managable. > > Are there other options/parameters I'm missing here, in configuring slurm > for fair-share? It doesn't seem like it was a very fair distribution of > resources... > > Thanks > gerry > -- > Gerry Creager > NSSL/CIMMS > 405.325.6371 > ++++++++++++++++++++++ > “Big whorls have little whorls, > That feed on their velocity; > And little whorls have lesser whorls, > And so on to viscosity.” > Lewis Fry Richardson (1881-1953) >
