I'd like a way to be able to limit the number of jobs that a user is allowed to run before we only allow them to run by backfilling.
For example, let's say we'd like to allow users to run lots of jobs, but only allow them to "reserve" resources for their first few jobs. That way, a user with no jobs running submitting a job requesting 50 nodes will have their job start reserving nodes until the job starts. Once some number of jobs have started for this user, I'd still like to allow them to run, but I'd like to only allow them to run if their jobs can fit in whatever backfill window is available. Once the number of running jobs falls below the threshold, another of their jobs (or maybe several) would be allowed to begin reserving resources in the backfill schedule's map, until another of their jobs starts, at which point the rest would be relegated to backfilling only again. We've been using the Moab soft/hard limits for this functionality, but I'd like to be able to do the same thing using Slurm directly. I've looked at QOS and while the NoReserve flag kind of describes what I'm looking for, I only want it to apply to the jobs in the queue once they've reached their running jobs threshold, I couldn't see how to make that work though. Any thoughts or other options to accomplish something like this with Slurm? ----- Gary Skouson
