In fact if the stuck user submits a job to the batch partition which includes all the nodes (sade[01-15] + many others) and has much lower priority, the jobs do enter immediately.
Now what's probably not ok in this case is that these batch jobs do enter immediately even if their runt time limit (-t) exceeds that of the sade1 jobs (3 days vs 12 hours). Backfill scheduling algorithm is turned on in slurm.conf. ________________________________________ From: Moe Jette [[email protected]] Sent: Tuesday, January 08, 2013 6:20 PM To: slurm-dev Subject: [slurm-dev] Re: Debug Stuck User Offhand I do not know the answer to your question, but perhaps creating three additional partitions (one for each user) that overlap the nodes in "sade1" would solve the problem. Quoting Guglielmi Matteo <[email protected]>: > Hello everyone. > > Here is something i find difficult to debug: > > 15 identical nodes "sade[01-15]" are part of a partition > called "sade1" > > 3 users were granted to run jobs on partition "sade1" > > the 3 users have decided to "manually subdivided" sade1 > among themselves using the "--exclude=nodelist" option: > > user1: #SBATCH --exclude=sade[06-15] > > user2: #SBATCH --exclude=sade[01-05,11-15] > > user2: #SBATCH --exclude=sade[01-10] > > 2 qos were defined: > "qos1" adds 5000 pts to the total priority count (default) > "qos2" adds 10000 pts to the total priority count > > total job's priority includes also a bit of job aging coeff: > PriorityWeightAge=1000 vs PriorityWeightQOS=10000 > > Problem: > > if all the three users do use the default qos (qos1) there > is always one user having all of his jobs in pending state > with reason "priority". > > if the stuck user goes for "qos2", jobs do enter immediately. > > Please note that the --exclude options do make sure to have > three disjoint "sub partitions". > > > How can I find if this is a bug or some error in my slurm.conf? > > I'm also using slurmdbd + cgroups... I'll post anything useful if needed. >
