Odds are the backfill loop is not penetrating far enough into the
queue. Recall that slurm has two scheduling loops. The primary is the
faster one that only penetrates as far as it can schedule. Thus in this
case the primary loop would stop immediately on the GPU jobs that it
can't schedule. Thus it would be up to the backfill loop to fill in the
gaps. So I would make sure that your backfill loop is actually doing
that. It will either be the case that the backfill loop isn't going
deep enough into the queue to pick up the cpu jobs that can run, or it
has adjudicated that those jobs can't run due to some vargarity in logic
(typically because it thinks that it won't fit due to time constraints).
Anyways that's where I would start.
-Paul Edmon-
On 7/3/2018 5:22 PM, Christopher Benjamin Coffey wrote:
Hello!
We are having an issue with high priority gpu jobs blocking low priority cpu
only jobs.
Our cluster is setup with one partition, "all". All nodes reside in this
cluster. In this all partition we have four generations of compute nodes, including gpu
nodes. We do this to make use of those unused cores on the gpu nodes for compute only
jobs. We schedule the various different generations, and gpu nodes by the user specifying
a constraint (if they care), and a --qos=gpu / --gres=gpu:tesla:1 for gpu nodes. The gpu
qos will give the jobs the highest priority in the queue, so that they can get scheduled
sooner onto the limited resource that we have in gpu's. So this has worked out real nice
for quite some time. But lately we've noticed that the gpu jobs are blocking the cpu only
jobs. Yes, the gpu jobs have higher priority, yet, the gpu jobs can only run on a very
small subset of nodes compared to the cpu only jobs. But it appears that slurm isn't
taking into consideration the limited set of nodes the gpu job can run on. That’s the
only possibility that I see to the gpu jobs blocking the cpu only jobs. I'm not sure if
this is due to a recent slurm change, or if we just never noticed, but its definitely
happening.
For example, the behavior happens in the following scenario
- 15 compute nodes (no gpus) are idle
- All of the gpus are occupied
- 1000's of low priority compute only jobs in the pending queue
- 100's of highest priority gpu jobs in the pending queue
In the above scenario, the above low priority jobs are not backfilled, or
started, yet compute only nodes remain idle. If I hold the gpu jobs, the lower
priority compute only jobs are then started.
Anyone seen this? Am I thinking about this wrong? I would think that slurm
should not be considering the nodes with no gpus to fulfill the gpu jobs.
I have an idea how to fix this scenario, but I think our current config should
work. The fix I am mulling over is to create a gpu partition, and place the gpu
nodes into that partition. Then, use the all_partitions job submit plugin to
schedule compute only jobs into both partitions. The gpu jobs would then only
land in the gpu partition. I'd think that would definitely fix the issue, but
maybe there is a down side. Yet, I think how we have it should be working!?
Thanks for your advice!
Best,
Chris
—
Christopher Coffey
High-Performance Computing
Northern Arizona University
928-523-1167
