Alex,
There are definitely a couple of bugs with respect to the NoReserve
QOS. I believe the attached patch will fix these problems (based upon
your patch plus moving where the qos_ptr is set). I updated the
documentation too.
Thanks,
Moe
Quoting Alejandro Lucero Palau <[email protected]>:
Hi,
Running simulator with a long trace shows a bug in the backfilling code.
Although I'm using a 2.2.6 version it seems it remains in 2.3.
Line number 570 at plugins/sched/backfill/backfill.c checks for a job
being from a qos with NoReserve flags on, but qos_ptr variable is
updated just at the end of the loop so when used this is pointing to a
wrong job. I do not add any code for solving this in the patch attached.
So, line 571 modifies time_limit for a job to 1 minute. I can not
understand why this is done since it can lead to a job from a NoReserve
qos overtaking a more priority job. Maybe there's a reason for this but
I can not see it.
This modification needs to be changed back to avoid a job runnig with
the wrong time_limit value, but this is not done in all the places.
Patch attached solves this problem.
WARNING / LEGAL TEXT: This message is intended only for the use of the
individual or entity to which it is addressed and may contain
information which is privileged, confidential, proprietary, or exempt
from disclosure under applicable law. If you are not the intended
recipient or the person responsible for delivering the message to the
intended recipient, you are strictly prohibited from disclosing,
distributing, copying, or in any way using this message. If you have
received this communication in error, please notify the sender and
destroy and delete any copies you may have received.
http://www.bsc.es/disclaimer.htm
diff --git a/doc/man/man1/sacctmgr.1 b/doc/man/man1/sacctmgr.1
index 9fe7558..35c1186 100644
--- a/doc/man/man1/sacctmgr.1
+++ b/doc/man/man1/sacctmgr.1
@@ -914,12 +914,12 @@ any jobs submitted with this QOS that fall below the UsageThreshold
will be held until their Fairshare Usage goes above the Threshold.
.TP
\fINoReserve\fP
-If set, and using backfill, jobs using this QOS will all be considered
-at the same level within this QOS meaning if a larger, higher priority
-job is unable to run a smaller job will run if possible even if the
-larger higher priority job will be delayed starting.
-NOTE: This could cause starvation on these larger jobs, but if that is
-ok, this flag most likely will increase utilization.
+If this flag is set and backfill scheduling is used, jobs using this QOS will
+not reserve resources in thebackfill schedule's map of resources allocated
+through time. This flag is intended for use with a QOS that may be preempted
+by jobs associated with all other QOS (e.g use with a "standby" QOS). If the
+allocated is used with a QOS which can not be preempted by all other QOS, it
+could result in starvation of larger jobs.
.TP
\fIPartitionMaxNodes\fP
If set jobs using this QOS will be able to
diff --git a/src/plugins/sched/backfill/backfill.c b/src/plugins/sched/backfill/backfill.c
index f1471ee..513c38a 100644
--- a/src/plugins/sched/backfill/backfill.c
+++ b/src/plugins/sched/backfill/backfill.c
@@ -567,6 +567,7 @@ static int _attempt_backfill(void)
}
comp_time_limit = time_limit;
orig_time_limit = job_ptr->time_limit;
+ qos_ptr = job_ptr->qos_ptr;
if (qos_ptr && (qos_ptr->flags & QOS_FLAG_NO_RESERVE))
time_limit = job_ptr->time_limit = 1;
else if (job_ptr->time_min && (job_ptr->time_min < time_limit))
@@ -639,6 +640,7 @@ static int _attempt_backfill(void)
debug("backfill: system state changed, "
"breaking out");
rc = 1;
+ job_ptr->time_limit = orig_time_limit;
break;
} else {
this_sched_timeout += sched_timeout;
@@ -664,8 +666,11 @@ static int _attempt_backfill(void)
}
if (job_ptr->start_time <= now) {
int rc = _start_job(job_ptr, resv_bitmap);
- if (qos_ptr && (qos_ptr->flags & QOS_FLAG_NO_RESERVE))
+ if (qos_ptr && (qos_ptr->flags & QOS_FLAG_NO_RESERVE)){
job_ptr->time_limit = orig_time_limit;
+ job_ptr->end_time = job_ptr->start_time +
+ (orig_time_limit * 60);
+ }
else if ((rc == SLURM_SUCCESS) && job_ptr->time_min) {
/* Set time limit as high as possible */
job_ptr->time_limit = comp_time_limit;
@@ -724,7 +729,6 @@ static int _attempt_backfill(void)
/*
* Add reservation to scheduling table if appropriate
*/
- qos_ptr = job_ptr->qos_ptr;
if (qos_ptr && (qos_ptr->flags & QOS_FLAG_NO_RESERVE))
continue;
bit_not(avail_bitmap);
