This is on slurm 2.2.4.
We discovered that a few jobs that took very long time in the backfill
were specifying --ntasks-per-node and --mem-per-cpu such that only a few
nodes in our cluster had enough RAM. The job_req_node_filter function in
src/slurmctld/node_scheduler.c didn't filter out the nodes with too
little RAM for these jobs, so here is a suggested patch for it to do
so. We've tested it on our test cluster and are now using it on the
production cluster, and it seems to work well.
diff -B -c -r slurm-2.2.4/src/slurmctld/node_scheduler.c slurm-2.2.4.patched/src/slurmctld/node_scheduler.c
*** slurm-2.2.4/src/slurmctld/node_scheduler.c Wed Mar 23 18:00:05 2011
--- slurm-2.2.4.patched/src/slurmctld/node_scheduler.c Sat Apr 23 13:00:35 2011
***************
*** 1488,1493 ****
--- 1488,1497 ----
if ((detail_ptr->pn_min_cpus > config_ptr->cpus) ||
((detail_ptr->pn_min_memory & (~MEM_PER_CPU)) >
config_ptr->real_memory) ||
+ ((detail_ptr->pn_min_memory & (MEM_PER_CPU)) &&
+ ((detail_ptr->pn_min_memory & (~MEM_PER_CPU)) *
+ detail_ptr->pn_min_cpus) >
+ config_ptr->real_memory) ||
(detail_ptr->pn_min_tmp_disk >
config_ptr->tmp_disk)) {
bit_clear(avail_bitmap, i);
***************
*** 1507,1512 ****
--- 1511,1520 ----
if ((detail_ptr->pn_min_cpus > node_ptr->cpus) ||
((detail_ptr->pn_min_memory & (~MEM_PER_CPU)) >
node_ptr->real_memory) ||
+ ((detail_ptr->pn_min_memory & (MEM_PER_CPU)) &&
+ ((detail_ptr->pn_min_memory & (~MEM_PER_CPU)) *
+ detail_ptr->pn_min_cpus) >
+ node_ptr->real_memory) ||
(detail_ptr->pn_min_tmp_disk >
node_ptr->tmp_disk)) {
bit_clear(avail_bitmap, i);
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Research Computing Services, University of Oslo
