Thanks you for your analysis of the problem and the patch. It will be
in version 14.11.8 when released (probably in late June). The commit
with your patch is here:
https://github.com/SchedMD/slurm/commit/3c2b46af01c50bd03c85235d1aaeb75acd62bb20
Quoting Didier GAZEN <[email protected]>:
Hi,
In your node_mgr fix to keep rebooted nodes down (commit 9cd15dfe96b54), you
forgot to consider the case of nodes that are powered up but are
responding after
ResumeTimeout seconds (the maximum time permitted). Such nodes are
marked DOWN (because they didn't respond within ResumeTimeout seconds) than
should become silently available when ReturnToService=1 (as stated
in the slurm.conf manual)
With your modification when such nodes are finally responding, they
are seen as
rebooted nodes and remain in the DOWN state (with the new reason: Node
unexpectedly rebooted) even when ReturnToService=1 !
My patch to obtain the correct behaviour:
diff --git a/src/slurmctld/node_mgr.c b/src/slurmctld/node_mgr.c
index 9a22784..1644ba7 100644
--- a/src/slurmctld/node_mgr.c
+++ b/src/slurmctld/node_mgr.c
@@ -2291,8 +2291,9 @@ extern int
validate_node_specs(slurm_node_registration_status_msg_t *reg_msg,
!xstrcmp(node_ptr->reason, "Scheduled reboot") ||
((slurmctld_conf.ret2service == 1) &&
!xstrcmp(node_ptr->reason, "Not responding") &&
- (node_ptr->boot_time <
- node_ptr->last_response)))) {
+ ((node_ptr->boot_time <
+ node_ptr->last_response) ||
+ IS_NODE_POWER_UP(node_ptr))))) {
if (reg_msg->job_count) {
node_ptr->node_state = NODE_STATE_ALLOCATED |
node_flags;
Didier
--
Morris "Moe" Jette
CTO, SchedMD LLC
Commercial Slurm Development and Support
