Sorry Moe,
I am afraid I have sent you a bad patch.
The good one is :
diff --git a/src/slurmctld/node_mgr.c b/src/slurmctld/node_mgr.c
index 9a22784..a6771ef 100644
--- a/src/slurmctld/node_mgr.c
+++ b/src/slurmctld/node_mgr.c
@@ -2226,6 +2226,8 @@ extern int
validate_node_specs(slurm_node_registration_status_msg_t *reg_msg,
}
if (IS_NODE_NO_RESPOND(node_ptr)) {
+ if (IS_NODE_POWER_UP(node_ptr))
+ node_ptr->last_response = now;
node_ptr->node_state &= (~NODE_STATE_NO_RESPOND);
node_ptr->node_state &= (~NODE_STATE_POWER_UP);
last_node_update = time (NULL);
>
> Thanks you for your analysis of the problem and the patch. It will be
> in version 14.11.8 when released (probably in late June). The commit
> with your patch is here:
> https://github.com/SchedMD/slurm/commit/3c2b46af01c50bd03c85235d1aaeb75acd62bb20
>
> Quoting Didier GAZEN <[email protected]>:
>> Hi,
>>
>> In your node_mgr fix to keep rebooted nodes down (commit 9cd15dfe96b54),
>> you
>> forgot to consider the case of nodes that are powered up but are
>> responding after
>> ResumeTimeout seconds (the maximum time permitted). Such nodes are
>> marked DOWN (because they didn't respond within ResumeTimeout seconds)
>> than
>> should become silently available when ReturnToService=1 (as stated
>> in the slurm.conf manual)
>>
>> With your modification when such nodes are finally responding, they
>> are seen as
>> rebooted nodes and remain in the DOWN state (with the new reason: Node
>> unexpectedly rebooted) even when ReturnToService=1 !
>>
>> My patch to obtain the correct behaviour:
>>
>> diff --git a/src/slurmctld/node_mgr.c b/src/slurmctld/node_mgr.c
>> index 9a22784..1644ba7 100644
>> --- a/src/slurmctld/node_mgr.c
>> +++ b/src/slurmctld/node_mgr.c
>> @@ -2291,8 +2291,9 @@ extern int
>> validate_node_specs(slurm_node_registration_status_msg_t *reg_msg,
>> !xstrcmp(node_ptr->reason, "Scheduled
>> reboot") ||
>> ((slurmctld_conf.ret2service == 1) &&
>> !xstrcmp(node_ptr->reason, "Not
>> responding") &&
>> - (node_ptr->boot_time <
>> - node_ptr->last_response)))) {
>> + ((node_ptr->boot_time <
>> + node_ptr->last_response) ||
>> + IS_NODE_POWER_UP(node_ptr))))) {
>> if (reg_msg->job_count) {
>> node_ptr->node_state =
>> NODE_STATE_ALLOCATED |
>> node_flags;
>>
>>
>> Didier
>
>
> --
> Morris "Moe" Jette
> CTO, SchedMD LLC
> Commercial Slurm Development and Support
>
>
