I just submitted bug 1586 about this. The slurmctld log does say it is immediately returned to service after running scontrol reboot_nodes for us as well.
On Tue, Apr 07, 2015 at 10:45:39AM -0700, Trey Dockendorf wrote: > I've also had similar issues with reboot_nodes. In my case the nodes will > return to service before the system has a chance to reboot and is then > marked offline with unexpected reboot rather than Maint. I'm on 14.03.10. > I went so far as to create a bash script that I set as RebootProgram that > first killed slurmd then rebooted, but the node still returned to service > before the reboot. > > - Trey > > ============================= > > Trey Dockendorf > Systems Analyst I > Texas A&M University > Academy for Advanced Telecommunications and Learning Technologies > Phone: (979)458-2396 > Email: [email protected] > Jabber: [email protected] > > On Tue, Apr 7, 2015 at 12:26 PM, Andy Wettstein <[email protected]> > wrote: > > > > > Ah, this might be a slightly different issue. I see the problem when > > issuing "scontrol reboot_nodes" I changed ResumeTimeout to 600, but that > > didn't make a difference for that. > > > > On Mon, Apr 06, 2015 at 02:53:05PM -0700, Moe Jette wrote: > > > > > > This was addressed several days ago. Configure ResumeTimeout to an > > > appropriate value > > > > > > > > > Quoting Andy Wettstein <[email protected]>: > > > > > > >I've noticed this problem as well. This was working as expected at some > > > >point in time (maybe still on 2.6). I haven't gotten around to filing a > > > >bug report, but maybe I'll do that this week. > > > > > > > >On Mon, Mar 30, 2015 at 10:17:09AM -0700, Scott Yockel wrote: > > > >> > > > >>Just adding the /var/log/message to this: > > > >> > > > >>slurmd[4526]: error: _step_connect: connect() failed dir > > > >>/var/slurmd/spool/slurmd node holy2a14107 job 36557200 step -2 > > > >>Connection refused > > > >> > > > >> > > > >>> On Mar 30, 2015, at 1:02 PM, Paul Edmon <[email protected]> > > wrote: > > > >>> > > > >>> > > > >>> So point of information. My impression is that when you use > > > >>this flag, the node/s are rebooted (or the script defined is run) > > > >>and then the job runs as per normal. Is this correct? Because I > > > >>think in our case it is rebooting the nodes but then the node is > > > >>defined as closed due to unexpected reboot. Is there a way to > > > >>suppress this when the node is rebooted by this flag? Obviously > > > >>the reboot wasn't unexpected as slurm was aware of it due to the > > > >>flag. > > > >>> > > > >>> -Paul Edmon- > > > > > > > >-- > > > >andy wettstein > > > >hpc system administrator > > > >research computing center > > > >university of chicago > > > >773.702.1104 > > > > > > > > > -- > > > Morris "Moe" Jette > > > CTO, SchedMD LLC > > > Commercial Slurm Development and Support > > > > -- > > andy wettstein > > hpc system administrator > > research computing center > > university of chicago > > 773.702.1104 > > -- andy wettstein hpc system administrator research computing center university of chicago 773.702.1104
