I've also had similar issues with reboot_nodes. In my case the nodes will return to service before the system has a chance to reboot and is then marked offline with unexpected reboot rather than Maint. I'm on 14.03.10. I went so far as to create a bash script that I set as RebootProgram that first killed slurmd then rebooted, but the node still returned to service before the reboot.
- Trey ============================= Trey Dockendorf Systems Analyst I Texas A&M University Academy for Advanced Telecommunications and Learning Technologies Phone: (979)458-2396 Email: [email protected] Jabber: [email protected] On Tue, Apr 7, 2015 at 12:26 PM, Andy Wettstein <[email protected]> wrote: > > Ah, this might be a slightly different issue. I see the problem when > issuing "scontrol reboot_nodes" I changed ResumeTimeout to 600, but that > didn't make a difference for that. > > On Mon, Apr 06, 2015 at 02:53:05PM -0700, Moe Jette wrote: > > > > This was addressed several days ago. Configure ResumeTimeout to an > > appropriate value > > > > > > Quoting Andy Wettstein <[email protected]>: > > > > >I've noticed this problem as well. This was working as expected at some > > >point in time (maybe still on 2.6). I haven't gotten around to filing a > > >bug report, but maybe I'll do that this week. > > > > > >On Mon, Mar 30, 2015 at 10:17:09AM -0700, Scott Yockel wrote: > > >> > > >>Just adding the /var/log/message to this: > > >> > > >>slurmd[4526]: error: _step_connect: connect() failed dir > > >>/var/slurmd/spool/slurmd node holy2a14107 job 36557200 step -2 > > >>Connection refused > > >> > > >> > > >>> On Mar 30, 2015, at 1:02 PM, Paul Edmon <[email protected]> > wrote: > > >>> > > >>> > > >>> So point of information. My impression is that when you use > > >>this flag, the node/s are rebooted (or the script defined is run) > > >>and then the job runs as per normal. Is this correct? Because I > > >>think in our case it is rebooting the nodes but then the node is > > >>defined as closed due to unexpected reboot. Is there a way to > > >>suppress this when the node is rebooted by this flag? Obviously > > >>the reboot wasn't unexpected as slurm was aware of it due to the > > >>flag. > > >>> > > >>> -Paul Edmon- > > > > > >-- > > >andy wettstein > > >hpc system administrator > > >research computing center > > >university of chicago > > >773.702.1104 > > > > > > -- > > Morris "Moe" Jette > > CTO, SchedMD LLC > > Commercial Slurm Development and Support > > -- > andy wettstein > hpc system administrator > research computing center > university of chicago > 773.702.1104 >
