Ah, this might be a slightly different issue. I see the problem when issuing "scontrol reboot_nodes" I changed ResumeTimeout to 600, but that didn't make a difference for that.
On Mon, Apr 06, 2015 at 02:53:05PM -0700, Moe Jette wrote: > > This was addressed several days ago. Configure ResumeTimeout to an > appropriate value > > > Quoting Andy Wettstein <[email protected]>: > > >I've noticed this problem as well. This was working as expected at some > >point in time (maybe still on 2.6). I haven't gotten around to filing a > >bug report, but maybe I'll do that this week. > > > >On Mon, Mar 30, 2015 at 10:17:09AM -0700, Scott Yockel wrote: > >> > >>Just adding the /var/log/message to this: > >> > >>slurmd[4526]: error: _step_connect: connect() failed dir > >>/var/slurmd/spool/slurmd node holy2a14107 job 36557200 step -2 > >>Connection refused > >> > >> > >>> On Mar 30, 2015, at 1:02 PM, Paul Edmon <[email protected]> wrote: > >>> > >>> > >>> So point of information. My impression is that when you use > >>this flag, the node/s are rebooted (or the script defined is run) > >>and then the job runs as per normal. Is this correct? Because I > >>think in our case it is rebooting the nodes but then the node is > >>defined as closed due to unexpected reboot. Is there a way to > >>suppress this when the node is rebooted by this flag? Obviously > >>the reboot wasn't unexpected as slurm was aware of it due to the > >>flag. > >>> > >>> -Paul Edmon- > > > >-- > >andy wettstein > >hpc system administrator > >research computing center > >university of chicago > >773.702.1104 > > > -- > Morris "Moe" Jette > CTO, SchedMD LLC > Commercial Slurm Development and Support -- andy wettstein hpc system administrator research computing center university of chicago 773.702.1104
