Hi there,
Having read this:
---
reboot_nodes [NodeList]
Reboot all nodes in the system when they become idle using the
RebootProgram as configured in Slurm's slurm.conf file. Accepts an option list
of nodes to reboot. By default all nodes are rebooted. NOTE: This command does
not prevent additional jobs from being scheduled on these nodes, so many jobs
can be executed on the nodes prior to them being rebooted. You can explicitly
drain the nodes in order to reboot nodes as soon as possible, but the nodes
must also explicitly be returned to service after being rebooted. You can
alternately create an advanced reservation to prevent additional jobs from
being initiated on nodes to be rebooted. NOTE: Nodes will be placed in a state
of "MAINT" until rebooted and returned to service with a normal state.
Alternately the node's state "MAINT" may be cleared by using the scontrol
command to set the node state to "RESUME", which clears the "MAINT" flag.
---
Is there a good reason why nodes allow more jobs run before the nodes are
rebooted? Is this ever desirable behavior? I guess what might make the most
sense is to allow jobs that could also fit in the longest remaining walltime of
the currently-running jobs. Or maybe to make it configurable whether it will
disallow jobs to run while it waits to reboot. Currently, it would seem, it
just adds another step if you have any nodes that happen to be in the "mixed"
state. I imagine nodes that are in the "alloc" state take care of themselves,
unless they are multi-job nodes and one of the job finishes, moving it to
"mixed."
--
____ *Note: UMDNJ is now Rutgers-Biomedical and Health Sciences*
|| \\UTGERS |---------------------*O*---------------------
||_// Biomedical | Ryan Novosielski - Senior Technologist
|| \\ and Health | [email protected] - 973/972.0922 (2x0922)
|| \\ Sciences | OIRT/High Perf & Res Comp - MSB C630, Newark
`'
