Hello Hendryk,
You wrote:
we are using slurm 14.11 on a quite homogeneous cluster. Nevertheless, some nodes are equipped with a GPU for interactive graphic processing. These nodes usually idle over night. Since CPU, memory and IB connect are the same as any compute node, it would be beneficial to have these GPU nodes available for computing jobs over night. Currently, GPU nodes are within a separate partition with limitations on MaxNodes and MaxTime - furthermore they are shared by default. Do you know of a flexible way to expand/shrink a partition depending on the time of day? The following scenario would be great: GPU nodes are put into the compute partition every evening; compute jobs can be scheduled on these nodes as long as their expected end is before the next morning; each morning GPU nodes are put back into their "interactive partition" where no compute jobs can run. Changing the partition in slurm.conf would always need a restart of slurmctld which sounds a bit to heavy for this tasks, doesn't it?
I have at least a clumsy solution to a similar problem, if you receive no simple and smart solutions. :-) slurmctld needs not be restarted, because partitions may be updated with command scontrol. In cron, I do # Tricks with the devel and devcore partitions 0 9-19 * * 1-5 root (/usr/bin/scontrol update part=devel state=up; /usr/bin/scontrol update part=devcore state=up; /usr/bin/scontrol update part=node nodes=m[214-248]; /usr/bin/scontrol update part=core nodes=m[214-248]) 0 0-8,20-23 * * 1-5 root (/usr/bin/scontrol update part=devel state=inactive; /usr/bin/scontrol update part=devcore state=inactive; /usr/bin/scontrol update part=node nodes=m[209-248]; /usr/bin/scontrol update part=core nodes=m[209-248]) 0 * * * 6-7 root (/usr/bin/scontrol update part=devel state=inactive; /usr/bin/scontrol update part=devcore state=inactive; /usr/bin/scontrol update part=node nodes=m[209-248]; /usr/bin/scontrol update part=core nodes=m[209-248]) and I have reservations for monday-friday for the change (please note the WEEKLY flag): ReservationName=devel_prepare_monday StartTime=2015-12-21T08:59:00 EndTime=2015-12-21T09:04:00 Duration=00:05:00 Nodes=m[209-213] NodeCnt=5 CoreCnt=80 Features=(null) PartitionName=core Flags=OVERLAP,IGNORE_JOBS,WEEKLY,SPEC_NODES,STATIC TRES=cpu=80 Users=lka Accounts=(null) Licenses=(null) State=INACTIVE BurstBuffer=(null) Watts=0 ReservationName=devel_prepare_tuesday StartTime=2015-12-15T08:59:00 EndTime=2015-12-15T09:04:00 Duration=00:05:00 Nodes=m[209-213] NodeCnt=5 CoreCnt=80 Features=(null) PartitionName=core Flags=OVERLAP,IGNORE_JOBS,WEEKLY,SPEC_NODES,STATIC TRES=cpu=80 Users=lka Accounts=(null) Licenses=(null) State=INACTIVE BurstBuffer=(null) Watts=0 ReservationName=devel_prepare_wednesday StartTime=2015-12-16T08:59:00 EndTime=2015-12-16T09:04:00 Duration=00:05:00 Nodes=m[209-213] NodeCnt=5 CoreCnt=80 Features=(null) PartitionName=core Flags=OVERLAP,IGNORE_JOBS,WEEKLY,SPEC_NODES,STATIC TRES=cpu=80 Users=lka Accounts=(null) Licenses=(null) State=INACTIVE BurstBuffer=(null) Watts=0 ReservationName=devel_prepare_thursday StartTime=2015-12-17T08:59:00 EndTime=2015-12-17T09:04:00 Duration=00:05:00 Nodes=m[209-213] NodeCnt=5 CoreCnt=80 Features=(null) PartitionName=core Flags=OVERLAP,IGNORE_JOBS,WEEKLY,SPEC_NODES,STATIC TRES=cpu=80 Users=lka Accounts=(null) Licenses=(null) State=INACTIVE BurstBuffer=(null) Watts=0 ReservationName=devel_prepare_friday StartTime=2015-12-18T08:59:00 EndTime=2015-12-18T09:04:00 Duration=00:05:00 Nodes=m[209-213] NodeCnt=5 CoreCnt=80 Features=(null) PartitionName=core Flags=OVERLAP,IGNORE_JOBS,WEEKLY,SPEC_NODES,STATIC TRES=cpu=80 Users=lka Accounts=(null) Licenses=(null) State=INACTIVE BurstBuffer=(null) Watts=0 In case you want to treat each day of the week in the same way, one reservation would of course be enough. Cheers, -- Lennart Karlsson, UPPMAX, Uppsala University, Sweden http://www.uppmax.uu.se/
