Hi,
You may increase the priority of the jobs which have gres definition
(you may do that in jobsubmit.lua script) , and allow partition to be
submitted both cpu and gpu jobs.. You may also make some extra
definitions in jobsubmit.lua tocompletely fit your needs
Sefa
On 12/15/2015 10:31 AM, Lennart Karlsson wrote:
Hello Hendryk,
You wrote:
we are using slurm 14.11 on a quite homogeneous cluster.
Nevertheless, some nodes are equipped with a GPU for interactive
graphic processing. These nodes usually idle over night. Since CPU,
memory and IB connect are the same as any compute node, it would be
beneficial to have these GPU nodes available for computing jobs over
night.
Currently, GPU nodes are within a separate partition with limitations
on MaxNodes and MaxTime - furthermore they are shared by default.
Do you know of a flexible way to expand/shrink a partition depending
on the time of day?
The following scenario would be great: GPU nodes are put into the
compute partition every evening; compute jobs can be scheduled on
these nodes as long as their expected end is before the next morning;
each morning GPU nodes are put back into their "interactive
partition" where no compute jobs can run.
Changing the partition in slurm.conf would always need a restart of
slurmctld which sounds a bit to heavy for this tasks, doesn't it?
I have at least a clumsy solution to a similar problem, if you receive
no simple and smart solutions. :-)
slurmctld needs not be restarted, because partitions may be updated
with command scontrol.
In cron, I do
# Tricks with the devel and devcore partitions
0 9-19 * * 1-5 root (/usr/bin/scontrol update part=devel state=up;
/usr/bin/scontrol update part=devcore state=up; /usr/bin/scontrol
update part=node nodes=m[214-248]; /usr/bin/scontrol update part=core
nodes=m[214-248])
0 0-8,20-23 * * 1-5 root (/usr/bin/scontrol update part=devel
state=inactive; /usr/bin/scontrol update part=devcore state=inactive;
/usr/bin/scontrol update part=node nodes=m[209-248]; /usr/bin/scontrol
update part=core nodes=m[209-248])
0 * * * 6-7 root (/usr/bin/scontrol update part=devel state=inactive;
/usr/bin/scontrol update part=devcore state=inactive;
/usr/bin/scontrol update part=node nodes=m[209-248]; /usr/bin/scontrol
update part=core nodes=m[209-248])
and I have reservations for monday-friday for the change (please note
the WEEKLY flag):
ReservationName=devel_prepare_monday StartTime=2015-12-21T08:59:00
EndTime=2015-12-21T09:04:00 Duration=00:05:00
Nodes=m[209-213] NodeCnt=5 CoreCnt=80 Features=(null)
PartitionName=core Flags=OVERLAP,IGNORE_JOBS,WEEKLY,SPEC_NODES,STATIC
TRES=cpu=80
Users=lka Accounts=(null) Licenses=(null) State=INACTIVE
BurstBuffer=(null) Watts=0
ReservationName=devel_prepare_tuesday StartTime=2015-12-15T08:59:00
EndTime=2015-12-15T09:04:00 Duration=00:05:00
Nodes=m[209-213] NodeCnt=5 CoreCnt=80 Features=(null)
PartitionName=core Flags=OVERLAP,IGNORE_JOBS,WEEKLY,SPEC_NODES,STATIC
TRES=cpu=80
Users=lka Accounts=(null) Licenses=(null) State=INACTIVE
BurstBuffer=(null) Watts=0
ReservationName=devel_prepare_wednesday StartTime=2015-12-16T08:59:00
EndTime=2015-12-16T09:04:00 Duration=00:05:00
Nodes=m[209-213] NodeCnt=5 CoreCnt=80 Features=(null)
PartitionName=core Flags=OVERLAP,IGNORE_JOBS,WEEKLY,SPEC_NODES,STATIC
TRES=cpu=80
Users=lka Accounts=(null) Licenses=(null) State=INACTIVE
BurstBuffer=(null) Watts=0
ReservationName=devel_prepare_thursday StartTime=2015-12-17T08:59:00
EndTime=2015-12-17T09:04:00 Duration=00:05:00
Nodes=m[209-213] NodeCnt=5 CoreCnt=80 Features=(null)
PartitionName=core Flags=OVERLAP,IGNORE_JOBS,WEEKLY,SPEC_NODES,STATIC
TRES=cpu=80
Users=lka Accounts=(null) Licenses=(null) State=INACTIVE
BurstBuffer=(null) Watts=0
ReservationName=devel_prepare_friday StartTime=2015-12-18T08:59:00
EndTime=2015-12-18T09:04:00 Duration=00:05:00
Nodes=m[209-213] NodeCnt=5 CoreCnt=80 Features=(null)
PartitionName=core Flags=OVERLAP,IGNORE_JOBS,WEEKLY,SPEC_NODES,STATIC
TRES=cpu=80
Users=lka Accounts=(null) Licenses=(null) State=INACTIVE
BurstBuffer=(null) Watts=0
In case you want to treat each day of the week in the same way, one
reservation
would of course be enough.
Cheers,
-- Lennart Karlsson, UPPMAX, Uppsala University, Sweden
http://www.uppmax.uu.se/
