Folks,
I am putting together a system where resources are generally managed by
Mesos and slurm runs in elastic mode (cloud mode) using Mesos (actually
Marathon) as its "cloud" provider. I have most of this working, but I
have one persistent problem that I can't seem to work around. That is,
when Mesos (Marathon) does not have the necessary resources available at
the time that a slurm job is launched, there does not seem to be a way
to tell slurmctld NOT to try to launch the job at this time (either
queue it or fail the launch would be acceptable).
What I currently do in this situation is set the elastic mode nodes that
cannot be resumed into DRAIN state to indicate that they are not
available. Unfortunately, this does not stop slurmctld from trying to
launch the job on the remaining elastic nodes that have become
available. This results in two possible outcomes:
- A job that runs on some but not all of its nodes and then hangs
waiting for the remaining nodes to run,
- A job that hangs before starting on any of its nodes and waits
for all the nodes to start,
Eventually, slurmctld will declare the offending nodes DOWN with a the
reason that they could not be reached, but this takes a long time.
I think the problem here is that slurmctld allocates elastic nodes while
they are still in the powered down state, instead of waiting for them to
become powered up and idle before actually trying to allocate them.
This means that a job that sees powered down and idle nodes gets well
into its launch before discovering that the nodes cannot actually be
used. At that point, there is no obvious way to push the job back off
the nodes.
This seems like a common enough kind of problem that there must be a
solution to it. Can someone suggest a way to tell slurmctld that the
job it is trying to launch on a node that is not going to become
available should either be aborted or queued for lack of resources?
Thanks!
Eric
