Jack Crawford created MESOS-8209:
------------------------------------
Summary: mesos master should revoke offers when executor state
changes
Key: MESOS-8209
URL: https://issues.apache.org/jira/browse/MESOS-8209
Project: Mesos
Issue Type: Bug
Reporter: Jack Crawford
Currently, the mesos master does not revoke offers when the number of executors
on an agent decreases. This is a problem under certain conditions, such when
running a workflow that starts lots of small tasks on agents, with a one
executor per task model, a master that does not revoke resources after a set
amount of time, and a scheduler that does not reject resources.
The problem is that when running a mono-scheduler framework (which you might
want to do to easily enforce authentication requirements, have a full view of
all scheduled tasks, etc), in order to respond instantly when new tasks come in
I have the scheduler simply hang on to all resource offers it receives, and the
master is set to never revoke offers. This way the scheduler always has a pool
of resources to quickly service new requests as they come in.
However, if you start tasks fast enough, the agents can fill up with executors,
making it appear as there are no resources available for the scheduler to use.
Ive seen this on r4.4xlarge machines on aws with executors that consume 0.1
cpus, 32mb mem where the entire machine will be appear to be filled with
executors according to the master resource offers. The executors are exiting
(just after the task finishes), but the resources are not reclaimed because the
master does not revoke the outstanding resource offers to reflect the change.
You can replicate this pretty easily if you schedule tasks that finish
instantly with a 1-1 executor to task ratio. I find that if I schedule ~1000
tasks this way on a single machine, usually 600-700 will finish before all the
resource offers to the scheduler fill up and the agent appears to be "full" of
executors.
Changing the scheduler/master to periodically reject/revoke resources fixes the
problem.
My suggestion is for the master to revoke and reissue resource offers when the
executor count changes on an agent.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
