Jagadish created MESOS-4616:
-------------------------------
Summary: Support specifying a preferred host with a Resource
Request
Key: MESOS-4616
URL: https://issues.apache.org/jira/browse/MESOS-4616
Project: Mesos
Issue Type: Story
Reporter: Jagadish
When stateful services like Apache Samza, Kafka must be restarted with using
Mesos, the framework must have a way of specifying a preferred host with the
request.
More background:
I work on Apache Samza , a distributed stream processing framework. Currently
Samza supports only Yarn as a resource manager. (there have been requests to
run Samza with mesos). A cluster (200 nodes 'ish) runs many Samza Jobs (about
3500). Each Samza Job has its own framework that requests resources
(containers) for the job to run. Each such container uses GBs of local state .
When such a container(resource) is started on a different host by the
framework, the local state must be re-bootstrapped. (this results in a long
bootstrap time, which is essentially down time).
The same is true for Apache Kafka, a distributed pub-sub logging system. When
a Kafka broker must be restarted by the framework, it should ideally be
re-started on the same host. (otherwise, each broker has to re-bootstrap
several GBs of logs from its peers before it can start to service a request.)
I'm sure many stateful services have similar requirements.
My framework can update/ filter resource offers, but with many frameworks
(thousands) on several nodes, I'm concerned about the wait-time for a
particular available resource to be offered to a framework that needs it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
