@Kurt
You raise some good points. These are tricky issues indeed. Here are some
thoughts:
(1)
I think the resources required for a function can only be decided by the
user (at least in a first version).
If I recall correctly think Blink used annotations in Yarn to the user code
to define how
Let me rephrase my comment on the dispatcher.I mean that its API would be
job-centric, i.e. with operations like `execute(jobspec)` rather than
operations like `createSession` that the status-quo would suggest.
Since writing those comments I’ve put more time into developing the Mesos
@Aljoscha
I would not make the ResourceManager a subcomponent of the JobManager.
While that may be simpler initially, I would like to keep the door open to
let RM and JM run in different processes/nodes.
Also, for Yarn/Mesos sessions, the ResourceManager may run longer than the
JobManager.
On
Hi Eron!
Some comments on your comments:
*Dispatcher*
- The dispatcher should NOT be job-centric. The dispatcher should take
over the "multi job" responsibilities here, now that the JobManager is
single-job only.
- An abstract dispatcher would be great. It could implement the
connection/HTTP
Thanks for the great proposal.
There are still 2 issues i concerned with which i want to discuss with.
#1 Who should decide the resources one operator uses, user or framework?
Like how much cpu or memory will cost by my "map" operator, does it seem a
little bit too low level for the users,
+1
I don't have much to say since this already seems very well worked out.
Just some small remarks:
- This sentence that describes TaskManager behavior will probably have to
be adapted for FLIP-1, correct? "Loss of connection to the JobManager
results in triggering master-failure recovery
The design looks great - it solves for very diverse deployment modes, allows
for heterogeneous TMs, and promotes job isolation.
Some feedback:
*Dispatcher*
The dispatcher concept here expands nicely on what was introduced in the Mesos
design doc (MESOS-1984). The most significant difference
Hi Stephan,
Thanks for the nice wrap-up of ideas and discussions we had over the
last months (not all on the mailing list though because we were just
getting started with the FLIP process). The document is very
comprehensive and explains the changes in great details, even up to
the message
