Michael Gummelt created SPARK-14822:
---------------------------------------
Summary: Add lazy executor startup to Mesos Scheduler
Key: SPARK-14822
URL: https://issues.apache.org/jira/browse/SPARK-14822
Project: Spark
Issue Type: Task
Components: Mesos
Reporter: Michael Gummelt
As we deprecate fine-grained mode, we need to make sure we have alternative
solutions for its benefits.
Its two benefits are:
0. lazy executor startup
In fine-grained mode, executors are brought up only as tasks are scheduled.
This means that a user doesn't have to set {{spark.cores.max}} to ensure that
the app doesn't consume all resources in the cluster.
1. relinquishing cores
As Spark tasks terminate, the mesos task it was bound to terminates as well,
thus relinquishing the cores assigned to it.
I'd like to add {{0.}} to coarse-grained mode, possibly enabled with a
configuration param. If https://issues.apache.org/jira/browse/MESOS-1279 ever
happens, we can add {{1.}} as well.
cc [~tnachen] [~dragos] [~skonto] [~andrewor14]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
