Timothy Chen created SPARK-5095:
-----------------------------------
Summary: Support launching multiple mesos executors in coarse
grained mesos mode
Key: SPARK-5095
URL: https://issues.apache.org/jira/browse/SPARK-5095
Project: Spark
Issue Type: Improvement
Reporter: Timothy Chen
Currently in coarse grained mesos mode, it's expected that we only launch one
Mesos executor that launches one JVM process to launch multiple spark executors.
However, this become a problem when the JVM process launched is larger than an
ideal size (30gb is recommended value from databricks), which causes GC
problems reported on the mailing list.
We should support launching mulitple executors when large enough resources are
available for spark to use, and these resources are still under the configured
limit.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
