Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-85810614
@tnachen I just thought of something while I was reviewing the standalone
mode equivalent of this patch at #731. I wonder if it makes sense to allow the
user to specify all of the following:
- Memory per executor
- Max number of executors per slave
- Max cores per executor
For instance, if each slave has 20G, and memory per executor is 4G, then it
would seem reasonable to launch 5 executors on that slave and somehow
distribute the cores. However, now we require the user to also explicitly state
the number of executors they want per slave, which may not be 5. The
conflicting semantics of these configs here seem confusing to me.
Because Mesos (and standalone) mode greedily grabs all the cores in the
cluster, we need to limit the number of cores the first executor can take, so
it seems that some cap on cores per executor is necessary. However, another cap
on the number of executors per slave seems to add more complexity. Would it
make sense instead to just launch as many executors on each slave as the
available resources allow? (i.e. keep `spark.mesos.coarse.maxCoresPerExecutor`
but not the other config)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
