GitHub user CodingCat opened a pull request:
https://github.com/apache/spark/pull/636
SPARK-1706: Allow multiple executors per worker in Standalone mode
https://issues.apache.org/jira/browse/SPARK-1706
In current implementation, the user has to start multiple workers in a
server for starting multiple executors in a server, which introduces additional
overhead for more JVM processes...
In this patch, I changed the scheduling logic in master to enable the user
to start multiple executor processes within the same JVM process.
Other small change includes
1. change memoryPerSlave in ApplicationDescription to memoryPerExecutor,
as "Slave" is overrided to represent both worker and executor in the
documents... (we have some discussion on this before?)
@pwendell, I think we don't need to change anything in scheduler part, as
we indexed the executor by executorId instead of host IP address?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/CodingCat/spark SPARK-1706
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/636.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #636
----
commit 704696bcca28f7aff0f759551add6da8f17efc0d
Author: CodingCat <[email protected]>
Date: 2014-05-04T18:13:41Z
make master support multiple executors per worker
commit 9f8a8b20e40dae69fbd394593dde7179e562f504
Author: CodingCat <[email protected]>
Date: 2014-05-04T18:17:12Z
doc update
commit 13be5e52e3b0bccb1c120375d24638ebcd9f1543
Author: CodingCat <[email protected]>
Date: 2014-05-04T22:03:54Z
doc update
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
