GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/6394
[SPARK-4352][YARN][WIP] Incorporate locality preferences in dynamic
allocation requests
Currently there's no locality preference for container request in YARN
mode, this will affect the performance if fetching data remotely, so here
proposed to add locality in Yarn dynamic allocation mode.
Ping @sryza, please help to review, thanks a lot.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jerryshao/apache-spark SPARK-4352
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/6394.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6394
----
commit 328ba533396a49de1d1cf83d3b2c5aee22081c6b
Author: jerryshao <[email protected]>
Date: 2015-05-19T02:53:18Z
Support node locality for dynamic allocation initial commit
commit 424966413a11f8216ddfc533ce17c04d5dda232d
Author: jerryshao <[email protected]>
Date: 2015-05-20T09:18:18Z
Enable node locality information in YarnAllocator
commit 4fed42409378cb546e8b8f4afc53edf4d1d7676e
Author: jerryshao <[email protected]>
Date: 2015-05-21T07:25:00Z
Further polish the algorithm
commit 4cf5fba26c2e74dfeecaa467f13f61aa2f18e013
Author: jerryshao <[email protected]>
Date: 2015-05-22T06:29:36Z
Fix some bugs
commit 873ee9b8909c58a903113f4d07a0e5e4e55006ef
Author: jerryshao <[email protected]>
Date: 2015-05-25T06:53:42Z
Still fix the code
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
