GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/9127
[SPARK-10984] [WIP] Simplify *MemoryManager class structure
This patch refactors the MemoryManager class structure. After #9000, Spark
had the following classes:
- MemoryManager
- StaticMemoryManager
- ExecutorMemoryManager
- TaskMemoryManager
- ShuffleMemoryManager
This is fairly confusing. To simplify things, this patch consolidates
several of these classes:
- ShuffleMemoryManager and ExecutorMemoryManager were merged into
MemoryManager.
- TaskMemoryManager is moved into Spark Core.
This patch is currently a work-in-progress; the remaining TODOs:
- [x] Merge ExecutorMemoryManager into MemoryManager.
- [ ] Merge ShuffleMemoryManager into MemoryManager.
- [x] Move TaskMemoryManager from `spark-unsafe` to `spark-core`.
- [ ] Refactor TaskMemoryManager interactions so tasks use only this and
not both this and ShuffleMemoryManager.
- [ ] Misc. docs and cleanup.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark SPARK-10984
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9127.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9127
----
commit b7c9c2395393e8f910648180b456d66731bdd403
Author: Josh Rosen <[email protected]>
Date: 2015-10-14T22:52:28Z
Move Unsafe mem. mgrs. to spark-core subproject.
commit 25ba4b54b0a88ed7f6db75a2f1d4a9b55b98b294
Author: Josh Rosen <[email protected]>
Date: 2015-10-14T23:47:08Z
Merge ExecutorMemoryManager into MemoryManager.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
