GitHub user andrewor14 opened a pull request:
https://github.com/apache/spark/pull/9000
[SPARK-10956] Common MemoryManager interface for storage and execution
This patch introduces a `MemoryManager` that is the central arbiter of how
much memory to grant to storage and execution. This patch is primarily
concerned only with refactoring while preserving the existing behavior as much
as possible.
This is the first step away from the existing rigid separation of storage
and execution memory, which has several major drawbacks discussed on the
[issue](https://issues.apache.org/jira/browse/SPARK-10956). It is the precursor
of a series of patches that will attempt to address those drawbacks.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/andrewor14/spark memory-manager
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9000.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9000
----
commit 9e73981280b6dfdfdd6132e2a213b6c4a95c026c
Author: Andrew Or <[email protected]>
Date: 2015-09-30T23:57:06Z
Introduce MemoryManager + StaticMemoryManager
This commit does not represent any change in behavior. The
MemoryManager's introduced are not actually used anywhere yet.
This will come in the next commit.
commit b5df241669f6679742ec782d41b96174967f3911
Author: Andrew Or <[email protected]>
Date: 2015-10-01T01:32:48Z
Derive max memory from MemoryManager
This commit refactors BlockManager and ShuffleMemoryManager to
take in a MemoryManager in their constructors, such that the max
memory is derived from the MemoryManager instead of from a Long
that is passed around everywhere.
Note that no memory acquisition or release goes through the
MemoryManager yet. This will come in future commits.
commit 700e5d12c2f644a48f067f0e344f46aeabbc40c7
Author: Andrew Or <[email protected]>
Date: 2015-10-01T20:29:17Z
Route ShuffleMemoryManager through new interface
All execution memory is now acquired through the new MemoryManager
interface as of this commit. The next step is to do it for storage
memory.
commit 933de614df6e84d7776aa1d11ae1d82042f3f6dd
Author: Andrew Or <[email protected]>
Date: 2015-10-05T22:29:38Z
Route MemoryStore through new interface
All storage memory is now acquired through the new MemoryManager
interface. This requires non-trivial refactoring due to the fact
that the unrolling and eviction logic are closely intertwined.
commit e73e463ccaf8220387903b59b9a6ab71662d2ee7
Author: Andrew Or <[email protected]>
Date: 2015-10-05T23:57:41Z
Limit scope of synchronized blocks to avoid deadlocks
commit 52e201480e07974e9d13936d779b7c57e7c349a1
Author: Andrew Or <[email protected]>
Date: 2015-10-06T01:01:36Z
Remove ResultWithDroppedBlocks
It was an awkward case class that didn't really do much. Instead
we'll pass a mutable list into the methods to get the dropped
blocks.
commit 88faede150c83b197d17f0571fa2af2503a76179
Author: Andrew Or <[email protected]>
Date: 2015-10-06T19:17:28Z
Add unit test for StaticMemoryManager
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
