GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/11899
[SPARK-14075] Refactor MemoryStore to be testable independent of
BlockManager
This patch refactors the `MemoryStore` so that it can be tested without
needing to construct / mock an entire `BlockManager`.
- The block manager's serialization- and compression-related methods have
been moved from `BlockManager` to `SerializerManager`.
- `BlockInfoManager `is now passed directly to classes that need it, rather
than being passed via the `BlockManager`.
- The `MemoryStore` now calls `dropFromMemory` via a new
`BlockEvictionHandler` interface rather than directly calling the
`BlockManager`. This change helps to enforce a narrow interface between the
`MemoryStore` and `BlockManager` functionality and makes this interface easier
to mock in tests.
- Several of the block unrolling tests have been moved from
`BlockManagerSuite` into a new `MemoryStoreSuite`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark
reduce-memorystore-blockmanager-coupling
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11899.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11899
----
commit 5e71e96249adede8fe8984dda8e34eb9f7144e8c
Author: Josh Rosen <[email protected]>
Date: 2016-03-22T18:13:48Z
Stop indirectly accessing BlockManager's BlockInfoManager
commit 6478c348fab8bc61c5bf1146589f13dd9d94dd63
Author: Josh Rosen <[email protected]>
Date: 2016-03-22T18:57:53Z
Move compression and serialization methods out of BlockManager.
commit 83b5bdae8526fb86cb9b849377c4b9c5e32f05a5
Author: Josh Rosen <[email protected]>
Date: 2016-03-22T19:10:45Z
Introduce BlockEvictionHandler interface to decouple MemoryStore from
BlockManager interface.
commit 0716030824458b4bdf4acc027aa590f993e67bbb
Author: Josh Rosen <[email protected]>
Date: 2016-03-22T20:37:58Z
Move unroll-related tests to new MemoryStoreSuite
commit a02a3cb5c6f37aa9bc93790ad3af7e0dccc85ccd
Author: Josh Rosen <[email protected]>
Date: 2016-03-22T20:44:22Z
Add back MEMORY_AND_DISK tests so that path continues to be exercised.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
