Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/10170#discussion_r47024437
--- Diff:
core/src/test/scala/org/apache/spark/memory/UnifiedMemoryManagerSuite.scala ---
@@ -143,7 +148,25 @@ class UnifiedMemoryManagerSuite extends
MemoryManagerSuite with PrivateMethodTes
assert(mm.acquireExecutionMemory(400L, taskAttemptId,
MemoryMode.ON_HEAP) === 300L)
assert(mm.executionMemoryUsed === 600L)
assert(mm.storageMemoryUsed === 400L)
- assertEnsureFreeSpaceNotCalled(ms)
+ assertEvictBlocksToFreeSpaceNotCalled(ms)
+ assert(evictedBlocks.isEmpty)
+ }
+
+ test("execution can evict storage blocks when storage memory is below
max mem (SPARK-12165)") {
+ val maxMemory = 1000L
+ val taskAttemptId = 0L
+ val (mm, ms) = makeThings(maxMemory)
+ // Acquire enough storage memory to exceed the storage region size
+ assert(mm.acquireStorageMemory(dummyBlock, 750L, evictedBlocks))
+ assertEvictBlocksToFreeSpaceNotCalled(ms)
+ assert(mm.executionMemoryUsed === 0L)
+ assert(mm.storageMemoryUsed === 750L)
+ // Should now be able to require up to 500 bytes of memory
+ assert(mm.acquireExecutionMemory(500L, taskAttemptId,
MemoryMode.ON_HEAP) === 500L)
--- End diff --
Should we briefly explain SPARK-12165 since it's not at all obvious? Also I
would change the numbers a little so it's easier to explain. Maybe something
like:
```
...
...
assert(mm.storageMemoryUsed === 700L)
// SPARK-12165: previously, MemoryStore would not evict anything because it
would
// mistakenly think that the 300 bytes of free space is still available
even after
// using it to expand the execution pool. Consequently, no blocks were
dropped
// and the following call used to grant only 300 bytes to execution.
assert(mm.acquireExecutionMemory(500L, taskAttemptId, MemoryMode.ON_HEAP)
=== 500L)
assertEvictBlocksToFreeSpaceCalled(ms, 200L)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
