Github user suyanNone commented on the pull request:
https://github.com/apache/spark/pull/3629#issuecomment-69301441
@andrewor14 sorry for my poor english, and @liyezhang556520 has explained
well.
now, I just talk the situation if just remove the memory release in
`unrollSafely`.
In MemoryStore, the memory has 2 use, one for `unroll` and the other for
`entrys`.Every thread has a preserved unroll memory in current design, which
will be stored in `unrollMemoryMap`.
actual free memory in the memory store = maxMemory - entrys.size -
currenUnrollMemory(=unrollMemoryMap.size).
Thread A: Thread A's preserved unroll memory, let suppose it equals = 20MB
1. after Thread A unrollSafely(), 20MB will still hold in
`currentUnrollMemory `
2. atter thread A tryToPut this block, 20MB will be hold in `entrys`
3. after thread A done the task, it will release the preserved unroll
memory in `currentUnrollMemory`, 20MB.
if thread A now between step 2 and step 3. 20MB be counted both in
`entrys` and `currentMemoryMap`
Thread B begin to put a block in memory, he will have 3 places to be
affected by the Thread A unreleased unroll Memory(20MB), where use freeMemory
or currentUnrollMemory.
we can says
trueFreeMemory should = freeMemory + 20MB
trueCurrentUnrollMemory should = currentUnrollMemory - 20MB
first place:
reserveUnrollMemoryForThisThread, to judge current memoryStore whether have
enough free memory to allocate preserved memory for current thread.
```
def reserveUnrollMemoryForThisThread(memory: Long): Boolean = {
accountingLock.synchronized {
val granted = freeMemory > currentUnrollMemory + memory
if (granted) {
val threadId = Thread.currentThread().getId
unrollMemoryMap(threadId) = unrollMemoryMap.getOrElse(threadId, 0L)
+ memory
}
granted
}
}
```
second place
ensureFreeSpace: to get actual free memory to judge if need to drop old
blocks.
```
// Take into account the amount of memory currently occupied by
unrolling blocks
val actualFreeMemory = freeMemory - currentUnrollMemory
if (actualFreeMemory < space) {
```
third place
unrollSafely: To make sure if currentUnrollMemory is larger than max,
will not allocate more memory for current thread.
```
val spaceToEnsure = maxUnrollMemory - currentUnrollMemory
if (spaceToEnsure > 0) {
....
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
