Github user suyanNone commented on the pull request:
https://github.com/apache/spark/pull/3629#issuecomment-69527640
@andrewor14
`shouldn't we release the pending memory after we actually put the block
(i.e. after this line), not before?`
Agree. I think [this
line](https://github.com/apache/spark/blob/4e1f12d997426560226648d62ee17c90352613e7/core/src/main/scala/org/apache/spark/storage/MemoryStore.scala#L355)
is better. Make sure we always release the pending unroll memory never mind
ensureFreeSpace is true or false.
CacheManager.putInBlockManager,
do sth like:
1. after unroll, may the block can put in memoryStore, it will return a
unrolled data(array), and then call `blockManager.putArray`, it will finally to
call memoryStore.tryToPut, because we call unrollSafelly in CacheManager only
if the block level have memory_level.
2. in contrast, it can't put in memoryStore, then return an iterator, and
unrollMemory will be preserved until the task is completed.
```
}
} finally {
// Release memory used by this thread for shuffles
env.shuffleMemoryManager.releaseMemoryForThisThread()
// Release memory used by this thread for unrolling blocks
env.blockManager.memoryStore.releaseUnrollMemoryForThisThread()
// Release memory used by this thread for accumulators
Accumulators.clear()
runningTasks.remove(taskId)
}
```
In my code:
1. I only pending the unroll memory if we unroll a block successful. and
in current spark code, if a block unroll successful, it always will call
memoryStore.tryToPut.
right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
