Repository: spark Updated Branches: refs/heads/master 274f0efef -> b5c1ef7a8
[SPARK-22097][CORE] Request an accurate memory after we unrolled the block ## What changes were proposed in this pull request? We only need request `bbos.size - unrollMemoryUsedByThisBlock` after unrolled the block. ## How was this patch tested? Existing UT. Author: Xianyang Liu <[email protected]> Closes #19316 from ConeyLiu/putIteratorAsBytes. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b5c1ef7a Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b5c1ef7a Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b5c1ef7a Branch: refs/heads/master Commit: b5c1ef7a8e4db4067bc361d10d554ee9a538423f Parents: 274f0ef Author: Xianyang Liu <[email protected]> Authored: Thu Oct 12 20:26:51 2017 +0800 Committer: Wenchen Fan <[email protected]> Committed: Thu Oct 12 20:26:51 2017 +0800 ---------------------------------------------------------------------- .../scala/org/apache/spark/storage/memory/MemoryStore.scala | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/b5c1ef7a/core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala ---------------------------------------------------------------------- diff --git a/core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala b/core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala index 651e9c7..17f7a69 100644 --- a/core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala +++ b/core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala @@ -388,7 +388,13 @@ private[spark] class MemoryStore( // perform one final call to attempt to allocate additional memory if necessary. if (keepUnrolling) { serializationStream.close() - reserveAdditionalMemoryIfNecessary() + if (bbos.size > unrollMemoryUsedByThisBlock) { + val amountToRequest = bbos.size - unrollMemoryUsedByThisBlock + keepUnrolling = reserveUnrollMemoryForThisTask(blockId, amountToRequest, memoryMode) + if (keepUnrolling) { + unrollMemoryUsedByThisBlock += amountToRequest + } + } } if (keepUnrolling) { --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
