Repository: spark Updated Branches: refs/heads/branch-2.0 98bbc4410 -> 2cd327ef5
[SPARK-17056][CORE] Fix a wrong assert regarding unroll memory in MemoryStore ## What changes were proposed in this pull request? There is an assert in MemoryStore's putIteratorAsValues method which is used to check if unroll memory is not released too much. This assert looks wrong. ## How was this patch tested? Jenkins tests. Author: Liang-Chi Hsieh <sim...@tw.ibm.com> Closes #14642 from viirya/fix-unroll-memory. (cherry picked from commit e7bce9e1876de6ee975ccc89351db58119674aef) Signed-off-by: Josh Rosen <joshro...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/2cd327ef Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/2cd327ef Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/2cd327ef Branch: refs/heads/branch-2.0 Commit: 2cd327ef5e4c3f6b8468ebb2352479a1686b7888 Parents: 98bbc44 Author: Liang-Chi Hsieh <sim...@tw.ibm.com> Authored: Tue Sep 27 16:00:39 2016 -0700 Committer: Josh Rosen <joshro...@databricks.com> Committed: Tue Sep 27 16:01:04 2016 -0700 ---------------------------------------------------------------------- .../main/scala/org/apache/spark/storage/memory/MemoryStore.scala | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/2cd327ef/core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala ---------------------------------------------------------------------- diff --git a/core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala b/core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala index 161434c..9b87c42 100644 --- a/core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala +++ b/core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala @@ -271,7 +271,7 @@ private[spark] class MemoryStore( blockId, Utils.bytesToString(size), Utils.bytesToString(maxMemory - blocksMemoryUsed))) Right(size) } else { - assert(currentUnrollMemoryForThisTask >= currentUnrollMemoryForThisTask, + assert(currentUnrollMemoryForThisTask >= unrollMemoryUsedByThisBlock, "released too much unroll memory") Left(new PartiallyUnrolledIterator( this, --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
