Repository: spark Updated Branches: refs/heads/branch-1.0 6b5f64aaf -> 3d4fa2dab
[SPARK-2144] ExecutorsPage reports incorrect # of RDD blocks This is reproducible whenever we drop a block because of memory pressure. This is because StorageStatusListener actually never removes anything from the block maps of its StorageStatuses. Instead, when a block is dropped, it sets the block's storage level to `StorageLevel.NONE`, when it should just remove it from the map. This PR includes this simple fix. Author: Andrew Or <[email protected]> Closes #1080 from andrewor14/ui-blocks and squashes the following commits: fcf9f1a [Andrew Or] Remove BlockStatus if it is no longer cached (cherry picked from commit 09deb3eee090eb8ec1d9a0cd90825699748e3ffc) Signed-off-by: Patrick Wendell <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/3d4fa2da Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/3d4fa2da Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/3d4fa2da Branch: refs/heads/branch-1.0 Commit: 3d4fa2dab0ade3b7948497a3336fd3e238f93507 Parents: 6b5f64a Author: Andrew Or <[email protected]> Authored: Tue Jun 17 01:28:22 2014 -0700 Committer: Patrick Wendell <[email protected]> Committed: Tue Jun 17 01:28:37 2014 -0700 ---------------------------------------------------------------------- .../scala/org/apache/spark/storage/StorageStatusListener.scala | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/3d4fa2da/core/src/main/scala/org/apache/spark/storage/StorageStatusListener.scala ---------------------------------------------------------------------- diff --git a/core/src/main/scala/org/apache/spark/storage/StorageStatusListener.scala b/core/src/main/scala/org/apache/spark/storage/StorageStatusListener.scala index a6e6627..c694fc8 100644 --- a/core/src/main/scala/org/apache/spark/storage/StorageStatusListener.scala +++ b/core/src/main/scala/org/apache/spark/storage/StorageStatusListener.scala @@ -37,7 +37,11 @@ class StorageStatusListener extends SparkListener { val filteredStatus = storageStatusList.find(_.blockManagerId.executorId == execId) filteredStatus.foreach { storageStatus => updatedBlocks.foreach { case (blockId, updatedStatus) => - storageStatus.blocks(blockId) = updatedStatus + if (updatedStatus.storageLevel == StorageLevel.NONE) { + storageStatus.blocks.remove(blockId) + } else { + storageStatus.blocks(blockId) = updatedStatus + } } } }
