Repository: spark
Updated Branches:
refs/heads/branch-1.1 fe7d7a983 -> 9b9923744
[SPARK-4714] BlockManager.dropFromMemory() should check whether block has been
removed after synchronizing on BlockInfo instance.
After synchronizing on the `info` lock in the
`removeBlock`/`dropOldBlocks`/`dropFromMemory` methods in BlockManager, the
block that `info` represented may have already removed.
The three methods have the same logic to get the `info` lock:
```
info = blockInfo.get(id)
if (info != null) {
info.synchronized {
// do something
}
}
```
So, there is chance that when a thread enters the `info.synchronized` block,
`info` has already been removed from the `blockInfo` map by some other thread
who entered `info.synchronized` first.
The `removeBlock` and `dropOldBlocks` methods are idempotent, so it's safe for
them to run on blocks that have already been removed.
But in `dropFromMemory` it may be problematic since it may drop block data
which already removed into the diskstore, and this calls data store operations
that are not designed to handle missing blocks.
This patch fixes this issue by adding a check to `dropFromMemory` to test
whether blocks have been removed by a racing thread.
Author: hushan[è¡ç] <[email protected]>
Closes #3574 from suyanNone/refine-block-concurrency and squashes the following
commits:
edb989d [hushan[è¡ç]] Refine code style and comments position
55fa4ba [hushan[è¡ç]] refine code
e57e270 [hushan[è¡ç]] add check info is already remove or not while having
gotten info.syn
(cherry picked from commit 30dca924df0efbdc1b638fa7c705fe8743570783)
Signed-off-by: Josh Rosen <[email protected]>
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/9b992374
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/9b992374
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/9b992374
Branch: refs/heads/branch-1.1
Commit: 9b99237441c355179dbea2662c5baef0cef82e5d
Parents: fe7d7a9
Author: hushan[è¡ç] <[email protected]>
Authored: Tue Dec 9 15:11:20 2014 -0800
Committer: Josh Rosen <[email protected]>
Committed: Tue Dec 9 15:55:10 2014 -0800
----------------------------------------------------------------------
core/src/main/scala/org/apache/spark/storage/BlockManager.scala | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/9b992374/core/src/main/scala/org/apache/spark/storage/BlockManager.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/storage/BlockManager.scala
b/core/src/main/scala/org/apache/spark/storage/BlockManager.scala
index 3113d4a..881b956 100644
--- a/core/src/main/scala/org/apache/spark/storage/BlockManager.scala
+++ b/core/src/main/scala/org/apache/spark/storage/BlockManager.scala
@@ -975,8 +975,10 @@ private[spark] class BlockManager(
// If we get here, the block write failed.
logWarning(s"Block $blockId was marked as failure. Nothing to drop")
return None
+ } else if (blockInfo.get(blockId).isEmpty) {
+ logWarning(s"Block $blockId was already dropped.")
+ return None
}
-
var blockIsUpdated = false
val level = info.level
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
