Github user dibbhatt commented on a diff in the pull request:
https://github.com/apache/spark/pull/6990#discussion_r33349643
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -833,8 +833,10 @@ private[spark] class BlockManager(
logDebug("Put block %s locally took %s".format(blockId,
Utils.getUsedTimeMs(startTimeMs)))
// Either we're storing bytes and we asynchronously started
replication, or we're storing
- // values and need to serialize and replicate them now:
- if (putLevel.replication > 1) {
+ // values and need to serialize and replicate them now.
+ // Should not replicate the block if its StorageLevel is
StorageLevel.NONE or
+ // putting it to local is failed.
+ if (!putBlockInfo.isFailed && putLevel.replication > 1) {
--- End diff --
Agree with this. BlockInfo can be failed for MEMORY_ONLY ( or
MEMORY_ONLY_2) cases when Blocks not able to Unroll Safely (In case of
IteratorValue) or not able to store to memory for ArrayValue irrespective of
Block is serialized or not. Thus if Block is not stored , or in other words ,
if Block is either not Unrolled Safely or Try to Put the block failed because
there is not enough memory , we should not do Replicate the Block .
But I am not able to understand what @squito said "you don't even have
enough memory to create the message that requests the peer to replicate" .
If you see the test cases , if Block Unroll Failed , it can still Replicate
to Peer if BlockInfo Failed check is not done (as it was before the fix). I am
trying to prevent the failed block to replicate because store to local has
failed and block status is not reported to master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
