[
https://issues.apache.org/jira/browse/SPARK-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14205081#comment-14205081
]
Imran Rashid commented on SPARK-4206:
-------------------------------------
actually, it looks like this was kind of fixed by SPARK-3495. The messages are
now "WARN storage.BlockManager: Block input-0-1415641151000 replicated to only
0 peer(s) instead of 1 peers", which is a little better.
I still think it would be better to more clearly indicate when an impossible
amount of replication was requested, and also to replace some of the scary
warnings with exceptions if we really don't expect them to occur.
> BlockManager warnings in local mode: "Block $blockId already exists on this
> machine; not re-adding it
> -----------------------------------------------------------------------------------------------------
>
> Key: SPARK-4206
> URL: https://issues.apache.org/jira/browse/SPARK-4206
> Project: Spark
> Issue Type: Bug
> Environment: local mode, branch-1.1 & master
> Reporter: Imran Rashid
> Priority: Minor
>
> When running in local mode, you often get log warning messages like:
> WARN storage.BlockManager: Block input-0-1415022975000 already exists on this
> machine; not re-adding it
> (eg., try running the TwitterPopularTags example in local mode)
> I think these warning messages are pretty unsettling for a new user, and
> should be removed. If they are truly innocuous, they should be changed to
> logInfo, or maybe even logDebug. Or if they might actually indicate a
> problem, we should find the root cause and fix it.
> I *think* the problem is caused by a replication level > 1 when running in
> local mode. In BlockManager.doPut, first the block is put locally:
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L692
> and then if the replication level > 1, a request is sent out to replicate the
> block:
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L827
> However, in local mode, there isn't anywhere else to replicate the block; the
> request comes back to the same node, which then issues the warning that the
> block has already been added.
> If that analysis is right, the easy fix would be to make sure
> replicationLevel = 1 in local mode. But, its a little disturbing that a
> replication request could result in an attempt to replicate on the same node
> -- and that if something is wrong, we only issue a warning and keep going.
> If this really the culprit, then it might be worth taking a closer look at
> the logic of replication.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
