[
https://issues.apache.org/jira/browse/SPARK-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tony Stevenson updated SPARK-4206:
----------------------------------
Reporter: Imran Rashid (was: Imran Rashid)
> BlockManager warnings in local mode: "Block $blockId already exists on this
> machine; not re-adding it
> -----------------------------------------------------------------------------------------------------
>
> Key: SPARK-4206
> URL: https://issues.apache.org/jira/browse/SPARK-4206
> Project: Spark
> Issue Type: Bug
> Environment: local mode, branch-1.1 & master
> Reporter: Imran Rashid
> Priority: Minor
>
> When running in local mode, you often get log warning messages like:
> WARN storage.BlockManager: Block input-0-1415022975000 already exists on this
> machine; not re-adding it
> (eg., try running the TwitterPopularTags example in local mode)
> I think these warning messages are pretty unsettling for a new user, and
> should be removed. If they are truly innocuous, they should be changed to
> logInfo, or maybe even logDebug. Or if they might actually indicate a
> problem, we should find the root cause and fix it.
> I *think* the problem is caused by a replication level > 1 when running in
> local mode. In BlockManager.doPut, first the block is put locally:
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L692
> and then if the replication level > 1, a request is sent out to replicate the
> block:
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L827
> However, in local mode, there isn't anywhere else to replicate the block; the
> request comes back to the same node, which then issues the warning that the
> block has already been added.
> If that analysis is right, the easy fix would be to make sure
> replicationLevel = 1 in local mode. But, its a little disturbing that a
> replication request could result in an attempt to replicate on the same node
> -- and that if something is wrong, we only issue a warning and keep going.
> If this really the culprit, then it might be worth taking a closer look at
> the logic of replication.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
