[ https://issues.apache.org/jira/browse/SPARK-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tony Stevenson updated SPARK-4206: ---------------------------------- Reporter: Imran Rashid (was: Imran Rashid) > BlockManager warnings in local mode: "Block $blockId already exists on this > machine; not re-adding it > ----------------------------------------------------------------------------------------------------- > > Key: SPARK-4206 > URL: https://issues.apache.org/jira/browse/SPARK-4206 > Project: Spark > Issue Type: Bug > Environment: local mode, branch-1.1 & master > Reporter: Imran Rashid > Priority: Minor > > When running in local mode, you often get log warning messages like: > WARN storage.BlockManager: Block input-0-1415022975000 already exists on this > machine; not re-adding it > (eg., try running the TwitterPopularTags example in local mode) > I think these warning messages are pretty unsettling for a new user, and > should be removed. If they are truly innocuous, they should be changed to > logInfo, or maybe even logDebug. Or if they might actually indicate a > problem, we should find the root cause and fix it. > I *think* the problem is caused by a replication level > 1 when running in > local mode. In BlockManager.doPut, first the block is put locally: > https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L692 > and then if the replication level > 1, a request is sent out to replicate the > block: > https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L827 > However, in local mode, there isn't anywhere else to replicate the block; the > request comes back to the same node, which then issues the warning that the > block has already been added. > If that analysis is right, the easy fix would be to make sure > replicationLevel = 1 in local mode. But, its a little disturbing that a > replication request could result in an attempt to replicate on the same node > -- and that if something is wrong, we only issue a warning and keep going. > If this really the culprit, then it might be worth taking a closer look at > the logic of replication. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org