[
https://issues.apache.org/jira/browse/SPARK-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-11156.
----------------------------------
Resolution: Incomplete
> Web UI doesn't count or show info about replicated blocks
> ---------------------------------------------------------
>
> Key: SPARK-11156
> URL: https://issues.apache.org/jira/browse/SPARK-11156
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 1.5.1
> Reporter: Ryan Williams
> Priority: Major
> Labels: bulk-closed
>
> When executors receive a replica of a block, they [notify the driver with a
> {{UpdateBlockInfo}}
> message|https://github.com/apache/spark/blob/4ee2cea2a43f7d04ab8511d9c029f80c5dadd48e/core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala#L59-L61]
> which [sends a {{SparkListenerBlockUpdated}} event to
> SparkListeners|https://github.com/apache/spark/blob/4ee2cea2a43f7d04ab8511d9c029f80c5dadd48e/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala#L67].
> However, the web UI (via its BlockStatusListener) [ignores
> {{SparkListenerBlockUpdated}} events for non-streaming
> blocks|https://github.com/apache/spark/blob/4ee2cea2a43f7d04ab8511d9c029f80c5dadd48e/core/src/main/scala/org/apache/spark/storage/BlockStatusListener.scala#L57-L60].
> As a result, in non-streaming apps:
> * The "Executors" column on RDD Page doesn't show executors housing replicas;
> it can only show the executor that initially computed (and initiated
> replication of) the block.
> * The executor-memory-usage and related stats displayed throughout the web
> interface are undercounting due to ignorance of the existence of block
> replicas.
> For example, here is the Storage tab for a simple app with 3 identical RDDs
> cached with replication equal to 1, 2, and 3:
> !http://f.cl.ly/items/3m3B2v2k2J23350I3t1c/Screen%20Shot%202015-10-16%20at%2012.30.54%20AM.png!
> These were generated with:
> {code}
> val bar1 = sc.parallelize(1 to 100000000, 100).map(_ % 100 ->
> 1).reduceByKey(_+_, 100).setName("bar1").persist(StorageLevel(false, true,
> false, true, 1))
> bar1.count
> val bar2 = sc.parallelize(1 to 100000000, 100).map(_ % 100 ->
> 1).reduceByKey(_+_, 100).setName("bar2").persist(StorageLevel(false, true,
> false, true, 2))
> bar2.count
> val bar3 = sc.parallelize(1 to 100000000, 100).map(_ % 100 ->
> 1).reduceByKey(_+_, 100).setName("bar3").persist(StorageLevel(false, true,
> false, true, 3))
> bar3.count
> {code}
> Note the identically-reported memory usage across the three.
> Here is the RDD page for the 3x-replicated RDD above:
> !http://f.cl.ly/items/0t0H1o2S2g140s1A0X0k/Screen%20Shot%202015-10-16%20at%2012.31.24%20AM.png!
> Note that only one executor is listed for each partition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
