[
https://issues.apache.org/jira/browse/SPARK-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14074692#comment-14074692
]
Shivaram Venkataraman commented on SPARK-2316:
----------------------------------------------
On a related note, can we have flags to turn off some of the UI listeners ? If
the StorageTab is going to be too expensive to update, it'll be good to have a
way to turn it off and just have the JobProgress show up in the UI
> StorageStatusListener should avoid O(blocks) operations
> -------------------------------------------------------
>
> Key: SPARK-2316
> URL: https://issues.apache.org/jira/browse/SPARK-2316
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, Web UI
> Affects Versions: 1.0.0
> Reporter: Patrick Wendell
> Assignee: Andrew Or
> Priority: Critical
>
> In the case where jobs are frequently causing dropped blocks the storage
> status listener can bottleneck. This is slow for a few reasons, one being
> that we use Scala collection operations, the other being that we operations
> that are O(number of blocks). I think using a few indices here could make
> this much faster.
> {code}
> at java.lang.Integer.valueOf(Integer.java:642)
> at scala.runtime.BoxesRunTime.boxToInteger(BoxesRunTime.java:70)
> at
> org.apache.spark.storage.StorageUtils$$anonfun$9.apply(StorageUtils.scala:82)
> at
> scala.collection.TraversableLike$$anonfun$groupBy$1.apply(TraversableLike.scala:328)
> at
> scala.collection.TraversableLike$$anonfun$groupBy$1.apply(TraversableLike.scala:327)
> at
> scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:224)
> at
> scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403)
> at
> scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403)
> at
> scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403)
> at
> scala.collection.TraversableLike$class.groupBy(TraversableLike.scala:327)
> at scala.collection.AbstractTraversable.groupBy(Traversable.scala:105)
> at
> org.apache.spark.storage.StorageUtils$.rddInfoFromStorageStatus(StorageUtils.scala:82)
> at
> org.apache.spark.ui.storage.StorageListener.updateRDDInfo(StorageTab.scala:56)
> at
> org.apache.spark.ui.storage.StorageListener.onTaskEnd(StorageTab.scala:67)
> - locked <0x00000000a27ebe30> (a
> org.apache.spark.ui.storage.StorageListener)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
