Reynold Xin created SPARK-2299:
----------------------------------
Summary: Consolidate various stageIdTo* hash maps
Key: SPARK-2299
URL: https://issues.apache.org/jira/browse/SPARK-2299
Project: Spark
Issue Type: Improvement
Components: Web UI
Affects Versions: 1.0.0
Reporter: Reynold Xin
Assignee: Andrew Or
In JobProgressListener:
{code}
val stageIdToTime = HashMap[Int, Long]()
val stageIdToShuffleRead = HashMap[Int, Long]()
val stageIdToShuffleWrite = HashMap[Int, Long]()
val stageIdToMemoryBytesSpilled = HashMap[Int, Long]()
val stageIdToDiskBytesSpilled = HashMap[Int, Long]()
val stageIdToTasksActive = HashMap[Int, HashMap[Long, TaskInfo]]()
val stageIdToTasksComplete = HashMap[Int, Int]()
val stageIdToTasksFailed = HashMap[Int, Int]()
val stageIdToTaskData = HashMap[Int, HashMap[Long, TaskUIData]]()
val stageIdToExecutorSummaries = HashMap[Int, HashMap[String,
ExecutorSummary]]()
val stageIdToPool = HashMap[Int, String]()
val stageIdToDescription = HashMap[Int, String]()
{code}
We should consolidate them to reduce memory & be less error prone.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
