[ https://issues.apache.org/jira/browse/SPARK-8667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14604000#comment-14604000 ]
Patrick Wendell commented on SPARK-8667: ---------------------------------------- Thanks Sean. I looked for a while for an older JIRA on this, but couldn't find it. This is definitely a dup of SPARK-2015. > Improve Spark UI behavior at scale > ---------------------------------- > > Key: SPARK-8667 > URL: https://issues.apache.org/jira/browse/SPARK-8667 > Project: Spark > Issue Type: Improvement > Components: Web UI > Reporter: Patrick Wendell > Assignee: Shixiong Zhu > > This is a parent ticket and we can create child tickets when solving specific > issues. The main problem I would like to solve is the fact that the Spark UI > has issues at very large scale. > The worst issue is when there is a stage page with more than a few thousand > tasks. In this case: > 1. The page itself is very slow to load and becomes unresponsive with huge > number of tasks. > 2. The Scala XML output can become so large that it crashes the driver > program due to OOM for a page with a huge number of tasks. > I am not sure if (1) is caused by javascript slowness, or maybe just the raw > amount of data sent over the wire. If it is the latter, it might be possible > to add compression to the HTTP payload to help improve load time. > It would be nice to reproduce+investigate these issues further and create > specific sub tasks to improve them. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org