[
https://issues.apache.org/jira/browse/SPARK-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437688#comment-15437688
]
Gang Wu commented on SPARK-17243:
---------------------------------
Hi Alex, I think in Spark 1.5 history server obtains all application summary
metadata directly from class FsHistoryProvider. You can check in
HistoryPage.scala. While in Spark 2.0 it deals with JSON string (in
historypage.js) which is MUCH slower than before. It may make sense if the old
way is used?
> Spark 2.0 history server summary page gets stuck at "loading history summary"
> with 10K+ application history
> -----------------------------------------------------------------------------------------------------------
>
> Key: SPARK-17243
> URL: https://issues.apache.org/jira/browse/SPARK-17243
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 2.0.0
> Environment: Linux
> Reporter: Gang Wu
>
> The summary page of Spark 2.0 history server web UI keep displaying "Loading
> history summary..." all the time and crashes the browser when there are more
> than 10K application history event logs on HDFS.
> I did some investigation, "historypage.js" file sends a REST request to
> /api/v1/applications endpoint of history server REST endpoint and gets back
> json response. When there are more than 10K applications inside the event log
> directory it takes forever to parse them and render the page. When there are
> only hundreds or thousands of application history it is running fine.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
