[
https://issues.apache.org/jira/browse/SPARK-17671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-17671:
------------------------------------
Assignee: Apache Spark
> Spark 2.0 history server summary page is slow even set
> spark.history.ui.maxApplications
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-17671
> URL: https://issues.apache.org/jira/browse/SPARK-17671
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 2.0.0
> Reporter: Gang Wu
> Assignee: Apache Spark
>
> This is a subsequent task of
> [SPARK-17243|https://issues.apache.org/jira/browse/SPARK-17243]. After the
> fix of SPARK-17243 (limit the number of applications in the JSON string
> transferred from history server backend to web UI frontend), the history
> server does display the target number of history summaries.
> However, when there are more than 10k application history, it still gets
> slower and slower. The problem is in the following code:
> {code:title=ApplicationListResource.scala|borderStyle=solid}
> @Produces(Array(MediaType.APPLICATION_JSON))
> private[v1] class ApplicationListResource(uiRoot: UIRoot) {
> @GET
> def appList(
> @QueryParam("status") status: JList[ApplicationStatus],
> @DefaultValue("2010-01-01") @QueryParam("minDate") minDate:
> SimpleDateParam,
> @DefaultValue("3000-01-01") @QueryParam("maxDate") maxDate:
> SimpleDateParam,
> @QueryParam("limit") limit: Integer)
> : Iterator[ApplicationInfo] = {
> // although there is a limit operation in the end
> // the following line still does a transformation for all history
> // in the list
> val allApps = uiRoot.getApplicationInfoList
>
> // ...
> // irrelevant code is omitted
> // ...
> if (limit != null) {
> appList.take(limit)
> } else {
> appList
> }
> }
> }
> {code}
> What the code **uiRoot.getApplicationInfoList** does is to transform every
> application history from class ApplicationHistoryInfo to class
> ApplicationInfo. So if there are 10k applications, 10k transformations will
> be done even we have limited 5000 jobs here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
