[ https://issues.apache.org/jira/browse/SPARK-17671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15524595#comment-15524595 ]
Gang Wu commented on SPARK-17671: --------------------------------- Hi [~ajbozarth], can you take a look at the PR? Thanks! > Spark 2.0 history server summary page is slow even set > spark.history.ui.maxApplications > --------------------------------------------------------------------------------------- > > Key: SPARK-17671 > URL: https://issues.apache.org/jira/browse/SPARK-17671 > Project: Spark > Issue Type: Bug > Components: Web UI > Affects Versions: 2.0.0 > Reporter: Gang Wu > > This is a subsequent task of > [SPARK-17243|https://issues.apache.org/jira/browse/SPARK-17243]. After the > fix of SPARK-17243 (limit the number of applications in the JSON string > transferred from history server backend to web UI frontend), the history > server does display the target number of history summaries. > However, when there are more than 10k application history, it still gets > slower and slower. The problem is in the following code: > {code:title=ApplicationListResource.scala|borderStyle=solid} > @Produces(Array(MediaType.APPLICATION_JSON)) > private[v1] class ApplicationListResource(uiRoot: UIRoot) { > @GET > def appList( > @QueryParam("status") status: JList[ApplicationStatus], > @DefaultValue("2010-01-01") @QueryParam("minDate") minDate: > SimpleDateParam, > @DefaultValue("3000-01-01") @QueryParam("maxDate") maxDate: > SimpleDateParam, > @QueryParam("limit") limit: Integer) > : Iterator[ApplicationInfo] = { > // although there is a limit operation in the end > // the following line still does a transformation for all history > // in the list > val allApps = uiRoot.getApplicationInfoList > > // ... > // irrelevant code is omitted > // ... > if (limit != null) { > appList.take(limit) > } else { > appList > } > } > } > {code} > What the code **uiRoot.getApplicationInfoList** does is to transform every > application history from class ApplicationHistoryInfo to class > ApplicationInfo. So if there are 10k applications, 10k transformations will > be done even we have limited 5000 jobs here. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org