Gang Wu created SPARK-17671:
-------------------------------
Summary: Spark 2.0 history server summary page is slow even set
spark.history.ui.maxApplications
Key: SPARK-17671
URL: https://issues.apache.org/jira/browse/SPARK-17671
Project: Spark
Issue Type: Bug
Components: Web UI
Affects Versions: 2.0.0
Reporter: Gang Wu
This is a subsequent task of
[SPARK-17243|https://issues.apache.org/jira/browse/SPARK-17243]. After the fix
of SPARK-17243 (limit the number of applications in the JSON string transferred
from history server backend to web UI frontend), the history server does
display the target number of history summaries.
However, when there are more than 10k application history, it still gets slower
and slower. The problem is in the following code:
{code:title=ApplicationListResource.scala|borderStyle=solid}
@Produces(Array(MediaType.APPLICATION_JSON))
private[v1] class ApplicationListResource(uiRoot: UIRoot) {
@GET
def appList(
@QueryParam("status") status: JList[ApplicationStatus],
@DefaultValue("2010-01-01") @QueryParam("minDate") minDate:
SimpleDateParam,
@DefaultValue("3000-01-01") @QueryParam("maxDate") maxDate:
SimpleDateParam,
@QueryParam("limit") limit: Integer)
: Iterator[ApplicationInfo] = {
// although there is a limit operation in the end
// the following line still does a transformation for all history
// in the list
val allApps = uiRoot.getApplicationInfoList
// ...
// irrelevant code is omitted
// ...
if (limit != null) {
appList.take(limit)
} else {
appList
}
}
}
{code}
What the code **uiRoot.getApplicationInfoList** does is to transform every
application history from class ApplicationHistoryInfo to class ApplicationInfo.
So if there are 10k applications, 10k transformations will be done even we have
limited 5000 jobs here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
