Gang Wu created SPARK-17672:
-------------------------------
Summary: Spark 2.0 history server web Ui takes too long for a
single application
Key: SPARK-17672
URL: https://issues.apache.org/jira/browse/SPARK-17672
Project: Spark
Issue Type: Bug
Components: Web UI
Affects Versions: 2.0.0
Reporter: Gang Wu
When there are 10K application history in the history server back end, it can
take a very long time to even get a single application history page. After some
investigation, I found the root cause was the following piece of code:
{code:title=OneApplicationResource.scala|borderStyle=solid}
@Produces(Array(MediaType.APPLICATION_JSON))
private[v1] class OneApplicationResource(uiRoot: UIRoot) {
@GET
def getApp(@PathParam("appId") appId: String): ApplicationInfo = {
val apps = uiRoot.getApplicationInfoList.find { _.id == appId }
apps.getOrElse(throw new NotFoundException("unknown app: " + appId))
}
}
{code}
Although all application history infos are stored in a LinkedHashMap, here to
code transforms the map to an iterator and then uses the find() api which is
O(n) instead of O(1) from a map.get() operation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
