Gang Wu created SPARK-17672: ------------------------------- Summary: Spark 2.0 history server web Ui takes too long for a single application Key: SPARK-17672 URL: https://issues.apache.org/jira/browse/SPARK-17672 Project: Spark Issue Type: Bug Components: Web UI Affects Versions: 2.0.0 Reporter: Gang Wu
When there are 10K application history in the history server back end, it can take a very long time to even get a single application history page. After some investigation, I found the root cause was the following piece of code: {code:title=OneApplicationResource.scala|borderStyle=solid} @Produces(Array(MediaType.APPLICATION_JSON)) private[v1] class OneApplicationResource(uiRoot: UIRoot) { @GET def getApp(@PathParam("appId") appId: String): ApplicationInfo = { val apps = uiRoot.getApplicationInfoList.find { _.id == appId } apps.getOrElse(throw new NotFoundException("unknown app: " + appId)) } } {code} Although all application history infos are stored in a LinkedHashMap, here to code transforms the map to an iterator and then uses the find() api which is O(n) instead of O(1) from a map.get() operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org