[ https://issues.apache.org/jira/browse/SPARK-17672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Or resolved SPARK-17672. ------------------------------- Resolution: Fixed Fix Version/s: 2.0.1 Target Version/s: 2.0.1 > Spark 2.0 history server web Ui takes too long for a single application > ----------------------------------------------------------------------- > > Key: SPARK-17672 > URL: https://issues.apache.org/jira/browse/SPARK-17672 > Project: Spark > Issue Type: Bug > Components: Web UI > Affects Versions: 2.0.0 > Reporter: Gang Wu > Fix For: 2.0.1 > > > When there are 10K application history in the history server back end, it can > take a very long time to even get a single application history page. After > some investigation, I found the root cause was the following piece of code: > {code:title=OneApplicationResource.scala|borderStyle=solid} > @Produces(Array(MediaType.APPLICATION_JSON)) > private[v1] class OneApplicationResource(uiRoot: UIRoot) { > @GET > def getApp(@PathParam("appId") appId: String): ApplicationInfo = { > val apps = uiRoot.getApplicationInfoList.find { _.id == appId } > apps.getOrElse(throw new NotFoundException("unknown app: " + appId)) > } > } > {code} > Although all application history infos are stored in a LinkedHashMap, here to > code transforms the map to an iterator and then uses the find() api which is > O( n) instead of O(1) from a map.get() operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org