Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4845#discussion_r25652737
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/HistoryPage.scala ---
@@ -34,18 +37,31 @@ private[spark] class HistoryPage(parent: HistoryServer)
extends WebUIPage("") {
val requestedIncomplete =
Option(request.getParameter("showIncomplete")).getOrElse("false").toBoolean
- val allApps = parent.getApplicationList().filter(_.completed !=
requestedIncomplete)
- val actualFirst = if (requestedFirst < allApps.size) requestedFirst
else 0
- val apps = allApps.slice(actualFirst, Math.min(actualFirst + pageSize,
allApps.size))
-
+ val allCompletedAppsNAttempts =
+ parent.getApplicationList().filter(_.completed !=
requestedIncomplete)
+ val (hasAttemptInfo, appToAttemptMap) =
getApplicationLevelList(allCompletedAppsNAttempts)
--- End diff --
I'm a little confused about this code, so let me suggest something
different. Why not turn the listing into this:
// Map application ID to a list of attempts
val appAttempts: LinkedHashMap[String, Seq[ApplicationHistoryInfo]] =
...
That should make it easier to handle apps with a single and multiple
attemps with code that's mostly the same.
To avoid exploding memory usage, the code that does this translation (from
`Seq[ApplicationHistoryInfo]` to the above map) should also take care of only
looking at the app list for the current page being requested (instead of
building a map for all applications and then slicing it).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
