Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4845#discussion_r25965420
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/HistoryPage.scala ---
@@ -34,18 +37,31 @@ private[spark] class HistoryPage(parent: HistoryServer)
extends WebUIPage("") {
val requestedIncomplete =
Option(request.getParameter("showIncomplete")).getOrElse("false").toBoolean
- val allApps = parent.getApplicationList().filter(_.completed !=
requestedIncomplete)
- val actualFirst = if (requestedFirst < allApps.size) requestedFirst
else 0
- val apps = allApps.slice(actualFirst, Math.min(actualFirst + pageSize,
allApps.size))
-
+ val allCompletedAppsNAttempts =
+ parent.getApplicationList().filter(_.completed !=
requestedIncomplete)
+ val (hasAttemptInfo, appToAttemptMap) =
getApplicationLevelList(allCompletedAppsNAttempts)
--- End diff --
Yes, those methods will get a little more complicated. But without changing
those, HistoryPage will have to implement pretty much the same transformation.
I think doing that in HistoryPage is actually more complicated, and much less
efficient.
If you build the better data structure in FsHistoryProvider, it's done
once. If you do it in HistoryPage, it's done for every client request. So you
need to be doubly careful about memory and cpu usage, because for large
application lists, copying the whole thing would be very expensive.
The "level of indirection" you mention is a hashtable lookup, which is very
cheap. That will get you the list of attemps for a particular application, and
processing that list (which in the most common case will have a single entry)
is rather easy.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
