[
https://issues.apache.org/jira/browse/SPARK-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14579273#comment-14579273
]
Steve Loughran edited comment on SPARK-8275 at 6/10/15 11:43 AM:
-----------------------------------------------------------------
The cache code is from google; history server provides a method to get the data
for an entry, but there's no logic in the cache itself to have a refresh time
on entries.
One solution would be
# cache entries to include a timestamp and completed flag alongside the SparkUI
instances
# direct all cache.get operations through a single method in HistoryServer
# have that method do something like
{code}
def getUI(id: String): SparkUI = {
var cacheEntry = cache.get(id)
if (!cacheEntry.completed && (cacheEntry.timestamp + expiryTime) < now()) {
cache.release(id)
cache.get(id)
}
cacheEntry
}
{code}
this will leave out of date entries in the cache, but on any retrieval trigger
the rebuild.
was (Author: [email protected]):
The cache code is from google; history server provides a method to get the data
for an entry, but there's no logic in the cache itself to have a refresh time
on entries.
One solution would be
# cache entries to include a timestamp and completed flag alongside the SparkUI
instances
# direct all cache.get operations through a single method in HistoryServer
# have that method do something like
{code}
def getUI(id: String): SparkUI = {
var cacheEntry = cache.get(id)
if (!cacheEntry.completed && (cacheEntry.timestamp + expiryTime) < now()) {
cache.release(id)
cache.get(id)
}
cacheEntry
}
this will leave out of date entries in the cache, but on any retrieval trigger
the rebuild.
> HistoryServer caches incomplete App UIs
> ---------------------------------------
>
> Key: SPARK-8275
> URL: https://issues.apache.org/jira/browse/SPARK-8275
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 1.3.1
> Reporter: Steve Loughran
>
> The history server caches applications retrieved from the
> {{ApplicationHistoryProvider.getAppUI()}} call for performance: it's
> expensive to rebuild.
> However, this cache also includes incomplete applications, as well as
> completed ones —and it never attempts to refresh the incomplete application.
> As a result, if you do a GET of the history of a running application, even
> after the application is finished, you'll still get the web UI/history as it
> was when that first GET was issued.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
