[
https://issues.apache.org/jira/browse/FLINK-1843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14485084#comment-14485084
]
Stephan Ewen commented on FLINK-1843:
-------------------------------------
Keeping the Execution graph on the JobManager is only a temporary solution
anyways, until we have a proper integration with history servers.
Thus, let us do a simple and pragmatic fix.
I like the idea to keep n most recent graphs behind hard references and the
others behind soft references.
That sill gives a non-deterministic order of clearing of the older graphs,
though...
> Job History gets cleared too fast
> ---------------------------------
>
> Key: FLINK-1843
> URL: https://issues.apache.org/jira/browse/FLINK-1843
> Project: Flink
> Issue Type: Bug
> Components: JobManager
> Affects Versions: 0.9
> Reporter: Maximilian Michels
> Labels: starter
> Fix For: 0.9
>
>
> As per FLINK-1442, the JobManager stores the archived ExecutionGraph behind a
> SoftReference. At least for local setups, this mechanism doesn't seem to work
> properly. There are two issues:
> - The history gets cleared too fast
> - The history gets cleared in a non-sequential fashion, i.e. arbitrary old
> ExecutionGraph are discarded
> To solve these problems we might
> - Store the least recent ExecutionGraph behind a SoftReference
> - Store the most recent ExecutionGraphs without a SoftReference
> That way, we can save memory but have the latest history available to the
> user. We might introduce a configuration variable where the user can specify
> the number of ExecutionGraphs that should be held in memory. The remaining
> can be stored behind a SoftReference.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
