Stefan Richter created FLINK-10066:
--------------------------------------
Summary: Keep only archived version of previous executions
Key: FLINK-10066
URL: https://issues.apache.org/jira/browse/FLINK-10066
Project: Flink
Issue Type: Improvement
Components: JobManager
Affects Versions: 1.5.2, 1.4.3, 1.6.0
Reporter: Stefan Richter
Assignee: Stefan Richter
Currently, the execution vertex stores a limited amount of previous executions
in a bounded list. This happens primarily for archiving purposes and to
remember previous locations and allocation ids. We remember the whole execution
to eventually convert it into an archived execution.
This seems unnecessary and dangerous as we have observed that this strategy is
prone to memory leaks in the job manager. With a very high vertex count or
parallelism, remembering complete executions can become very memory intensive.
Instead I suggest to eagerly transform the executions into the archived version
before adding them to the list, i.e. only the archived version is ever still
referenced after the execution becomes obsolete. This gives better control over
which information about the execution should really be kept in memory.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
