[
https://issues.apache.org/jira/browse/FLINK-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen closed FLINK-1101.
-------------------------------
Resolution: Won't Fix
A different solution has been implemented in the meantime. Instead of making
the memory management adaptive, batch jobs scheduler smaller units (pipelined
regions) at a time and configure more precisely the memory needs of the
deployed tasks.
> Make memory management adaptive
> -------------------------------
>
> Key: FLINK-1101
> URL: https://issues.apache.org/jira/browse/FLINK-1101
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Task
> Affects Versions: 0.7.0-incubating
> Reporter: Stephan Ewen
> Assignee: Stephan Ewen
> Priority: Major
> Labels: stale-assigned
>
> I suggest to rework the memory management.
> Right now, it works the following way: When the program is submitted, it is
> checked how many memory consuming operations happen (sort, hash, explicit
> cache, ... ) Each one is assigned a static relative memory fraction, which
> the taskmanager provides.
> This is a very conservative planning and mostly due to the fact that with the
> streaming runtime, we may have all operations running concurrently. But in
> fact we mostly have not and are therefore wasting memory by being too
> conservative.
> To make the most of the available memory, I suggest to make the management
> adaptive:
> - Operators need to be able to request memory bit by bit
> - Operators need to be able to release memory on request. The sorter /
> hash table / cache do this naturally by spilling.
> - Memory has to be redistributed between operators when new requesters come.
> This also plays nicely with the idea of leaving all non-assigned memory to
> intermediate results, to allow for maximum caching of historic intermediate
> results.
> This would solve [FLINK-170] and [FLINK-84].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
