[
https://issues.apache.org/jira/browse/FLINK-15037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988444#comment-16988444
]
Zhu Zhu edited comment on FLINK-15037 at 12/5/19 4:09 AM:
----------------------------------------------------------
Thanks for the feedback [~ykt836]. This task is not for 1.10. We will review
the design later regarding a few other requirements:
1. shared managed memory for state backends. With FLINK-7289, RocksDb instances
in the same slot will share the same LRUCache. LRUCache would be the only
managed memory consumer (in streaming job cases) and it will take all the
managed memory from a slot. So that it requires slot wide
2. floating managed memory for operators as you mentioned
Ideally we should enable {{LimittingMemoryManager}} to support both exclusive
memory and shared memory.
was (Author: zhuzh):
Thanks for the feedback [~ykt836]. This task is not for 1.10. We will review
the design later regarding a few other requirements:
1. shared managed memory for state backends. With FLINK-7289, RocksDb instances
in the same slot will share the same LRUCache. LRUCache would be the only
managed memory consumer (in streaming job cases) and it will take all the
managed memory from a slot. So that it requires slot wide
2. floating managed memory for operators as you mentioned
> Introduce LimittingMemoryManager as operator scope MemoyManager
> ---------------------------------------------------------------
>
> Key: FLINK-15037
> URL: https://issues.apache.org/jira/browse/FLINK-15037
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
> Affects Versions: 1.10.0
> Reporter: Zhu Zhu
> Priority: Major
>
> Current {{MemoryManager}} is slot scope component, and the operators needs to
> use a fraction to compute memory size/pages it can allocate and use it to
> reserver memory or allocate pages.
> This, however, requires operators to be aware of the managed memory fraction.
> There is also risk that one operator may over allocate resources than it has
> declared and causes other operators to break.
> To separate concerns, we can introduce a {{LimittingMemoryManager}} which
> wraps the the original MemoryManager but limit the available memory size to
> the fraction of total memory governed by the original one. This wrapper would
> be a operator scope {{MemoryManager}}.
> Some more detailed discussions can be found in FLINK-14936.
> cc [~azagrebin]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
