[
https://issues.apache.org/jira/browse/FLINK-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16963844#comment-16963844
]
Zhu Zhu edited comment on FLINK-14566 at 10/31/19 10:36 AM:
------------------------------------------------------------
[~ykt836] When saying "little", I mean even "no" in the worst case. The
operator can throw error in this case.
But it might be acceptable, given that the resources is set to be UNKNOWN. Just
like OOM is acceptable if the heap resource is not specified(UNKNOWN) and the
TM heap memory is not properly configured.
To solve it, users need to specify the required resource.
Only counting operators really using managed memory(FLINK-14566 + FLINK-14062)
is an optimization to reduce managed memory waste, but it still cannot
guarantee an operator will always acquire enough managed memory to work. So
technically it's not necessary.
was (Author: zhuzh):
[~ykt836] When saying "little", I mean even "no" in the worst case. The
operator can throw error in this case.
But it might be acceptable, given that the resources is set to be UNKNOWN. Just
like OOM is acceptable if the heap resource is not specified(UNKNOWN) and the
TM heap memory is not properly configured.
To solve it, users need to specify the required resource.
Only counting operators really using managed memory is an optimization to
reduce managed memory waste, but it still cannot guarantee an operator will
always acquire enough managed memory to work. So technically it's not necessary.
> Enable ResourceSpec to get/set whether managed memory is used
> -------------------------------------------------------------
>
> Key: FLINK-14566
> URL: https://issues.apache.org/jira/browse/FLINK-14566
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
> Affects Versions: 1.10.0
> Reporter: Zhu Zhu
> Priority: Major
> Fix For: 1.10.0
>
>
> To calculate managed memory fraction for an operator with UNKNOWN resources,
> we need to know whether the operator will use managed memory, according to
> FLINK-14062.
> I'd propose to define special values of
> onHeapManagedMemory/offHeapManagedMemory
> in ResourceSpec(introduced in FLINK-14405) to distinguish whether managed
> memory is used.
> - For UNKNOWN resource spec, onHeapManagedMemory==MemorySize.ZERO means on
> heap managed memory is used, while onHeapManagedMemory==null means not. For
> non-UNKNOWN resource spec, managed memory size larger than zero indicates it
> is used.
> - similar for off heap managed memory.
> getters: isUsingManagedMemoryOnHeap()/isUsingManagedMemoryOffHeap()
> setters:
> setUsingManagedMemoryOnHeap(boolean)/setUsingManagedMemoryOffHeap(boolean)
> (setters are only allowed to invoke on UNKNOWN, should be ignored for
> non-UNKNOWN for compatibility)
> {{ResourceSpec#isKnown()}} should be added to replace existing UNKNOWN usages
> for equality checking.
> cc [~ykt836] [~trohrmann] [~azagrebin] [~xintongsong]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
