Nico Kruber commented on FLINK-7400:

Please note that this affects (at least) all (batch) programs with the 
following options set (which do not make much sense for streaming):
taskmanager.memory.size=<any value>

If, instead, {{taskmanager.memory.fraction}} is used, programs may be safe due 
to FLINK-7401 but the actual additional buffer that we get from that may be too 
small, especially if RocksDB or other libraries using off-heap memory are used.

> off-heap limits set to conservatively in yarn
> ---------------------------------------------
>                 Key: FLINK-7400
>                 URL: https://issues.apache.org/jira/browse/FLINK-7400
>             Project: Flink
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 1.3.0, 1.3.1, 1.3.2
>            Reporter: Nico Kruber
>            Assignee: Nico Kruber
> Inside {{ContaineredTaskManagerParameters}}, since FLINK-6217, the 
> {{offHeapSize}} is set to the amount of memory Flink will use off-heap which 
> will be set as the value for {{-XX:MaxDirectMemorySize}} in various cases. 
> This does not account for any off-heap use by other components than Flink, 
> e.g. RocksDB, other libraries, or the JVM itself.
> We should add the {{cutoff}} from the {{CONTAINERIZED_HEAP_CUTOFF_RATIO}} 
> configuration parameter to {{offHeapSize}} as implied by the description on 
> what this parameter is there for.

This message was sent by Atlassian JIRA

Reply via email to