Hi,
The relevant settings is:
|containerized.heap-cutoff-ratio|: (Default 0.25) Percentage of heap
space to remove from containers started by YARN. When a user requests a
certain amount of memory for each TaskManager container (for example 4
GB), we can not pass this amount as the maximum heap space for the JVM
(|-Xmx| argument) because the JVM is also allocating memory outside the
heap. YARN is very strict with killing containers which are using more
memory than requested. Therefore, we remove this fraction of the memory
from the requested heap as a safety margin and add it to the memory used
off-heap.
You can reduce this in order to get a bigger JVM heap size or increase
it in order to reserve more memory for off-heap usage (for jobs with
large rocksdb state),
but I suggest you not changing this setting without careful consideration.
Regards,
Kien
On 2/16/2018 12:01 AM, Pawel Bartoszek wrote:
I tried also setting |taskmanager.memory.off-heap|| to true|
I still get around 42GB (Heap + DirectMemory)
yarn 56827 837 16.6 16495964 10953748 ? Sl 16:53 34:10
/usr/lib/jvm/java-openjdk/bin/java -Xms12409m -Xmx12409m
-XX:MaxDirectMemorySize=29591m
Cheers,
Pawel
On 15 February 2018 at 16:03, Pawel Bartoszek
<pawelbartosze...@gmail.com <mailto:pawelbartosze...@gmail.com>> wrote:
Hi,
I have a question regarding configuration of task manager heap
size when running YARN session on EMR.
I am running 2 task managers on m4.4xlarge (64GB RAM). I would
like to use as much as possible of that memory for the task
manager heap.
However when requesting 56000 MB when staring YARN actually only
around 42GB is assigned to TM. Do you know how I can increase that?
This is how I start YARN session:
/usr/lib/flink/bin/yarn-session.sh --container 2
--taskManagerMemory 56000 --slots 16 --detached
-Dparallelism.default=32
-Dtaskmanager.network.numberOfBuffers=20480 ...
This is the output of /ps aux /on TM box
yarn 42843 1030 67.7 46394740 44688084 ? Sl 15:27 175:56
/usr/lib/jvm/java-openjdk/bin/java -Xms42000m -Xmx42000m ....
yarn 42837 0.0 0.0 113104 2684 ? Ss 15:27 0:00
/bin/bash -c /usr/lib/jvm/java-openjdk/bin/java -Xms42000m
-Xmx42000m ....
I would expect around 56GB set as max heap size for TM.
some settings from yarn-site.xml that might be of interest:
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>57344</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>57344</value>
</property>
<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>5</value>
</property>
Cheers,
Pawel
