Thanks Kien. I will at least play with the setting :) We use hadoop (s3) as a chekpoint store. In our case off heap memory is around 300MB as reported on task manager statistic page.
15 lut 2018 17:24 "Kien Truong" <duckientru...@gmail.com> napisał(a): > Hi, > > The relevant settings is: > > containerized.heap-cutoff-ratio: (Default 0.25) Percentage of heap space > to remove from containers started by YARN. When a user requests a certain > amount of memory for each TaskManager container (for example 4 GB), we can > not pass this amount as the maximum heap space for the JVM (-Xmx > argument) because the JVM is also allocating memory outside the heap. YARN > is very strict with killing containers which are using more memory than > requested. Therefore, we remove this fraction of the memory from the > requested heap as a safety margin and add it to the memory used off-heap. > > > You can reduce this in order to get a bigger JVM heap size or increase it > in order to reserve more memory for off-heap usage (for jobs with large > rocksdb state), > > but I suggest you not changing this setting without careful consideration. > > > Regards, > > Kien > > > On 2/16/2018 12:01 AM, Pawel Bartoszek wrote: > > I tried also setting taskmanager.memory.off-heap to true > > I still get around 42GB (Heap + DirectMemory) > > yarn 56827 837 16.6 16495964 10953748 ? Sl 16:53 34:10 > /usr/lib/jvm/java-openjdk/bin/java -Xms12409m -Xmx12409m > -XX:MaxDirectMemorySize=29591m > > Cheers, > Pawel > > > On 15 February 2018 at 16:03, Pawel Bartoszek <pawelbartosze...@gmail.com> > wrote: > >> Hi, >> >> I have a question regarding configuration of task manager heap size when >> running YARN session on EMR. >> >> I am running 2 task managers on m4.4xlarge (64GB RAM). I would like to >> use as much as possible of that memory for the task manager heap. >> >> However when requesting 56000 MB when staring YARN actually only around >> 42GB is assigned to TM. Do you know how I can increase that? >> >> >> This is how I start YARN session: >> /usr/lib/flink/bin/yarn-session.sh --container 2 --taskManagerMemory >> 56000 --slots 16 --detached -Dparallelism.default=32 >> -Dtaskmanager.network.numberOfBuffers=20480 ... >> >> >> This is the output of *ps aux *on TM box >> >> yarn 42843 1030 67.7 46394740 44688084 ? Sl 15:27 175:56 >> /usr/lib/jvm/java-openjdk/bin/java -Xms42000m -Xmx42000m .... >> >> yarn 42837 0.0 0.0 113104 2684 ? Ss 15:27 0:00 >> /bin/bash -c /usr/lib/jvm/java-openjdk/bin/java -Xms42000m -Xmx42000m >> .... >> >> >> I would expect around 56GB set as max heap size for TM. >> >> some settings from yarn-site.xml that might be of interest: >> >> <property> >> <name>yarn.scheduler.maximum-allocation-mb</name> >> <value>57344</value> >> </property> >> >> <property> >> <name>yarn.nodemanager.resource.memory-mb</name> >> <value>57344</value> >> </property> >> >> <property> >> <name>yarn.nodemanager.vmem-pmem-ratio</name> >> <value>5</value> >> </property> >> >> >> Cheers, >> Pawel >> >> >> >
