Re: YARN terminating TaskNode

Hi Timur, Indeed, if you use JNI libraries then the memory will be off-heap and the -XmX limit will not be respected. Currently, we don't expect users to use JNI memory allocation. We might want to enforce a more strict direct memory limit in the future. In this case, you would get an

Re: YARN terminating TaskNode

Great answer, thanks you Max for a very detailed explanation! Illuminating how off-heap parameter affects the memory allocation. I read this post: https://blogs.oracle.com/jrockit/entry/why_is_my_jvm_process_larger_t and the thing that jumped on me is the allocation of memory for jni libs. I do

Re: YARN terminating TaskNode

Hello Maximilian, I'm using 1.0.0 compiled with Scala 2.11 and Hadoop 2.7. I'm running this on EMR. I didn't see any exceptions in other logs. What are the logs you are interested in? Thanks, Timur On Mon, Apr 25, 2016 at 3:44 AM, Maximilian Michels wrote: > Hi Timur, > >

Re: YARN terminating TaskNode

Hi Timur, Which version of Flink are you using? Could you share the entire logs? Thanks, Max On Mon, Apr 25, 2016 at 12:05 PM, Robert Metzger wrote: > Hi Timur, > > The reason why we only allocate 570mb for the heap is because you are > allocating most of the memory as off

Re: YARN terminating TaskNode

Hi Timur, The reason why we only allocate 570mb for the heap is because you are allocating most of the memory as off heap (direct byte buffers). In theory, the memory footprint of the JVM is limited to 570 (heap) + 1900 (direct mem) = 2470 MB (which is below 2500). But in practice thje JVM is

YARN terminating TaskNode

Hello, Next issue in a string of things I'm solving is that my application fails with the message 'Connection unexpectedly closed by remote task manager'. Yarn log shows the following: Container [pid=4102,containerID=container_1461341357870_0004_01_15] is running beyond physical memory