When was the last time you updated your 0.10-SNAPSHOT Flink cluster? If it has been more than a couple of weeks, then I'd advise you to update to the latest snapshot version. There has been an issue with the calculation of the off-heap memory limit in the past.
Thanks, Max On Mon, Oct 19, 2015 at 5:26 PM, Gyula Fóra <gyula.f...@gmail.com> wrote: > It's 0.10-SNAPSHOT > > Gyula > > Maximilian Michels <m...@apache.org> ezt írta (időpont: 2015. okt. 19., H, > 17:13): >> >> I forgot to ask you: Which version of Flink are you using? 0.9.1 or >> 0.10-SNAPSHOT? >> >> On Mon, Oct 19, 2015 at 5:05 PM, Maximilian Michels <m...@apache.org> >> wrote: >> > Hi Jakob, >> > >> > Thanks. Flink allocates its network memory as direct memory outside >> > the normal Java heap. By default, that is 64MB but can grow up to >> > 128MB on heavy network transfer. How much memory does your machine >> > have? Could it be that your upper memory bound is lower than 2048 + >> > 128 MB? >> > >> > Best, >> > Max >> > >> > On Mon, Oct 19, 2015 at 4:32 PM, Jakob Ericsson >> > <jakob.erics...@gmail.com> wrote: >> >> Hi, >> >> >> >> See answers below. >> >> >> >> /Jakob >> >> >> >> On Mon, Oct 19, 2015 at 4:03 PM, Maximilian Michels <m...@apache.org> >> >> wrote: >> >>> >> >>> Hi Jakob, >> >>> >> >>> Thank you for reporting the bug. Could you please post your >> >>> configuration here? In particular, could you please tell us the value >> >>> of the following configuration variables: >> >>> >> >>> taskmanager.heap.mb >> >> >> >> taskmanager.heap.mb: 2048 >> >>> >> >>> taskmanager.network.numberOfBuffers >> >> >> >> >> >> Default value. Not changed. >> >> >> >>> >> >>> taskmanager.memory.off-heap >> >>> >> >> Default value Not changed. >> >> >> >>> >> >>> Are you running the Flink cluster in batch or streaming mode? >> >>> >> >> Started in streaming mode. Running with two nodes. In the cluster. >> >> Also, I have set the "env.java.opts: -XX:+UseConcMarkSweepGC" due to >> >> some >> >> strange java core dumps in the G1 GC. >> >> >> >>> >> >>> Direct memory is used by Flink's network layer. My guess is that you >> >>> have set taskmanager.heap.mb too low (it constraints the number of >> >>> direct memory at the moment). >> >>> >> >>> Thank you, >> >>> Max >> >>> >> >>> >> >>> On Mon, Oct 19, 2015 at 3:24 PM, Jakob Ericsson >> >>> <jakob.erics...@gmail.com> wrote: >> >>> > Hello, >> >>> > >> >>> > We are running into a strange problem with Direct Memory buffers. >> >>> > From >> >>> > what >> >>> > I know, we are not using any direct memory buffers inside our code. >> >>> > This is pretty trivial streaming application just doing some >> >>> > dedupliction >> >>> > and union some kafka streams. >> >>> > >> >>> > /Jakob >> >>> > >> >>> > >> >>> > >> >>> > 2015-10-19 13:27:59,064 INFO >> >>> > org.apache.flink.runtime.taskmanager.Task >> >>> > - FilterAndTransform -> (Filter, Filter) (3/4) switched to FAILED >> >>> > with >> >>> > exception. >> >>> > >> >>> > >> >>> > org.apache.flink.runtime.io.network.netty.exception.LocalTransportException: >> >>> > java.lang.OutOfMemoryError: Direct buffer memory >> >>> > at >> >>> > >> >>> > >> >>> > org.apache.flink.runtime.io.network.netty.PartitionRequestClientHandler.exceptionCaught(PartitionRequestClientHandler.java:153) >> >>> > at >> >>> > >> >>> > >> >>> > io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:246) >> >>> > at >> >>> > >> >>> > >> >>> > io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:224) >> >>> > at >> >>> > >> >>> > >> >>> > io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) >> >>> > at >> >>> > >> >>> > >> >>> > io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:246) >> >>> > at >> >>> > >> >>> > >> >>> > io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:224) >> >>> > at >> >>> > >> >>> > >> >>> > io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) >> >>> > at >> >>> > >> >>> > >> >>> > io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:246) >> >>> > at >> >>> > >> >>> > >> >>> > io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:737) >> >>> > at >> >>> > >> >>> > >> >>> > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:310) >> >>> > at >> >>> > >> >>> > >> >>> > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294) >> >>> > at >> >>> > >> >>> > >> >>> > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846) >> >>> > at >> >>> > >> >>> > >> >>> > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) >> >>> > at >> >>> > >> >>> > >> >>> > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) >> >>> > at >> >>> > >> >>> > >> >>> > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) >> >>> > at >> >>> > >> >>> > >> >>> > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) >> >>> > at >> >>> > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) >> >>> > at >> >>> > >> >>> > >> >>> > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112) >> >>> > at java.lang.Thread.run(Thread.java:745) >> >>> > Caused by: io.netty.handler.codec.DecoderException: >> >>> > java.lang.OutOfMemoryError: Direct buffer memory >> >>> > at >> >>> > >> >>> > >> >>> > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:234) >> >>> > at >> >>> > >> >>> > >> >>> > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308) >> >>> > ... 9 more >> >>> > Caused by: java.lang.OutOfMemoryError: Direct buffer memory >> >>> > at java.nio.Bits.reserveMemory(Bits.java:658) >> >>> > at >> >>> > java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) >> >>> > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) >> >>> > at >> >>> > >> >>> > >> >>> > io.netty.buffer.PoolArena$DirectArena.newUnpooledChunk(PoolArena.java:651) >> >>> > at >> >>> > io.netty.buffer.PoolArena.allocateHuge(PoolArena.java:237) >> >>> > at io.netty.buffer.PoolArena.allocate(PoolArena.java:215) >> >>> > at io.netty.buffer.PoolArena.reallocate(PoolArena.java:358) >> >>> > at >> >>> > io.netty.buffer.PooledByteBuf.capacity(PooledByteBuf.java:111) >> >>> > at >> >>> > >> >>> > io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:251) >> >>> > at >> >>> > io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:849) >> >>> > at >> >>> > io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:841) >> >>> > at >> >>> > io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:831) >> >>> > at >> >>> > >> >>> > >> >>> > io.netty.handler.codec.ByteToMessageDecoder$1.cumulate(ByteToMessageDecoder.java:92) >> >>> > at >> >>> > >> >>> > >> >>> > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:228) >> >>> > ... 10 more >> >>> > >> >> >> >>
