I forgot to ask you: Which version of Flink are you using? 0.9.1 or 0.10-SNAPSHOT?
On Mon, Oct 19, 2015 at 5:05 PM, Maximilian Michels <m...@apache.org> wrote: > Hi Jakob, > > Thanks. Flink allocates its network memory as direct memory outside > the normal Java heap. By default, that is 64MB but can grow up to > 128MB on heavy network transfer. How much memory does your machine > have? Could it be that your upper memory bound is lower than 2048 + > 128 MB? > > Best, > Max > > On Mon, Oct 19, 2015 at 4:32 PM, Jakob Ericsson > <jakob.erics...@gmail.com> wrote: >> Hi, >> >> See answers below. >> >> /Jakob >> >> On Mon, Oct 19, 2015 at 4:03 PM, Maximilian Michels <m...@apache.org> wrote: >>> >>> Hi Jakob, >>> >>> Thank you for reporting the bug. Could you please post your >>> configuration here? In particular, could you please tell us the value >>> of the following configuration variables: >>> >>> taskmanager.heap.mb >> >> taskmanager.heap.mb: 2048 >>> >>> taskmanager.network.numberOfBuffers >> >> >> Default value. Not changed. >> >>> >>> taskmanager.memory.off-heap >>> >> Default value Not changed. >> >>> >>> Are you running the Flink cluster in batch or streaming mode? >>> >> Started in streaming mode. Running with two nodes. In the cluster. >> Also, I have set the "env.java.opts: -XX:+UseConcMarkSweepGC" due to some >> strange java core dumps in the G1 GC. >> >>> >>> Direct memory is used by Flink's network layer. My guess is that you >>> have set taskmanager.heap.mb too low (it constraints the number of >>> direct memory at the moment). >>> >>> Thank you, >>> Max >>> >>> >>> On Mon, Oct 19, 2015 at 3:24 PM, Jakob Ericsson >>> <jakob.erics...@gmail.com> wrote: >>> > Hello, >>> > >>> > We are running into a strange problem with Direct Memory buffers. From >>> > what >>> > I know, we are not using any direct memory buffers inside our code. >>> > This is pretty trivial streaming application just doing some >>> > dedupliction >>> > and union some kafka streams. >>> > >>> > /Jakob >>> > >>> > >>> > >>> > 2015-10-19 13:27:59,064 INFO org.apache.flink.runtime.taskmanager.Task >>> > - FilterAndTransform -> (Filter, Filter) (3/4) switched to FAILED with >>> > exception. >>> > >>> > org.apache.flink.runtime.io.network.netty.exception.LocalTransportException: >>> > java.lang.OutOfMemoryError: Direct buffer memory >>> > at >>> > >>> > org.apache.flink.runtime.io.network.netty.PartitionRequestClientHandler.exceptionCaught(PartitionRequestClientHandler.java:153) >>> > at >>> > >>> > io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:246) >>> > at >>> > >>> > io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:224) >>> > at >>> > >>> > io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) >>> > at >>> > >>> > io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:246) >>> > at >>> > >>> > io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:224) >>> > at >>> > >>> > io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) >>> > at >>> > >>> > io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:246) >>> > at >>> > >>> > io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:737) >>> > at >>> > >>> > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:310) >>> > at >>> > >>> > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294) >>> > at >>> > >>> > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846) >>> > at >>> > >>> > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) >>> > at >>> > >>> > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) >>> > at >>> > >>> > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) >>> > at >>> > >>> > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) >>> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) >>> > at >>> > >>> > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112) >>> > at java.lang.Thread.run(Thread.java:745) >>> > Caused by: io.netty.handler.codec.DecoderException: >>> > java.lang.OutOfMemoryError: Direct buffer memory >>> > at >>> > >>> > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:234) >>> > at >>> > >>> > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308) >>> > ... 9 more >>> > Caused by: java.lang.OutOfMemoryError: Direct buffer memory >>> > at java.nio.Bits.reserveMemory(Bits.java:658) >>> > at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) >>> > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) >>> > at >>> > >>> > io.netty.buffer.PoolArena$DirectArena.newUnpooledChunk(PoolArena.java:651) >>> > at io.netty.buffer.PoolArena.allocateHuge(PoolArena.java:237) >>> > at io.netty.buffer.PoolArena.allocate(PoolArena.java:215) >>> > at io.netty.buffer.PoolArena.reallocate(PoolArena.java:358) >>> > at >>> > io.netty.buffer.PooledByteBuf.capacity(PooledByteBuf.java:111) >>> > at >>> > io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:251) >>> > at >>> > io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:849) >>> > at >>> > io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:841) >>> > at >>> > io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:831) >>> > at >>> > >>> > io.netty.handler.codec.ByteToMessageDecoder$1.cumulate(ByteToMessageDecoder.java:92) >>> > at >>> > >>> > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:228) >>> > ... 10 more >>> > >> >>
