Hi Jakob, Thanks. Flink allocates its network memory as direct memory outside the normal Java heap. By default, that is 64MB but can grow up to 128MB on heavy network transfer. How much memory does your machine have? Could it be that your upper memory bound is lower than 2048 + 128 MB?
Best, Max On Mon, Oct 19, 2015 at 4:32 PM, Jakob Ericsson <jakob.erics...@gmail.com> wrote: > Hi, > > See answers below. > > /Jakob > > On Mon, Oct 19, 2015 at 4:03 PM, Maximilian Michels <m...@apache.org> wrote: >> >> Hi Jakob, >> >> Thank you for reporting the bug. Could you please post your >> configuration here? In particular, could you please tell us the value >> of the following configuration variables: >> >> taskmanager.heap.mb > > taskmanager.heap.mb: 2048 >> >> taskmanager.network.numberOfBuffers > > > Default value. Not changed. > >> >> taskmanager.memory.off-heap >> > Default value Not changed. > >> >> Are you running the Flink cluster in batch or streaming mode? >> > Started in streaming mode. Running with two nodes. In the cluster. > Also, I have set the "env.java.opts: -XX:+UseConcMarkSweepGC" due to some > strange java core dumps in the G1 GC. > >> >> Direct memory is used by Flink's network layer. My guess is that you >> have set taskmanager.heap.mb too low (it constraints the number of >> direct memory at the moment). >> >> Thank you, >> Max >> >> >> On Mon, Oct 19, 2015 at 3:24 PM, Jakob Ericsson >> <jakob.erics...@gmail.com> wrote: >> > Hello, >> > >> > We are running into a strange problem with Direct Memory buffers. From >> > what >> > I know, we are not using any direct memory buffers inside our code. >> > This is pretty trivial streaming application just doing some >> > dedupliction >> > and union some kafka streams. >> > >> > /Jakob >> > >> > >> > >> > 2015-10-19 13:27:59,064 INFO org.apache.flink.runtime.taskmanager.Task >> > - FilterAndTransform -> (Filter, Filter) (3/4) switched to FAILED with >> > exception. >> > >> > org.apache.flink.runtime.io.network.netty.exception.LocalTransportException: >> > java.lang.OutOfMemoryError: Direct buffer memory >> > at >> > >> > org.apache.flink.runtime.io.network.netty.PartitionRequestClientHandler.exceptionCaught(PartitionRequestClientHandler.java:153) >> > at >> > >> > io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:246) >> > at >> > >> > io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:224) >> > at >> > >> > io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) >> > at >> > >> > io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:246) >> > at >> > >> > io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:224) >> > at >> > >> > io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) >> > at >> > >> > io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:246) >> > at >> > >> > io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:737) >> > at >> > >> > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:310) >> > at >> > >> > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294) >> > at >> > >> > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846) >> > at >> > >> > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) >> > at >> > >> > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) >> > at >> > >> > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) >> > at >> > >> > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) >> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) >> > at >> > >> > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112) >> > at java.lang.Thread.run(Thread.java:745) >> > Caused by: io.netty.handler.codec.DecoderException: >> > java.lang.OutOfMemoryError: Direct buffer memory >> > at >> > >> > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:234) >> > at >> > >> > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308) >> > ... 9 more >> > Caused by: java.lang.OutOfMemoryError: Direct buffer memory >> > at java.nio.Bits.reserveMemory(Bits.java:658) >> > at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) >> > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) >> > at >> > >> > io.netty.buffer.PoolArena$DirectArena.newUnpooledChunk(PoolArena.java:651) >> > at io.netty.buffer.PoolArena.allocateHuge(PoolArena.java:237) >> > at io.netty.buffer.PoolArena.allocate(PoolArena.java:215) >> > at io.netty.buffer.PoolArena.reallocate(PoolArena.java:358) >> > at >> > io.netty.buffer.PooledByteBuf.capacity(PooledByteBuf.java:111) >> > at >> > io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:251) >> > at >> > io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:849) >> > at >> > io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:841) >> > at >> > io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:831) >> > at >> > >> > io.netty.handler.codec.ByteToMessageDecoder$1.cumulate(ByteToMessageDecoder.java:92) >> > at >> > >> > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:228) >> > ... 10 more >> > > >
