You can see the revision number and the build date in the JobManager log file, e.g. "Starting JobManager (Version: 0.10-SNAPSHOT, Rev:1b79bc1, Date:18.10.2015 @ 20:15:08 CEST)"
On Mon, Oct 19, 2015 at 5:53 PM, Maximilian Michels <m...@apache.org> wrote: > When was the last time you updated your 0.10-SNAPSHOT Flink cluster? > If it has been more than a couple of weeks, then I'd advise you to > update to the latest snapshot version. There has been an issue with > the calculation of the off-heap memory limit in the past. > > Thanks, > Max > > On Mon, Oct 19, 2015 at 5:26 PM, Gyula Fóra <gyula.f...@gmail.com> wrote: >> It's 0.10-SNAPSHOT >> >> Gyula >> >> Maximilian Michels <m...@apache.org> ezt írta (időpont: 2015. okt. 19., H, >> 17:13): >>> >>> I forgot to ask you: Which version of Flink are you using? 0.9.1 or >>> 0.10-SNAPSHOT? >>> >>> On Mon, Oct 19, 2015 at 5:05 PM, Maximilian Michels <m...@apache.org> >>> wrote: >>> > Hi Jakob, >>> > >>> > Thanks. Flink allocates its network memory as direct memory outside >>> > the normal Java heap. By default, that is 64MB but can grow up to >>> > 128MB on heavy network transfer. How much memory does your machine >>> > have? Could it be that your upper memory bound is lower than 2048 + >>> > 128 MB? >>> > >>> > Best, >>> > Max >>> > >>> > On Mon, Oct 19, 2015 at 4:32 PM, Jakob Ericsson >>> > <jakob.erics...@gmail.com> wrote: >>> >> Hi, >>> >> >>> >> See answers below. >>> >> >>> >> /Jakob >>> >> >>> >> On Mon, Oct 19, 2015 at 4:03 PM, Maximilian Michels <m...@apache.org> >>> >> wrote: >>> >>> >>> >>> Hi Jakob, >>> >>> >>> >>> Thank you for reporting the bug. Could you please post your >>> >>> configuration here? In particular, could you please tell us the value >>> >>> of the following configuration variables: >>> >>> >>> >>> taskmanager.heap.mb >>> >> >>> >> taskmanager.heap.mb: 2048 >>> >>> >>> >>> taskmanager.network.numberOfBuffers >>> >> >>> >> >>> >> Default value. Not changed. >>> >> >>> >>> >>> >>> taskmanager.memory.off-heap >>> >>> >>> >> Default value Not changed. >>> >> >>> >>> >>> >>> Are you running the Flink cluster in batch or streaming mode? >>> >>> >>> >> Started in streaming mode. Running with two nodes. In the cluster. >>> >> Also, I have set the "env.java.opts: -XX:+UseConcMarkSweepGC" due to >>> >> some >>> >> strange java core dumps in the G1 GC. >>> >> >>> >>> >>> >>> Direct memory is used by Flink's network layer. My guess is that you >>> >>> have set taskmanager.heap.mb too low (it constraints the number of >>> >>> direct memory at the moment). >>> >>> >>> >>> Thank you, >>> >>> Max >>> >>> >>> >>> >>> >>> On Mon, Oct 19, 2015 at 3:24 PM, Jakob Ericsson >>> >>> <jakob.erics...@gmail.com> wrote: >>> >>> > Hello, >>> >>> > >>> >>> > We are running into a strange problem with Direct Memory buffers. >>> >>> > From >>> >>> > what >>> >>> > I know, we are not using any direct memory buffers inside our code. >>> >>> > This is pretty trivial streaming application just doing some >>> >>> > dedupliction >>> >>> > and union some kafka streams. >>> >>> > >>> >>> > /Jakob >>> >>> > >>> >>> > >>> >>> > >>> >>> > 2015-10-19 13:27:59,064 INFO >>> >>> > org.apache.flink.runtime.taskmanager.Task >>> >>> > - FilterAndTransform -> (Filter, Filter) (3/4) switched to FAILED >>> >>> > with >>> >>> > exception. >>> >>> > >>> >>> > >>> >>> > org.apache.flink.runtime.io.network.netty.exception.LocalTransportException: >>> >>> > java.lang.OutOfMemoryError: Direct buffer memory >>> >>> > at >>> >>> > >>> >>> > >>> >>> > org.apache.flink.runtime.io.network.netty.PartitionRequestClientHandler.exceptionCaught(PartitionRequestClientHandler.java:153) >>> >>> > at >>> >>> > >>> >>> > >>> >>> > io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:246) >>> >>> > at >>> >>> > >>> >>> > >>> >>> > io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:224) >>> >>> > at >>> >>> > >>> >>> > >>> >>> > io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) >>> >>> > at >>> >>> > >>> >>> > >>> >>> > io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:246) >>> >>> > at >>> >>> > >>> >>> > >>> >>> > io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:224) >>> >>> > at >>> >>> > >>> >>> > >>> >>> > io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) >>> >>> > at >>> >>> > >>> >>> > >>> >>> > io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:246) >>> >>> > at >>> >>> > >>> >>> > >>> >>> > io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:737) >>> >>> > at >>> >>> > >>> >>> > >>> >>> > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:310) >>> >>> > at >>> >>> > >>> >>> > >>> >>> > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294) >>> >>> > at >>> >>> > >>> >>> > >>> >>> > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846) >>> >>> > at >>> >>> > >>> >>> > >>> >>> > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) >>> >>> > at >>> >>> > >>> >>> > >>> >>> > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) >>> >>> > at >>> >>> > >>> >>> > >>> >>> > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) >>> >>> > at >>> >>> > >>> >>> > >>> >>> > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) >>> >>> > at >>> >>> > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) >>> >>> > at >>> >>> > >>> >>> > >>> >>> > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112) >>> >>> > at java.lang.Thread.run(Thread.java:745) >>> >>> > Caused by: io.netty.handler.codec.DecoderException: >>> >>> > java.lang.OutOfMemoryError: Direct buffer memory >>> >>> > at >>> >>> > >>> >>> > >>> >>> > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:234) >>> >>> > at >>> >>> > >>> >>> > >>> >>> > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308) >>> >>> > ... 9 more >>> >>> > Caused by: java.lang.OutOfMemoryError: Direct buffer memory >>> >>> > at java.nio.Bits.reserveMemory(Bits.java:658) >>> >>> > at >>> >>> > java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) >>> >>> > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) >>> >>> > at >>> >>> > >>> >>> > >>> >>> > io.netty.buffer.PoolArena$DirectArena.newUnpooledChunk(PoolArena.java:651) >>> >>> > at >>> >>> > io.netty.buffer.PoolArena.allocateHuge(PoolArena.java:237) >>> >>> > at io.netty.buffer.PoolArena.allocate(PoolArena.java:215) >>> >>> > at io.netty.buffer.PoolArena.reallocate(PoolArena.java:358) >>> >>> > at >>> >>> > io.netty.buffer.PooledByteBuf.capacity(PooledByteBuf.java:111) >>> >>> > at >>> >>> > >>> >>> > io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:251) >>> >>> > at >>> >>> > io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:849) >>> >>> > at >>> >>> > io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:841) >>> >>> > at >>> >>> > io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:831) >>> >>> > at >>> >>> > >>> >>> > >>> >>> > io.netty.handler.codec.ByteToMessageDecoder$1.cumulate(ByteToMessageDecoder.java:92) >>> >>> > at >>> >>> > >>> >>> > >>> >>> > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:228) >>> >>> > ... 10 more >>> >>> > >>> >> >>> >>
