Hi Ori, that looks odd. The message seems to exceed the maximum size of 2147483647 bytes (2GB). I couldn't find anything similar in the ML or in Jira that supports a bug in Flink. Could it be that there was some network issue?
Matthias On Tue, Mar 15, 2022 at 6:52 AM Ori Popowski <ori....@gmail.com> wrote: > I am running a production job for at least 1 year, and I got to day this > error: > > > org.apache.flink.runtime.io.network.netty.exception.LocalTransportException: > Adjusted frame length exceeds 2147483647: 2969686273 - discarded > (connection to > 'flink-session-playback-prod-1641716499-sw-6q8p.c.data-prod-292614.internal/ > 10.208.65.38:40737') > > Nothing was changed in the code for a long time. What's causing this error > and how to fix it? I am running Flink 1.10.3 on YARN. > > This is the full stack trace: > > 2022-03-15 03:22:13 > org.apache.flink.runtime.io.network.netty.exception. > LocalTransportException: Adjusted frame length exceeds 2147483647: > 2969686273 - discarded (connection to > 'flink-session-playback-prod-1641716499-sw-6q8p.c.data-prod-292614.internal/ > 10.208.65.38:40737') > at org.apache.flink.runtime.io.network.netty. > CreditBasedPartitionRequestClientHandler.exceptionCaught( > CreditBasedPartitionRequestClientHandler.java:165) > at org.apache.flink.shaded.netty4.io.netty.channel. > AbstractChannelHandlerContext.invokeExceptionCaught( > AbstractChannelHandlerContext.java:297) > at org.apache.flink.shaded.netty4.io.netty.channel. > AbstractChannelHandlerContext.invokeExceptionCaught( > AbstractChannelHandlerContext.java:276) > at org.apache.flink.shaded.netty4.io.netty.channel. > AbstractChannelHandlerContext.fireExceptionCaught( > AbstractChannelHandlerContext.java:268) > at org.apache.flink.shaded.netty4.io.netty.channel. > ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter > .java:143) > at org.apache.flink.shaded.netty4.io.netty.channel. > AbstractChannelHandlerContext.invokeExceptionCaught( > AbstractChannelHandlerContext.java:297) > at org.apache.flink.shaded.netty4.io.netty.channel. > AbstractChannelHandlerContext.notifyHandlerException( > AbstractChannelHandlerContext.java:831) > at org.apache.flink.shaded.netty4.io.netty.channel. > AbstractChannelHandlerContext.invokeChannelRead( > AbstractChannelHandlerContext.java:376) > at org.apache.flink.shaded.netty4.io.netty.channel. > AbstractChannelHandlerContext.invokeChannelRead( > AbstractChannelHandlerContext.java:360) > at org.apache.flink.shaded.netty4.io.netty.channel. > AbstractChannelHandlerContext.fireChannelRead( > AbstractChannelHandlerContext.java:352) > at org.apache.flink.shaded.netty4.io.netty.channel. > DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline > .java:1421) > at org.apache.flink.shaded.netty4.io.netty.channel. > AbstractChannelHandlerContext.invokeChannelRead( > AbstractChannelHandlerContext.java:374) > at org.apache.flink.shaded.netty4.io.netty.channel. > AbstractChannelHandlerContext.invokeChannelRead( > AbstractChannelHandlerContext.java:360) > at org.apache.flink.shaded.netty4.io.netty.channel. > DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) > at org.apache.flink.shaded.netty4.io.netty.channel.nio. > AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) > at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop > .processSelectedKey(NioEventLoop.java:697) > at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop > .processSelectedKeysOptimized(NioEventLoop.java:632) > at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop > .processSelectedKeys(NioEventLoop.java:549) > at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop > .run(NioEventLoop.java:511) > at org.apache.flink.shaded.netty4.io.netty.util.concurrent. > SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) > at org.apache.flink.shaded.netty4.io.netty.util.internal. > ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.flink.shaded.netty4.io.netty.handler.codec. > TooLongFrameException: Adjusted frame length exceeds 2147483647: > 2969686273 - discarded > at org.apache.flink.shaded.netty4.io.netty.handler.codec. > LengthFieldBasedFrameDecoder.fail(LengthFieldBasedFrameDecoder.java:513) > at org.apache.flink.shaded.netty4.io.netty.handler.codec. > LengthFieldBasedFrameDecoder.failIfNecessary(LengthFieldBasedFrameDecoder > .java:491) > at org.apache.flink.shaded.netty4.io.netty.handler.codec. > LengthFieldBasedFrameDecoder.exceededFrameLength( > LengthFieldBasedFrameDecoder.java:378) > at org.apache.flink.shaded.netty4.io.netty.handler.codec. > LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:421) > at org.apache.flink.runtime.io.network.netty. > NettyMessage$NettyMessageDecoder.decode(NettyMessage.java:214) > at org.apache.flink.shaded.netty4.io.netty.handler.codec. > LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:334) > at org.apache.flink.shaded.netty4.io.netty.handler.codec. > ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder > .java:505) > at org.apache.flink.shaded.netty4.io.netty.handler.codec. > ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444) > at org.apache.flink.shaded.netty4.io.netty.handler.codec. > ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283) > at org.apache.flink.shaded.netty4.io.netty.channel. > AbstractChannelHandlerContext.invokeChannelRead( > AbstractChannelHandlerContext.java:374) > ... 14 more > > >
