I am not aware that we use Nessus. We are running on Google Cloud Dataproc vanilla. On the other hand, Flink restarted from a checkpoint and reread the Kafka offset and didn't encounter this error again, so if it was something in the input I would expect it to happen again when it processes the Kafka record that caused the problem.
If it happens again I will enable TLS as described in the ticket. On Fri, Mar 18, 2022 at 8:57 AM Matthias Pohl <map...@apache.org> wrote: > One other pointer: Martijn mentioned in FLINK-24923 [1] tools like Nessus > could generate traffic while scanning for ports. It's just the size of the > request that is suspicious. > > [1] https://issues.apache.org/jira/browse/FLINK-24923 > > On Thu, Mar 17, 2022 at 5:29 PM Ori Popowski <ori....@gmail.com> wrote: > >> This issue did not repeat, so it may be a network issue >> >> On Thu, Mar 17, 2022 at 6:12 PM Matthias Pohl <map...@apache.org> wrote: >> >>> Hi Ori, >>> that looks odd. The message seems to exceed the maximum size >>> of 2147483647 bytes (2GB). I couldn't find anything similar in the ML or in >>> Jira that supports a bug in Flink. Could it be that there was some network >>> issue? >>> >>> Matthias >>> >>> On Tue, Mar 15, 2022 at 6:52 AM Ori Popowski <ori....@gmail.com> wrote: >>> >>>> I am running a production job for at least 1 year, and I got to day >>>> this error: >>>> >>>> >>>> org.apache.flink.runtime.io.network.netty.exception.LocalTransportException: >>>> Adjusted frame length exceeds 2147483647: 2969686273 - discarded >>>> (connection to >>>> 'flink-session-playback-prod-1641716499-sw-6q8p.c.data-prod-292614.internal/ >>>> 10.208.65.38:40737') >>>> >>>> Nothing was changed in the code for a long time. What's causing this >>>> error and how to fix it? I am running Flink 1.10.3 on YARN. >>>> >>>> This is the full stack trace: >>>> >>>> 2022-03-15 03:22:13 >>>> org.apache.flink.runtime.io.network.netty.exception. >>>> LocalTransportException: Adjusted frame length exceeds 2147483647: >>>> 2969686273 - discarded (connection to >>>> 'flink-session-playback-prod-1641716499-sw-6q8p.c.data-prod-292614.internal/ >>>> 10.208.65.38:40737') >>>> at org.apache.flink.runtime.io.network.netty. >>>> CreditBasedPartitionRequestClientHandler.exceptionCaught( >>>> CreditBasedPartitionRequestClientHandler.java:165) >>>> at org.apache.flink.shaded.netty4.io.netty.channel. >>>> AbstractChannelHandlerContext.invokeExceptionCaught( >>>> AbstractChannelHandlerContext.java:297) >>>> at org.apache.flink.shaded.netty4.io.netty.channel. >>>> AbstractChannelHandlerContext.invokeExceptionCaught( >>>> AbstractChannelHandlerContext.java:276) >>>> at org.apache.flink.shaded.netty4.io.netty.channel. >>>> AbstractChannelHandlerContext.fireExceptionCaught( >>>> AbstractChannelHandlerContext.java:268) >>>> at org.apache.flink.shaded.netty4.io.netty.channel. >>>> ChannelInboundHandlerAdapter.exceptionCaught( >>>> ChannelInboundHandlerAdapter.java:143) >>>> at org.apache.flink.shaded.netty4.io.netty.channel. >>>> AbstractChannelHandlerContext.invokeExceptionCaught( >>>> AbstractChannelHandlerContext.java:297) >>>> at org.apache.flink.shaded.netty4.io.netty.channel. >>>> AbstractChannelHandlerContext.notifyHandlerException( >>>> AbstractChannelHandlerContext.java:831) >>>> at org.apache.flink.shaded.netty4.io.netty.channel. >>>> AbstractChannelHandlerContext.invokeChannelRead( >>>> AbstractChannelHandlerContext.java:376) >>>> at org.apache.flink.shaded.netty4.io.netty.channel. >>>> AbstractChannelHandlerContext.invokeChannelRead( >>>> AbstractChannelHandlerContext.java:360) >>>> at org.apache.flink.shaded.netty4.io.netty.channel. >>>> AbstractChannelHandlerContext.fireChannelRead( >>>> AbstractChannelHandlerContext.java:352) >>>> at org.apache.flink.shaded.netty4.io.netty.channel. >>>> DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline >>>> .java:1421) >>>> at org.apache.flink.shaded.netty4.io.netty.channel. >>>> AbstractChannelHandlerContext.invokeChannelRead( >>>> AbstractChannelHandlerContext.java:374) >>>> at org.apache.flink.shaded.netty4.io.netty.channel. >>>> AbstractChannelHandlerContext.invokeChannelRead( >>>> AbstractChannelHandlerContext.java:360) >>>> at org.apache.flink.shaded.netty4.io.netty.channel. >>>> DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) >>>> at org.apache.flink.shaded.netty4.io.netty.channel.nio. >>>> AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java: >>>> 163) >>>> at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop >>>> .processSelectedKey(NioEventLoop.java:697) >>>> at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop >>>> .processSelectedKeysOptimized(NioEventLoop.java:632) >>>> at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop >>>> .processSelectedKeys(NioEventLoop.java:549) >>>> at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop >>>> .run(NioEventLoop.java:511) >>>> at org.apache.flink.shaded.netty4.io.netty.util.concurrent. >>>> SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) >>>> at org.apache.flink.shaded.netty4.io.netty.util.internal. >>>> ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) >>>> at java.lang.Thread.run(Thread.java:748) >>>> Caused by: org.apache.flink.shaded.netty4.io.netty.handler.codec. >>>> TooLongFrameException: Adjusted frame length exceeds 2147483647: >>>> 2969686273 - discarded >>>> at org.apache.flink.shaded.netty4.io.netty.handler.codec. >>>> LengthFieldBasedFrameDecoder.fail(LengthFieldBasedFrameDecoder.java:513 >>>> ) >>>> at org.apache.flink.shaded.netty4.io.netty.handler.codec. >>>> LengthFieldBasedFrameDecoder.failIfNecessary( >>>> LengthFieldBasedFrameDecoder.java:491) >>>> at org.apache.flink.shaded.netty4.io.netty.handler.codec. >>>> LengthFieldBasedFrameDecoder.exceededFrameLength( >>>> LengthFieldBasedFrameDecoder.java:378) >>>> at org.apache.flink.shaded.netty4.io.netty.handler.codec. >>>> LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java: >>>> 421) >>>> at org.apache.flink.runtime.io.network.netty. >>>> NettyMessage$NettyMessageDecoder.decode(NettyMessage.java:214) >>>> at org.apache.flink.shaded.netty4.io.netty.handler.codec. >>>> LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java: >>>> 334) >>>> at org.apache.flink.shaded.netty4.io.netty.handler.codec. >>>> ByteToMessageDecoder.decodeRemovalReentryProtection( >>>> ByteToMessageDecoder.java:505) >>>> at org.apache.flink.shaded.netty4.io.netty.handler.codec. >>>> ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444) >>>> at org.apache.flink.shaded.netty4.io.netty.handler.codec. >>>> ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283) >>>> at org.apache.flink.shaded.netty4.io.netty.channel. >>>> AbstractChannelHandlerContext.invokeChannelRead( >>>> AbstractChannelHandlerContext.java:374) >>>> ... 14 more >>>> >>>> >>>>
