When checked in the source nodes, we got similar errors. Forgot to mention, we also received below error message : ERROR [Messaging-EventLoop-3-3] 2023-06-27 18:57:09,128 InboundConnectionInitiator.java:360 - Failed to properly handshake with peer /10.127.2.10:58490. Closing the channel. io.netty.handler.codec.DecoderException: org.apache.cassandra.net.Message$InvalidLegacyProtocolMagic: Read 218762506, Expected -900387334
On Thu, Jun 29, 2023 at 2:57 PM Bowen Song via user < user@cassandra.apache.org> wrote: > The expected value "-900387334" is the little endian decimal > representation of the PROTOCOL_MAGIC value 0xCA552DFA defined in the > net/Message.java > <https://github.com/apache/cassandra/blob/c579faa488ec156a59ed8e15dd6db55759b9c942/src/java/org/apache/cassandra/net/Message.java#L393> > file. > > The read value "-720899" converts to hex is 0xFFF4FFFD, that's not a > valid TLS header which should start with 0x16, so I don't think has > anything to do with the server encryption related options. It also does not > look like a valid version number from pre-4.0 Cassandra, so we can rule > that out too. Since it's nether a valid Cassandra 4.0+ magic, a TLS header > nor a pre-4.0 version number, I have reason to believe the connection was > not initiated by another Cassandra server for inter-node communication, but > from another program. Can you follow the source IP and port number back to > the originating host, and find out what is that program? or indeed it was > one of the servers in the cluster, not something else, which could indicate > a misconfiguration of the firewall rules. > > > On 29/06/2023 01:26, MyWorld wrote: > > Hi all, > We are currently using Apache cassandra 4.0.7 in our environment. While > adding a new node in the existing 3-node DC, we found below error. > This error is observed multiple times when the node was in the UJ (up and > joining) state. > > Our current server-to-server internode encryption settings are default. > server_encryption_options: > internode_encryption: none > enable_legacy_ssl_storage_port: false > require_client_auth: false > require_endpoint_verification: false > > Please help to debug the root cause of this error. > Is it a point to worry about or is it just a Warning issue? > Also, our API properties have received a few 5xx messages "Operation timed > out. received only 2 responses" during this time(addition of new node), > which we have not received when we were on the 3.11.x version. What > could be the possible reason? > However things are stable once the node comes to the UN state. > > ERROR [Messaging-EventLoop-3-10] 2023-06-27 18:37:14,931 > InboundConnectionInitiator.java:360 - Failed to properly handshake with > peer /x.x.x.x:35894. Closing the channel. > io.netty.handler.codec.DecoderException: > org.apache.cassandra.net.Message$InvalidLegacyProtocolMagic: Read -720899, > Expected -900387334 > at > io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:478) > at > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) > at > io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) > at > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) > at > io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) > at > io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) > at > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) > at > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) > at > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:829) > Caused by: org.apache.cassandra.net.Message$InvalidLegacyProtocolMagic: > Read -720899, Expected -900387334 > at > org.apache.cassandra.net.Message.validateLegacyProtocolMagic(Message.java:340) > at > org.apache.cassandra.net.HandshakeProtocol$Initiate.maybeDecode(HandshakeProtocol.java:167) > at > org.apache.cassandra.net.InboundConnectionInitiator$Handler.initiate(InboundConnectionInitiator.java:242) > at > org.apache.cassandra.net.InboundConnectionInitiator$Handler.decode(InboundConnectionInitiator.java:235) > at > io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508) > at > io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447) > ... 15 common frames omitted > > Regards, > Ashish > >
