Bingo.. Yes API team was cross checkin their whitelisting to the db server at the same time and checking via telnet. Thanks Bowen
On Thu, Jun 29, 2023 at 5:49 PM Bowen Song via user < user@cassandra.apache.org> wrote: > Talking about telnet, a closer look at the sequence 'FF F4 FF FD' makes > me thinking about telnet commands code. Based on RFC 854 > <https://www.rfc-editor.org/rfc/rfc854.html>, the sequence is IAC, > Interrupt Process, IAC, DO, which is basically the key sequence 'ctrl-c' > in telnet. > On 29/06/2023 12:42, Bowen Song wrote: > > Did anyone connecting to the servers' storage port via telnet, nc (netcat) > or something similar? 218762506 is 0x0D0A0D0A, which is two newlines. > > > On 29/06/2023 11:49, MyWorld wrote: > > When checked in the source nodes, we got similar errors. > > Forgot to mention, we also received below error message : > ERROR [Messaging-EventLoop-3-3] 2023-06-27 18:57:09,128 > InboundConnectionInitiator.java:360 - Failed to properly handshake with > peer /10.127.2.10:58490. Closing the channel. > io.netty.handler.codec.DecoderException: > org.apache.cassandra.net.Message$InvalidLegacyProtocolMagic: Read > 218762506, Expected -900387334 > > On Thu, Jun 29, 2023 at 2:57 PM Bowen Song via user < > user@cassandra.apache.org> wrote: > >> The expected value "-900387334" is the little endian decimal >> representation of the PROTOCOL_MAGIC value 0xCA552DFA defined in the >> net/Message.java >> <https://github.com/apache/cassandra/blob/c579faa488ec156a59ed8e15dd6db55759b9c942/src/java/org/apache/cassandra/net/Message.java#L393> >> file. >> >> The read value "-720899" converts to hex is 0xFFF4FFFD, that's not a >> valid TLS header which should start with 0x16, so I don't think has >> anything to do with the server encryption related options. It also does not >> look like a valid version number from pre-4.0 Cassandra, so we can rule >> that out too. Since it's nether a valid Cassandra 4.0+ magic, a TLS header >> nor a pre-4.0 version number, I have reason to believe the connection was >> not initiated by another Cassandra server for inter-node communication, but >> from another program. Can you follow the source IP and port number back to >> the originating host, and find out what is that program? or indeed it was >> one of the servers in the cluster, not something else, which could indicate >> a misconfiguration of the firewall rules. >> >> >> On 29/06/2023 01:26, MyWorld wrote: >> >> Hi all, >> We are currently using Apache cassandra 4.0.7 in our environment. While >> adding a new node in the existing 3-node DC, we found below error. >> This error is observed multiple times when the node was in the UJ (up and >> joining) state. >> >> Our current server-to-server internode encryption settings are default. >> server_encryption_options: >> internode_encryption: none >> enable_legacy_ssl_storage_port: false >> require_client_auth: false >> require_endpoint_verification: false >> >> Please help to debug the root cause of this error. >> Is it a point to worry about or is it just a Warning issue? >> Also, our API properties have received a few 5xx messages "Operation >> timed out. received only 2 responses" during this time(addition of new >> node), which we have not received when we were on the 3.11.x version. What >> could be the possible reason? >> However things are stable once the node comes to the UN state. >> >> ERROR [Messaging-EventLoop-3-10] 2023-06-27 18:37:14,931 >> InboundConnectionInitiator.java:360 - Failed to properly handshake with >> peer /x.x.x.x:35894. Closing the channel. >> io.netty.handler.codec.DecoderException: >> org.apache.cassandra.net.Message$InvalidLegacyProtocolMagic: Read -720899, >> Expected -900387334 >> at >> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:478) >> at >> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) >> at >> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) >> at >> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) >> at >> io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) >> at >> io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) >> at >> io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) >> at >> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) >> at >> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) >> at >> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) >> at java.base/java.lang.Thread.run(Thread.java:829) >> Caused by: org.apache.cassandra.net.Message$InvalidLegacyProtocolMagic: >> Read -720899, Expected -900387334 >> at >> org.apache.cassandra.net.Message.validateLegacyProtocolMagic(Message.java:340) >> at >> org.apache.cassandra.net.HandshakeProtocol$Initiate.maybeDecode(HandshakeProtocol.java:167) >> at >> org.apache.cassandra.net.InboundConnectionInitiator$Handler.initiate(InboundConnectionInitiator.java:242) >> at >> org.apache.cassandra.net.InboundConnectionInitiator$Handler.decode(InboundConnectionInitiator.java:235) >> at >> io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508) >> at >> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447) >> ... 15 common frames omitted >> >> Regards, >> Ashish >> >>
