The expected value "-900387334" is the little endian decimal
representation of the PROTOCOL_MAGIC value 0xCA552DFA defined in the
net/Message.java
<https://github.com/apache/cassandra/blob/c579faa488ec156a59ed8e15dd6db55759b9c942/src/java/org/apache/cassandra/net/Message.java#L393>
file.
The read value "-720899" converts to hex is 0xFFF4FFFD, that's not a
valid TLS header which should start with 0x16, so I don't think has
anything to do with the server encryption related options. It also does
not look like a valid version number from pre-4.0 Cassandra, so we can
rule that out too. Since it's nether a valid Cassandra 4.0+ magic, a TLS
header nor a pre-4.0 version number, I have reason to believe the
connection was not initiated by another Cassandra server for inter-node
communication, but from another program. Can you follow the source IP
and port number back to the originating host, and find out what is that
program? or indeed it was one of the servers in the cluster, not
something else, which could indicate a misconfiguration of the firewall
rules.
On 29/06/2023 01:26, MyWorld wrote:
Hi all,
We are currently using Apache cassandra 4.0.7 in our environment.
While adding a new node in the existing 3-node DC, we found below error.
This error is observed multiple times when the node was in the UJ (up
and joining) state.
Our current server-to-server internode encryption settings are default.
server_encryption_options:
internode_encryption: none
enable_legacy_ssl_storage_port: false
require_client_auth: false
require_endpoint_verification: false
Please help to debug the root cause of this error.
Is it a point to worry about or is it just a Warning issue?
Also, our API properties have received a few 5xx messages "Operation
timed out. received only 2 responses" during this time(addition of new
node), which we have not received when we were on the 3.11.x version.
What could be the possible reason?
However things are stable once the node comes to the UN state.
ERROR [Messaging-EventLoop-3-10] 2023-06-27 18:37:14,931
InboundConnectionInitiator.java:360 - Failed to properly handshake
with peer /x.x.x.x:35894. Closing the channel.
io.netty.handler.codec.DecoderException:
org.apache.cassandra.net.Message$InvalidLegacyProtocolMagic: Read
-720899, Expected -900387334
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:478)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at
io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
at
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
at
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by:
org.apache.cassandra.net.Message$InvalidLegacyProtocolMagic: Read
-720899, Expected -900387334
at
org.apache.cassandra.net.Message.validateLegacyProtocolMagic(Message.java:340)
at
org.apache.cassandra.net.HandshakeProtocol$Initiate.maybeDecode(HandshakeProtocol.java:167)
at
org.apache.cassandra.net.InboundConnectionInitiator$Handler.initiate(InboundConnectionInitiator.java:242)
at
org.apache.cassandra.net.InboundConnectionInitiator$Handler.decode(InboundConnectionInitiator.java:235)
at
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508)
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447)
... 15 common frames omitted
Regards,
Ashish
