Did anyone connecting to the servers' storage port via telnet, nc
(netcat) or something similar? 218762506 is 0x0D0A0D0A, which is two
newlines.
On 29/06/2023 11:49, MyWorld wrote:
When checked in the source nodes, we got similar errors.
Forgot to mention, we also received below error message :
ERROR [Messaging-EventLoop-3-3] 2023-06-27 18:57:09,128
InboundConnectionInitiator.java:360 - Failed to properly handshake
with peer /10.127.2.10:58490 <http://10.127.2.10:58490>. Closing the
channel.
io.netty.handler.codec.DecoderException:
org.apache.cassandra.net.Message$InvalidLegacyProtocolMagic: Read
218762506, Expected -900387334
On Thu, Jun 29, 2023 at 2:57 PM Bowen Song via user
<user@cassandra.apache.org> wrote:
The expected value "-900387334" is the little endian decimal
representation of the PROTOCOL_MAGIC value 0xCA552DFA defined in
the net/Message.java
<https://github.com/apache/cassandra/blob/c579faa488ec156a59ed8e15dd6db55759b9c942/src/java/org/apache/cassandra/net/Message.java#L393>
file.
The read value "-720899" converts to hex is 0xFFF4FFFD, that's not
a valid TLS header which should start with 0x16, so I don't think
has anything to do with the server encryption related options. It
also does not look like a valid version number from pre-4.0
Cassandra, so we can rule that out too. Since it's nether a valid
Cassandra 4.0+ magic, a TLS header nor a pre-4.0 version number, I
have reason to believe the connection was not initiated by another
Cassandra server for inter-node communication, but from another
program. Can you follow the source IP and port number back to the
originating host, and find out what is that program? or indeed it
was one of the servers in the cluster, not something else, which
could indicate a misconfiguration of the firewall rules.
On 29/06/2023 01:26, MyWorld wrote:
Hi all,
We are currently using Apache cassandra 4.0.7 in our environment.
While adding a new node in the existing 3-node DC, we found below
error.
This error is observed multiple times when the node was in the UJ
(up and joining) state.
Our current server-to-server internode encryption settings are
default.
server_encryption_options:
internode_encryption: none
enable_legacy_ssl_storage_port: false
require_client_auth: false
require_endpoint_verification: false
Please help to debug the root cause of this error.
Is it a point to worry about or is it just a Warning issue?
Also, our API properties have received a few 5xx messages
"Operation timed out. received only 2 responses" during this
time(addition of new node), which we have not received when we
were on the 3.11.x version. What could be the possible reason?
However things are stable once the node comes to the UN state.
ERROR [Messaging-EventLoop-3-10] 2023-06-27 18:37:14,931
InboundConnectionInitiator.java:360 - Failed to properly
handshake with peer /x.x.x.x:35894. Closing the channel.
io.netty.handler.codec.DecoderException:
org.apache.cassandra.net.Message$InvalidLegacyProtocolMagic: Read
-720899, Expected -900387334
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:478)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at
io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
at
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
at
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by:
org.apache.cassandra.net.Message$InvalidLegacyProtocolMagic: Read
-720899, Expected -900387334
at
org.apache.cassandra.net.Message.validateLegacyProtocolMagic(Message.java:340)
at
org.apache.cassandra.net.HandshakeProtocol$Initiate.maybeDecode(HandshakeProtocol.java:167)
at
org.apache.cassandra.net.InboundConnectionInitiator$Handler.initiate(InboundConnectionInitiator.java:242)
at
org.apache.cassandra.net.InboundConnectionInitiator$Handler.decode(InboundConnectionInitiator.java:235)
at
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508)
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447)
... 15 common frames omitted
Regards,
Ashish
