[ 
https://issues.apache.org/jira/browse/IGNITE-24369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Roman Puchkovskiy updated IGNITE-24369:
---------------------------------------
    Description: 
When a networking protocol handshake (server-to-server) occurs, one of the 
participants is called a 'client', another is a 'server'. Client sends a 
ProbeMessage to make sure we detect a broken channel. So, only the server can 
get a ProbeMessage. There is the corresponding code in 
RecoveryServerHandshakeManager:
 
{{if (message instanceof ProbeMessage) {}}
{{    // No action required, just ignore it.}}
{{    return;}}
{{}}}
 
As the client should never get such a message, there is no such code, so, if 
ProbeMessage arrives in RecoveryClientHandshakeManager#onMessage(), an 
assertion should happen. And sometimes we actually see such assertions (though 
very rarely):
 
java.lang.AssertionError: Wrong client handshake flow, message is 
ProbeMessageImpl [dummy=0]
 
This means that sometimes a client gets a message that it should never get. 
This has to be investigated.

An example stack trace:
{noformat}
14:22:08     
[2025-06-04T14:22:08,146][ERROR][icft_tscoisd_40000-network-worker-2][HandshakeHandler]
 Error when performing handshake
14:22:08     java.lang.AssertionError: Wrong initiator handshake flow, message 
is ProbeMessageImpl [dummy=0]
14:22:08       at 
org.apache.ignite.internal.network.recovery.RecoveryInitiatorHandshakeManager.onMessage(RecoveryInitiatorHandshakeManager.java:214)
 ~[ignite-network-9.1.127-SNAPSHOT.jar:?]
14:22:08       at 
org.apache.ignite.internal.network.netty.HandshakeHandler.channelRead(HandshakeHandler.java:98)
 ~[ignite-network-9.1.127-SNAPSHOT.jar:?]
14:22:08       at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
 [netty-transport-4.1.119.Final.jar:4.1.119.Final]
14:22:08       at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
 [netty-transport-4.1.119.Final.jar:4.1.119.Final]
14:22:08       at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
 [netty-transport-4.1.119.Final.jar:4.1.119.Final]
14:22:08       at 
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
 [netty-codec-4.1.119.Final.jar:4.1.119.Final]
14:22:08       at 
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
 [netty-codec-4.1.119.Final.jar:4.1.119.Final]
14:22:08       at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
 [netty-transport-4.1.119.Final.jar:4.1.119.Final]
14:22:08       at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
 [netty-transport-4.1.119.Final.jar:4.1.119.Final]
14:22:08       at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
 [netty-transport-4.1.119.Final.jar:4.1.119.Final]
14:22:08       at 
io.netty.handler.flush.FlushConsolidationHandler.channelRead(FlushConsolidationHandler.java:152)
 [netty-handler-4.1.119.Final.jar:4.1.119.Final]
14:22:08       at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
 [netty-transport-4.1.119.Final.jar:4.1.119.Final]
14:22:08       at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
 [netty-transport-4.1.119.Final.jar:4.1.119.Final]
14:22:08       at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
 [netty-transport-4.1.119.Final.jar:4.1.119.Final]
14:22:08       at 
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
 [netty-transport-4.1.119.Final.jar:4.1.119.Final]
14:22:08       at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
 [netty-transport-4.1.119.Final.jar:4.1.119.Final]
14:22:08       at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
 [netty-transport-4.1.119.Final.jar:4.1.119.Final]
14:22:08       at 
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
 [netty-transport-4.1.119.Final.jar:4.1.119.Final]
14:22:08       at 
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
 [netty-transport-4.1.119.Final.jar:4.1.119.Final]
14:22:08       at 
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796) 
[netty-transport-4.1.119.Final.jar:4.1.119.Final]
14:22:08       at 
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:732)
 [netty-transport-4.1.119.Final.jar:4.1.119.Final]
14:22:08       at 
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:658) 
[netty-transport-4.1.119.Final.jar:4.1.119.Final]
14:22:08       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) 
[netty-transport-4.1.119.Final.jar:4.1.119.Final]
14:22:08       at 
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
 [netty-common-4.1.119.Final.jar:4.1.119.Final]
14:22:08       at 
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) 
[netty-common-4.1.119.Final.jar:4.1.119.Final]
14:22:08       at 
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
 [netty-common-4.1.119.Final.jar:4.1.119.Final]
14:22:08       at java.base/java.lang.Thread.run(Thread.java:833) 
[?:?]{noformat}

  was:
When a networking protocol handshake (server-to-server) occurs, one of the 
participants is called a 'client', another is a 'server'. Client sends a 
ProbeMessage to make sure we detect a broken channel. So, only the server can 
get a ProbeMessage. There is the corresponding code in 
RecoveryServerHandshakeManager:
 
{{if (message instanceof ProbeMessage) {}}
{{    // No action required, just ignore it.}}
{{    return;}}
{{}}}
 
As the client should never get such a message, there is no such code, so, if 
ProbeMessage arrives in RecoveryClientHandshakeManager#onMessage(), an 
assertion should happen. And sometimes we actually see such assertions (though 
very rarely):
 
java.lang.AssertionError: Wrong client handshake flow, message is 
ProbeMessageImpl [dummy=0]
 
This means that sometimes a client gets a message that it should never get. 
This has to be investigated.


> ProbeMessage arrives at client handshake handler
> ------------------------------------------------
>
>                 Key: IGNITE-24369
>                 URL: https://issues.apache.org/jira/browse/IGNITE-24369
>             Project: Ignite
>          Issue Type: Bug
>            Reporter: Roman Puchkovskiy
>            Priority: Major
>              Labels: ignite-3
>
> When a networking protocol handshake (server-to-server) occurs, one of the 
> participants is called a 'client', another is a 'server'. Client sends a 
> ProbeMessage to make sure we detect a broken channel. So, only the server can 
> get a ProbeMessage. There is the corresponding code in 
> RecoveryServerHandshakeManager:
>  
> {{if (message instanceof ProbeMessage) {}}
> {{    // No action required, just ignore it.}}
> {{    return;}}
> {{}}}
>  
> As the client should never get such a message, there is no such code, so, if 
> ProbeMessage arrives in RecoveryClientHandshakeManager#onMessage(), an 
> assertion should happen. And sometimes we actually see such assertions 
> (though very rarely):
>  
> java.lang.AssertionError: Wrong client handshake flow, message is 
> ProbeMessageImpl [dummy=0]
>  
> This means that sometimes a client gets a message that it should never get. 
> This has to be investigated.
> An example stack trace:
> {noformat}
> 14:22:08     
> [2025-06-04T14:22:08,146][ERROR][icft_tscoisd_40000-network-worker-2][HandshakeHandler]
>  Error when performing handshake
> 14:22:08     java.lang.AssertionError: Wrong initiator handshake flow, 
> message is ProbeMessageImpl [dummy=0]
> 14:22:08       at 
> org.apache.ignite.internal.network.recovery.RecoveryInitiatorHandshakeManager.onMessage(RecoveryInitiatorHandshakeManager.java:214)
>  ~[ignite-network-9.1.127-SNAPSHOT.jar:?]
> 14:22:08       at 
> org.apache.ignite.internal.network.netty.HandshakeHandler.channelRead(HandshakeHandler.java:98)
>  ~[ignite-network-9.1.127-SNAPSHOT.jar:?]
> 14:22:08       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>  [netty-transport-4.1.119.Final.jar:4.1.119.Final]
> 14:22:08       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>  [netty-transport-4.1.119.Final.jar:4.1.119.Final]
> 14:22:08       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>  [netty-transport-4.1.119.Final.jar:4.1.119.Final]
> 14:22:08       at 
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
>  [netty-codec-4.1.119.Final.jar:4.1.119.Final]
> 14:22:08       at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
>  [netty-codec-4.1.119.Final.jar:4.1.119.Final]
> 14:22:08       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>  [netty-transport-4.1.119.Final.jar:4.1.119.Final]
> 14:22:08       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>  [netty-transport-4.1.119.Final.jar:4.1.119.Final]
> 14:22:08       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>  [netty-transport-4.1.119.Final.jar:4.1.119.Final]
> 14:22:08       at 
> io.netty.handler.flush.FlushConsolidationHandler.channelRead(FlushConsolidationHandler.java:152)
>  [netty-handler-4.1.119.Final.jar:4.1.119.Final]
> 14:22:08       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
>  [netty-transport-4.1.119.Final.jar:4.1.119.Final]
> 14:22:08       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>  [netty-transport-4.1.119.Final.jar:4.1.119.Final]
> 14:22:08       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>  [netty-transport-4.1.119.Final.jar:4.1.119.Final]
> 14:22:08       at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
>  [netty-transport-4.1.119.Final.jar:4.1.119.Final]
> 14:22:08       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
>  [netty-transport-4.1.119.Final.jar:4.1.119.Final]
> 14:22:08       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>  [netty-transport-4.1.119.Final.jar:4.1.119.Final]
> 14:22:08       at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
>  [netty-transport-4.1.119.Final.jar:4.1.119.Final]
> 14:22:08       at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
>  [netty-transport-4.1.119.Final.jar:4.1.119.Final]
> 14:22:08       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796) 
> [netty-transport-4.1.119.Final.jar:4.1.119.Final]
> 14:22:08       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:732)
>  [netty-transport-4.1.119.Final.jar:4.1.119.Final]
> 14:22:08       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:658) 
> [netty-transport-4.1.119.Final.jar:4.1.119.Final]
> 14:22:08       at 
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) 
> [netty-transport-4.1.119.Final.jar:4.1.119.Final]
> 14:22:08       at 
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
>  [netty-common-4.1.119.Final.jar:4.1.119.Final]
> 14:22:08       at 
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) 
> [netty-common-4.1.119.Final.jar:4.1.119.Final]
> 14:22:08       at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>  [netty-common-4.1.119.Final.jar:4.1.119.Final]
> 14:22:08       at java.base/java.lang.Thread.run(Thread.java:833) 
> [?:?]{noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to