[
https://issues.apache.org/jira/browse/CASSANDRA-16156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Petrov updated CASSANDRA-16156:
------------------------------------
Description:
After node is decommissioned, it is still considered for gossip via
“unreachable” nodes, which results into following exceptions:
{code}
INFO [node4_Messaging-EventLoop-3-3] node4 2020-09-29 16:37:37,527
NoSpamLogger.java:91 -
/127.0.0.4:7012->/127.0.0.1:7012-URGENT_MESSAGES-[no-channel] failed to connect
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused:
/127.0.0.1:7012
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at
io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
at
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
{code}
Trace of the method that attempts to establish connection:
{code}
org.apache.cassandra.net.MessagingService.getOutbound(MessagingService.java:492)
at
org.apache.cassandra.net.MessagingService.doSend(MessagingService.java:335)
at
org.apache.cassandra.net.OutboundSink$Filtered.accept(OutboundSink.java:55)
at org.apache.cassandra.net.OutboundSink.accept(OutboundSink.java:70)
at
org.apache.cassandra.net.MessagingService.send(MessagingService.java:327)
at
org.apache.cassandra.net.MessagingService.send(MessagingService.java:314)
at org.apache.cassandra.gms.Gossiper.sendGossip(Gossiper.java:813)
at
org.apache.cassandra.gms.Gossiper.maybeGossipToUnreachableMember(Gossiper.java:840)
at org.apache.cassandra.gms.Gossiper.access$400(Gossiper.java:86)
{code}
LEFT and other nodes that are considered dead should not be picked for gossip
with unreachable nodes.
was:
After node is decommissioned, it is still considered for gossip via
“unreachable” nodes, which results into following exceptions:
INFO [node4_Messaging-EventLoop-3-3] node4 2020-09-29 16:37:37,527
NoSpamLogger.java:91 -
/127.0.0.4:7012->/127.0.0.1:7012-URGENT_MESSAGES-[no-channel] failed to connect
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused:
/127.0.0.1:7012
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at
io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
at
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Trace of the method that attempts to establish connection:
org.apache.cassandra.net.MessagingService.getOutbound(MessagingService.java:492)
at
org.apache.cassandra.net.MessagingService.doSend(MessagingService.java:335)
at
org.apache.cassandra.net.OutboundSink$Filtered.accept(OutboundSink.java:55)
at org.apache.cassandra.net.OutboundSink.accept(OutboundSink.java:70)
at
org.apache.cassandra.net.MessagingService.send(MessagingService.java:327)
at
org.apache.cassandra.net.MessagingService.send(MessagingService.java:314)
at org.apache.cassandra.gms.Gossiper.sendGossip(Gossiper.java:813)
at
org.apache.cassandra.gms.Gossiper.maybeGossipToUnreachableMember(Gossiper.java:840)
at org.apache.cassandra.gms.Gossiper.access$400(Gossiper.java:86)
LEFT and other nodes that are considered dead should not be picked for gossip
with unreachable nodes.
> Decomissioned nodes are picked for gossip when unreachable nodes are
> considered for gossiping
> ----------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-16156
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16156
> Project: Cassandra
> Issue Type: Bug
> Reporter: Alex Petrov
> Assignee: Alex Petrov
> Priority: Normal
>
> After node is decommissioned, it is still considered for gossip via
> “unreachable” nodes, which results into following exceptions:
>
> {code}
> INFO [node4_Messaging-EventLoop-3-3] node4 2020-09-29 16:37:37,527
> NoSpamLogger.java:91 -
> /127.0.0.4:7012->/127.0.0.1:7012-URGENT_MESSAGES-[no-channel] failed to
> connect
> io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection
> refused: /127.0.0.1:7012
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at
> io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
> at
> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> Trace of the method that attempts to establish connection:
> {code}
> org.apache.cassandra.net.MessagingService.getOutbound(MessagingService.java:492)
> at
> org.apache.cassandra.net.MessagingService.doSend(MessagingService.java:335)
> at
> org.apache.cassandra.net.OutboundSink$Filtered.accept(OutboundSink.java:55)
> at org.apache.cassandra.net.OutboundSink.accept(OutboundSink.java:70)
> at
> org.apache.cassandra.net.MessagingService.send(MessagingService.java:327)
> at
> org.apache.cassandra.net.MessagingService.send(MessagingService.java:314)
> at org.apache.cassandra.gms.Gossiper.sendGossip(Gossiper.java:813)
> at
> org.apache.cassandra.gms.Gossiper.maybeGossipToUnreachableMember(Gossiper.java:840)
> at org.apache.cassandra.gms.Gossiper.access$400(Gossiper.java:86)
> {code}
> LEFT and other nodes that are considered dead should not be picked for gossip
> with unreachable nodes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
