[
https://issues.apache.org/jira/browse/CASSANDRA-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14010403#comment-14010403
]
Brandon Williams commented on CASSANDRA-7307:
---------------------------------------------
I was just looking at this.. here are my results:
{noformat}
INFO 21:58:45,820 Node /10.208.8.63 state jump to normal
INFO 22:07:58,880 InetAddress /10.208.8.63 is now DOWN
{noformat}
I'm 100% sure the cause of this is CASSANDRA-6385. Luckily, after
CASSANDRA-4375 you can override this. Unluckily, you have to apply a two liner
from CASSANDRA-6751 because of stupid static initialization. We should,
perhaps, revisit our motivation for CASSANDRA-6385 and make the default much
lower (for the common case) and let people who are using 1000 node clusters
with vnodes perform the override instead.
> New nodes mark dead nodes as up for 10 minutes
> ----------------------------------------------
>
> Key: CASSANDRA-7307
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7307
> Project: Cassandra
> Issue Type: Bug
> Reporter: Richard Low
> Assignee: Brandon Williams
> Fix For: 1.2.17
>
>
> When doing a node replacement when other nodes are down we see the down nodes
> marked as up for about 10 minutes. This means requests are routed to the dead
> nodes causing timeouts. It also means replacing a node when multiple nodes
> from a replica set is extremely difficult - the node usually tries to stream
> from a dead node and the replacement fails.
> This isn't limited to host replacement. I did a simple test:
> 1. Create a 2 node cluster
> 2. Kill node 2
> 3. Start a 3rd node with a unique token (I used auto_bootstrap=false but I
> don't think this is significant)
> The 3rd node lists node 2 (127.0.0.2) as up for almost 10 minutes:
> {code}
> INFO [main] 2014-05-27 14:28:24,753 CassandraDaemon.java (line 119) Logging
> initialized
> INFO [GossipStage:1] 2014-05-27 14:28:31,492 Gossiper.java (line 843) Node
> /127.0.0.2 is now part of the cluster
> INFO [GossipStage:1] 2014-05-27 14:28:31,495 Gossiper.java (line 809)
> InetAddress /127.0.0.2 is now UP
> INFO [GossipTasks:1] 2014-05-27 14:37:44,526 Gossiper.java (line 823)
> InetAddress /127.0.0.2 is now DOWN
> {code}
> I reproduced on 1.2.15 and 1.2.16.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
