[
https://issues.apache.org/jira/browse/CASSANDRA-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13895714#comment-13895714
]
Ravi Prasad commented on CASSANDRA-6622:
----------------------------------------
bq. You mean rise above it, so the node is still being convicted? Can you add
new logs? Maybe now it actually is the restart event, so trying that patch with
6658 might work.
Tried with 6658 patch and 0001-don-t-signal-restart-of-dead-states.txt applied
on cassandra-2.0.5 tag. Still see the same thing, where FD convicts the
streaming session. I'm attaching the logs(6622_logs.tgz). This should be
easily reproducible when replacing a dead node in a cluster with same ip
address. the issue is, the peer nodes could take 1-3 seconds to see the
previously down node (now replacing) to be up(to reset the PHI score of the
down node). Since, the streaming request arrives before this reset happens,
they could be convicted leading to stream close. So, i think a couple of
seconds sleep time for gossip to settle, before the bootstrap/streaming starts
is what is needed?
1.) node x.x.x.72 was dead
2.) node x.x.x.80's FD keeps notifying its listener to convict as PHI for .72 >
threshold, every minute.
3.) node x.x.x.72 is restarted with replace_address=x.x.x.72 at 18:56:27,806
4.) node x.x.x.72 : Gossip thread started at 18:56:33,308 after shadow gossip
round
5.) node x.xx.72: Starts stream request at 18:56:35,443
INFO [main] 2014-02-08 18:56:35,405 StorageService.java (line 947) JOINING:
Starting to bootstrap...
INFO [main] 2014-02-08 18:56:35,443 StreamResultFuture.java (line 82) [Stream
#bb897500-90f2-11e3-9d67-d5d417af8653] Executing streaming plan for Bootstrap
6.) node x.x.x.80 : still hasn't seen the gossip from .72 with new generation
at 18:56:35,031
TRACE [GossipTasks:1] 2014-02-08 18:56:35,031 FailureDetector.java (line 229)
PHI for /x.x.x.72 : 36700.042810594234
TRACE [GossipTasks:1] 2014-02-08 18:56:35,032 FailureDetector.java (line 233)
notifying listeners that /x.x.x.72 is down
7.) node x.x.x.80 : got the stream request at 18:56:35,450
INFO [STREAM-INIT-/x.x.x.72:47408] 2014-02-08 18:56:35,450
StreamResultFuture.java (line 116) [Stream
#bb897500-90f2-11e3-9d67-d5d417af8653] Received streaming plan for Bootstrap
8.) node x.x.x.80: at 18:56:36,090, still hasn't reset the interval times for
.72
TRACE [GossipTasks:1] 2014-02-08 18:56:36,090 FailureDetector.java (line 229)
PHI for /x.x.x.72 : 36700.87918907657
TRACE [GossipTasks:1] 2014-02-08 18:56:36,090 FailureDetector.java (line 233)
notifying listeners that /x.x.x.72 is down
9.) node x.x.x.80: closes the stream session due to convict() notification:
INFO [GossipTasks:1] 2014-02-08 18:56:36,090 StreamResultFuture.java (line
181) [Stream #bb897500-90f2-11e3-9d67-d5d417af8653] Session with /x.x.x.72 is
complete
WARN [GossipTasks:1] 2014-02-08 18:56:36,091 StreamResultFuture.java (line
210) [Stream #bb897500-90f2-11e3-9d67-d5d417af8653] Stream failed
10.) node x.x.x.80: at 18:56:36,097, Gossiper thread on x.x.x.80 clears the
interval times for .72, thereby resetting the PHI.
DEBUG [GossipStage:1] 2014-02-08 18:56:36,097 Gossiper.java (line 790) Clearing
interval times for /x.x.x.72 due to generation change
TRACE [GossipStage:1] 2014-02-08 18:56:36,097 FailureDetector.java (line 203)
reporting /x.x.x.72
11.) node x.x.x.80: PHI score for .72 at 18:56:37,094
TRACE [GossipTasks:1] 2014-02-08 18:56:37,094 FailureDetector.java (line 229)
PHI for /x.x.x.72 : 0.06483452387313912
> Streaming session failures during node replace of same address
> --------------------------------------------------------------
>
> Key: CASSANDRA-6622
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6622
> Project: Cassandra
> Issue Type: Bug
> Environment: RHEL6, cassandra-2.0.4
> Reporter: Ravi Prasad
> Assignee: Brandon Williams
> Attachments: 0001-don-t-signal-restart-of-dead-states.txt,
> 6622-2.0.txt, logs.tgz
>
>
> When using replace_address, Gossiper ApplicationState is set to hibernate,
> which is a down state. We are seeing that the peer nodes are seeing streaming
> plan request even before the Gossiper on them marks the replacing node as
> dead. As a result, streaming on peer nodes convicts the replacing node by
> closing the stream handler.
> I think, making the StorageService thread on the replacing node, sleep for
> BROADCAST_INTERVAL before bootstrapping, would avoid this scenario.
> Relevant logs from peer node (see that the Gossiper on peer node mark the
> replacing node as down, 2 secs after the streaming init request):
> {noformat}
> INFO [STREAM-INIT-/x.x.x.x:46436] 2014-01-26 20:42:24,388
> StreamResultFuture.java (line 116) [Stream
> #5c6cd940-86ca-11e3-90a0-411b913c0e88] Received streaming plan for Bootstrap
> ....
> INFO [GossipTasks:1] 2014-01-26 20:42:25,240 StreamResultFuture.java (line
> 181) [Stream #5c6cd940-86ca-11e3-90a0-411b913c0e88] Session with /x.x.x.x is
> complete
> WARN [GossipTasks:1] 2014-01-26 20:42:25,240 StreamResultFuture.java (line
> 210) [Stream #5c6cd940-86ca-11e3-90a0-411b913c0e88] Stream failed
> INFO [GossipStage:1] 2014-01-26 20:42:25,242 Gossiper.java (line 850)
> InetAddress /x.x.x.x is now DOWN
> ERROR [STREAM-IN-/x.x.x.x] 2014-01-26 20:42:25,766 StreamSession.java (line
> 410) [Stream #5c6cd940-86ca-11e3-90a0-411b913c0e88] Streaming error occurred
> java.lang.RuntimeException: Outgoing stream handler has been closed
> at
> org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:175)
> at
> org.apache.cassandra.streaming.StreamSession.prepare(StreamSession.java:436)
> at
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:358)
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:293)
> at java.lang.Thread.run(Thread.java:722)
> INFO [STREAM-IN-/x.x.x.x] 2014-01-26 20:42:25,768 StreamResultFuture.java
> (line 181) [Stream #5c6cd940-86ca-11e3-90a0-411b913c0e88] Session with
> /x.x.x.x is complete
> WARN [STREAM-IN-/x.x.x.x] 2014-01-26 20:42:25,768 StreamResultFuture.java
> (line 210) [Stream #5c6cd940-86ca-11e3-90a0-411b913c0e88] Stream failed
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
