[
https://issues.apache.org/jira/browse/CASSANDRA-6565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13868693#comment-13868693
]
Shao-Chuan Wang edited comment on CASSANDRA-6565 at 1/11/14 7:11 AM:
---------------------------------------------------------------------
Noted that we have replication factor 3 in both data centers. So ideally, we
have 6 replicas.
Also, we saw from *nodetool status*, some of nodes appear to be down but they
are not. Could it because the replicas owners are too busy or network is too
slow such that it only saw one available replica when this brand new node was
trying to join. Noted that by retrying a lot of times of adding this node node,
it finally joined the ring. But the joining new nodes have become a very
painful process for us.
was (Author: shaochuan):
Noted that we have replication factor 3 in both data centers. So ideally, we
have 6 replicas.
Also, we saw from nodetool status, some of nodes appear to be down but they are
not. Could it because the replicas owners are too busy or network is too slow
such that it only saw one available replica when this brand new node was trying
to join. Noted that by retrying a lot of times of adding this node node, it
finally joined the ring. But the joining new nodes have become a very painful
process for us.
> New node refuses to join the ring.
> ----------------------------------
>
> Key: CASSANDRA-6565
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6565
> Project: Cassandra
> Issue Type: Bug
> Reporter: Shao-Chuan Wang
>
> We have 30 nodes in one DC, 25 nodes in another. We are running 2.0.1.
> Two nodes are joining the ring, but one of them failed
> ARN [STREAM-IN-/10.4.197.53] 2014-01-09 19:41:40,418 StreamResultFuture.java
> (line 209) [Stream #e515d6e0-795d-11e3-b74a-b72892248056] Stream failed
> ERROR [main] 2014-01-09 19:41:40,418 CassandraDaemon.java (line 459)
> Exception encountered during startup
> java.lang.RuntimeException: Error during boostrap: Stream failed
> at
> org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:86)
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:901)
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:670)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:529)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:428)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:343)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210)
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:185)
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:321)
> at
> org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:501)
> at org.apache.cassandra.streaming.StreamSession.messageReceived(Stre
> amSession.java:376) at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:293)
> at java.lang.Thread.run(Thread.java:744)
> ERROR [StorageServiceShutdownHook] 2014-01-09 19:41:40,428
> CassandraDaemon.java (line 185) Exception in thread
> Thread[StorageServiceShutdownHook,5,main]
> java.lang.NullPointerException
> at
> org.apache.cassandra.service.StorageService.stopRPCServer(StorageService.java:312)
> at
> org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:361)
> at
> org.apache.cassandra.service.StorageService.access$000(StorageService.java:96)
> at
> org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:494)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at java.lang.Thread.run(Thread.java:744)
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
