Victor Trac created CASSANDRA-11982:
---------------------------------------
Summary: Cassandra 3.5 cluster join fails
Key: CASSANDRA-11982
URL: https://issues.apache.org/jira/browse/CASSANDRA-11982
Project: Cassandra
Issue Type: Bug
Environment: # java -version
openjdk version "1.8.0_91"
OpenJDK Runtime Environment (build 1.8.0_91-b14)
OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)
# uname -a
Linux cassandra-dfs-10-10-160-19 4.4.11-23.53.amzn1.x86_64 #1 SMP Wed Jun 1
22:22:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
# rpm -qa datastax-ddc
datastax-ddc-3.5.0-1.noarch
Reporter: Victor Trac
In an effort to work around a failing upgradesstables/scrub
(https://issues.apache.org/jira/browse/CASSANDRA-11981), I force-removed a node
from the cluster, deleted its data directory, and tried to join a node to the
cluster anew. After a few hours, the bootstrap thread failed:
cassandra.log
{code}
INFO 18:56:24 Redistributing index summaries
INFO 19:56:23 Saved KeyCache (17 items) in 386 ms
INFO 19:56:24 Redistributing index summaries
ERROR 20:40:27 [Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a] Remote peer
10.10.160.18 failed stream session.
INFO 20:40:27 [Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a] Session with
/10.10.160.18 is complete
WARN 20:40:27 [Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a] Stream failed
ERROR 20:40:27 Error while waiting on bootstrap to complete. Bootstrap will
have to be restarted.
java.util.concurrent.ExecutionException:
org.apache.cassandra.streaming.StreamException: Stream failed
at
com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
~[guava-18.0.jar:na]
at
org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1387)
[apache-cassandra-3.5.0.jar:3.5.0]
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:974)
[apache-cassandra-3.5.0.jar:3.5.0]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:748)
[apache-cassandra-3.5.0.jar:3.5.0]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:613)
[apache-cassandra-3.5.0.jar:3.5.0]
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:349)
[apache-cassandra-3.5.0.jar:3.5.0]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551)
[apache-cassandra-3.5.0.jar:3.5.0]
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:680)
[apache-cassandra-3.5.0.jar:3.5.0]
Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
at
org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
~[apache-cassandra-3.5.0.jar:3.5.0]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
~[guava-18.0.jar:na]
at
org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:213)
~[apache-cassandra-3.5.0.jar:3.5.0]
at
org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:189)
~[apache-cassandra-3.5.0.jar:3.5.0]
at
org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:429)
~[apache-cassandra-3.5.0.jar:3.5.0]
at
org.apache.cassandra.streaming.StreamSession.sessionFailed(StreamSession.java:636)
~[apache-cassandra-3.5.0.jar:3.5.0]
at
org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:489)
~[apache-cassandra-3.5.0.jar:3.5.0]
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:274)
~[apache-cassandra-3.5.0.jar:3.5.0]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_91]
Jun 08, 2016 8:40:27 PM com.google.common.util.concurrent.ExecutionList
executeListener
SEVERE: RuntimeException while executing runnable
com.google.common.util.concurrent.Futures$6@3592d1b5 with executor INSTANCE
java.lang.NullPointerException
at
org.apache.cassandra.service.StorageService$2.onFailure(StorageService.java:1382)
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
at
com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
at
com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
at
com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
at
com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
at
org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:213)
at
org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:189)
at
org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:429)
at
org.apache.cassandra.streaming.StreamSession.sessionFailed(StreamSession.java:636)
at
org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:489)
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:274)
at java.lang.Thread.run(Thread.java:745)
WARN 20:40:27 Some data streaming failed. Use nodetool to check bootstrap
state and resume. For more, see `nodetool help bootstrap`. IN_PROGRESS
INFO 20:40:27 Waiting for gossip to settle before accepting client requests...
INFO 20:40:35 No gossip backlog; proceeding
{code}
debug.log
{code}
DEBUG [GossipStage:1] 2016-06-08 20:40:27,370 FailureDetector.java:456 -
Ignoring interval time of 2001208211 for /10.240.0.4
DEBUG [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,701
ConnectionHandler.java:269 - [Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a]
Received Session Failed
ERROR [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,701 StreamSession.java:635
- [Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a] Remote peer 10.10.160.18
failed stream session.
DEBUG [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,708
ConnectionHandler.java:110 - [Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a]
Closing stream connection handler on /10.10.160.18
INFO [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,708
StreamResultFuture.java:185 - [Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a]
Session with /10.10.160.18 is complete
DEBUG [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,709
StreamCoordinator.java:146 - Finished connecting all sessions
WARN [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,710
StreamResultFuture.java:212 - [Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a]
Stream failed
ERROR [main] 2016-06-08 20:40:27,711 StorageService.java:1392 - Error while
waiting on bootstrap to complete. Bootstrap will have to be restarted.
java.util.concurrent.ExecutionException:
org.apache.cassandra.streaming.StreamException: Stream failed
at
com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
~[guava-18.0.jar:na]
at
org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1387)
[apache-cassandra-3.5.0.jar:3.5.0]
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:974)
[apache-cassandra-3.5.0.jar:3.5.0]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:748)
[apache-cassandra-3.5.0.jar:3.5.0]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:613)
[apache-cassandra-3.5.0.jar:3.5.0]
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:349)
[apache-cassandra-3.5.0.jar:3.5.0]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551)
[apache-cassandra-3.5.0.jar:3.5.0]
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:680)
[apache-cassandra-3.5.0.jar:3.5.0]
Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
at
org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
~[apache-cassandra-3.5.0.jar:3.5.0]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
~[guava-18.0.jar:na]
at
org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:213)
~[apache-cassandra-3.5.0.jar:3.5.0]
at
org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:189)
~[apache-cassandra-3.5.0.jar:3.5.0]
at
org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:429)
~[apache-cassandra-3.5.0.jar:3.5.0]
at
org.apache.cassandra.streaming.StreamSession.sessionFailed(StreamSession.java:636)
~[apache-cassandra-3.5.0.jar:3.5.0]
at
org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:489)
~[apache-cassandra-3.5.0.jar:3.5.0]
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:274)
~[apache-cassandra-3.5.0.jar:3.5.0]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_91]
WARN [main] 2016-06-08 20:40:27,777 StorageService.java:1029 - Some data
streaming failed. Use nodetool to check bootstrap state and resume. For more,
see `nodetool help bootstrap`. IN_PROGRESS
INFO [main] 2016-06-08 20:40:27,779 CassandraDaemon.java:639 - Waiting for
gossip to settle before accepting client requests...
DEBUG [GossipStage:1] 2016-06-08 20:40:31,371 FailureDetector.java:456 -
Ignoring interval time of 2256483778 for /10.10.128.18
DEBUG [GossipStage:1] 2016-06-08 20:40:31,371 FailureDetector.java:456 -
Ignoring interval time of 2001387252 for /10.10.160.18
{code}
system.log
{code}
INFO [IndexSummaryManager:1] 2016-06-08 19:56:24,495
IndexSummaryRedistribution.java:74 - Redistributing index summaries
ERROR [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,701 StreamSession.java:635
- [Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a] Remote peer 10.10.160.18
failed stream session.
INFO [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,708
StreamResultFuture.java:185 - [Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a]
Session with /10.10.160.18 is complete
WARN [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,710
StreamResultFuture.java:212 - [Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a]
Stream failed
ERROR [main] 2016-06-08 20:40:27,711 StorageService.java:1392 - Error while
waiting on bootstrap to complete. Bootstrap will have to be restarted.
java.util.concurrent.ExecutionException:
org.apache.cassandra.streaming.StreamException: Stream failed
at
com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
~[guava-18.0.jar:na]
at
org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1387)
[apache-cassandra-3.5.0.jar:3.5.0]
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:974)
[apache-cassandra-3.5.0.jar:3.5.0]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:748)
[apache-cassandra-3.5.0.jar:3.5.0]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:613)
[apache-cassandra-3.5.0.jar:3.5.0]
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:349)
[apache-cassandra-3.5.0.jar:3.5.0]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551)
[apache-cassandra-3.5.0.jar:3.5.0]
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:680)
[apache-cassandra-3.5.0.jar:3.5.0]
Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
at
org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
~[apache-cassandra-3.5.0.jar:3.5.0]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
~[guava-18.0.jar:na]
at
org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:213)
~[apache-cassandra-3.5.0.jar:3.5.0]
at
org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:189)
~[apache-cassandra-3.5.0.jar:3.5.0]
at
org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:429)
~[apache-cassandra-3.5.0.jar:3.5.0]
at
org.apache.cassandra.streaming.StreamSession.sessionFailed(StreamSession.java:636)
~[apache-cassandra-3.5.0.jar:3.5.0]
at
org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:489)
~[apache-cassandra-3.5.0.jar:3.5.0]
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:274)
~[apache-cassandra-3.5.0.jar:3.5.0]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_91]
WARN [main] 2016-06-08 20:40:27,777 StorageService.java:1029 - Some data
streaming failed. Use nodetool to check bootstrap state and resume. For more,
see `nodetool help bootstrap`. IN_PROGRESS
INFO [main] 2016-06-08 20:40:27,779 CassandraDaemon.java:639 - Waiting for
gossip to settle before accepting client requests...
INFO [main] 2016-06-08 20:40:35,780 CassandraDaemon.java:670 - No gossip
backlog; proceeding
INFO [main] 2016-06-08 20:40:35,876 NativeTransportService.java:70 - Netty
using native Epoll event loop
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
