[
https://issues.apache.org/jira/browse/CASSANDRA-10926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15068796#comment-15068796
]
Kai Wang commented on CASSANDRA-10926:
--------------------------------------
Joel,
Thanks. This is a long story.
First of all, nodetool status from a good node:
{noformat}
-- Address Load Tokens Owns Host ID
Rack
UN 192.168.0.10 225.89 GB 256 ?
2014f9be-423a-49cd-b8ef-252983c8f615 rack1
UN 192.168.0.11 242.84 GB 256 ?
b822b46f-9981-4105-8308-c3feebd16e32 rack1
UJ 192.168.0.12 29.22 MB 256 ?
ca519cf7-813f-4503-863a-7b5fb93cc3a8 rack1
{noformat}
I guess that pretty much explained the AssertionError.
I had a 3-node cluster. I rebuilt one of them from scratch. When I copied C*
config to it and tried to bootstrap, I got an error saying I can't add because
this node was already in the cluster. I needed to replace the node instead. So
I did it by adding -Dcassandra.replace_address. Then I found the bootstrap
didn't start (can't remember what exactly happened). The other two nodes have
250GB data but the new node only has 12MB.
Then I tried to delete the data and commit log folders from the new node,
remove the -Dcassandra.replace_address. I wanted to add the new node as a brand
new one.
But then I got streaming failure error. From the new node's system.log, I saw
this:
{noformat}
ERROR [STREAM-IN-/192.168.0.10] 2015-12-22 16:46:57,107 StreamSession.java:524
- [Stream #6d9603f0-a8f5-11e5-8eb5-67d8099a8b91] Streaming error occurred
java.nio.channels.ClosedChannelException: null
at sun.nio.ch.SocketChannelImpl.ensureReadOpen(Unknown Source)
~[na:1.8.0_65]
at sun.nio.ch.SocketChannelImpl.read(Unknown Source) ~[na:1.8.0_65]
at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:53)
~[apache-cassandra-2.2.4.jar:2.2.4]
...
{noformat}
and
{noformat}
ERROR [main] 2015-12-22 16:46:57,144 StorageService.java:1245 - Error while
waiting on bootstrap to complete. Bootstrap will have to be restarted.
java.util.concurrent.ExecutionException:
org.apache.cassandra.streaming.StreamException: Stream failed
...
at
org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1240)
[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:920)
[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:705)
[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:581)
[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:315)
[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:529)
[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:638)
[apache-cassandra-2.2.4.jar:2.2.4]
{noformat}
>From the node I streamed from, I saw this:
{noformat}
org.apache.cassandra.io.FSReadError: java.io.IOException: An existing
connection was forcibly closed by the remote host
at
org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:144)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:79)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:76)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.applyToChannel(BufferedDataOutputStreamPlus.java:297)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:75)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:90)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:48)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:363)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:335)
~[apache-cassandra-2.2.4.jar:2.2.4]
{noformat}
I was pretty much stuck at that point. All I wanted to do is just bootstrap the
new node as if it was a brand new one. That's why I tried to remove it using
removenode and added it back to see if it makes any difference.
> AssertionError when removing node
> ---------------------------------
>
> Key: CASSANDRA-10926
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10926
> Project: Cassandra
> Issue Type: Bug
> Environment: CentOS 7 x64, Java 1.8.0.65
> Reporter: Kai Wang
> Assignee: Joel Knighton
>
> {noformat}
> [root@centos-2 ~]# nodetool -u xxx -pw xxxx removenode
> 97a9042d-ea14-49a9-9f22-2dab4b762673
> error: null
> -- StackTrace --
> java.lang.AssertionError
> at
> org.apache.cassandra.locator.TokenMetadata.getTokens(TokenMetadata.java:474)
> at
> org.apache.cassandra.service.StorageService.removeNode(StorageService.java:3793)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at sun.reflect.misc.Trampoline.invoke(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at sun.reflect.misc.MethodUtil.invoke(Unknown Source)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown
> Source)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown
> Source)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
> at
> com.sun.jmx.remote.security.MBeanServerAccessController.invoke(Unknown Source)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown
> Source)
> at javax.management.remote.rmi.RMIConnectionImpl.access$300(Unknown
> Source)
> at
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown
> Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown
> Source)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown
> Source)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
> at sun.rmi.transport.Transport$1.run(Unknown Source)
> at sun.rmi.transport.Transport$1.run(Unknown Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Unknown Source)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown
> Source)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$81(Unknown
> Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
