Hi developers, Recently, I was removing one node from the cluster without downtime.
Cluster size :: 3 node [test cluster with nodeA, nodeB, nodeC] *Procedure:-*1 Change RF to 1 2 Check all the nodes are up 3 Run repair on all the nodes 4 Run decommission on node nodeC 5 After decommission 6 Check node status(status should be *UN*) 7 Clean up on all the nodes * repeat steps[2..7] for nodeB But we found some weird exception in the logs:- *level:WARN thread:STREAM-IN-/10.142.1.11 <http://10.142.1.11> @timestamp:2017-10-06T02:44:21.567+0000 hostname:ip-10-142-1-10.compute.internal loggerName:org.apache.cassandra.streaming.compress.CompressedStreamReader file:CompressedStreamReader.java:115 message:[Stream cc2f84d0-aa3e-11e7-a2fc-fd45c9a98801] Error while reading partition DecoratedKey(-9223372036854775808, ) from stream on ks='ks1' and table='cf1'. throwable:* *level:WARN thread:STREAM-IN-/10.142.1.11 <http://10.142.1.11> @timestamp:2017-10-06T02:44:21.568+0000 hostname:ip-10-142-1-10.compute.internal loggerName:org.apache.cassandra.streaming.StreamSession file:StreamSession.java:626 message:[Stream #cc2f84d0-aa3e-11e7-a2fc-fd45c9a98801] Retrying for following error throwable:java.lang.AssertionError: null\nat org.apache.cassandra.streaming.compress.CompressedInputStream.read(CompressedInputStream.java:106)\nat java.io.InputStream.read(InputStream.java:170)\nat java.io.InputStream.skip(InputStream.java:224)\nat org.apache.cassandra.streaming.StreamReader.drain(StreamReader.java:158)\nat org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:129)\nat org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:48)\nat org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:38)\nat org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:56)\nat org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:250)\nat java.lang.Thread.run(Thread.java:745)\n* assertion failing at:- https://github.com/apache/cassandra/blob/cassandra-2.1.13/src/java/org/apache/cassandra/streaming/compress/CompressedInputStream.java#L106 Could someone please tell me in which scenario this happens, Thanks!! Thanks in advance!! Regards Varun Barala
