Hi, I'm trying to add new data center - DC3 to existing c*-2.0.17 cluster with 2 data centers DC1, DC2 with replication DC1:3 , DC2:3 , DC3:3.
I'm getting following exception repeatedly on new nodes after I run 'nodetool rebuild'. *DEBUG [ScheduledTasks:1] 2016-09-27 04:24:00,416 GCInspector.java (line 118) GC for ParNew: 20 ms for 1 collections, 9837479688 used; max is 16760438784DEBUG [ScheduledTasks:1] 2016-09-27 04:24:03,417 GCInspector.java (line 118) GC for ParNew: 20 ms for 1 collections, 9871193904 used; max is 16760438784DEBUG [ScheduledTasks:1] 2016-09-27 04:24:06,418 GCInspector.java (line 118) GC for ParNew: 20 ms for 1 collections, 9950298136 used; max is 16760438784DEBUG [ScheduledTasks:1] 2016-09-27 04:24:09,419 GCInspector.java (line 118) GC for ParNew: 19 ms for 1 collections, 9941119568 used; max is 16760438784DEBUG [ScheduledTasks:1] 2016-09-27 04:24:12,421 GCInspector.java (line 118) GC for ParNew: 20 ms for 1 collections, 9864185024 used; max is 16760438784DEBUG [ScheduledTasks:1] 2016-09-27 04:24:15,422 GCInspector.java (line 118) GC for ParNew: 60 ms for 2 collections, 9730374352 used; max is 16760438784DEBUG [ScheduledTasks:1] 2016-09-27 04:24:18,423 GCInspector.java (line 118) GC for ParNew: 18 ms for 1 collections, 9775448168 used; max is 16760438784DEBUG [ScheduledTasks:1] 2016-09-27 04:24:21,424 GCInspector.java (line 118) GC for ParNew: 22 ms for 1 collections, 9850794272 used; max is 16760438784DEBUG [ScheduledTasks:1] 2016-09-27 04:24:24,425 GCInspector.java (line 118) GC for ParNew: 20 ms for 1 collections, 9729992448 used; max is 16760438784DEBUG [ScheduledTasks:1] 2016-09-27 04:24:27,426 GCInspector.java (line 118) GC for ParNew: 22 ms for 1 collections, 9699783920 used; max is 16760438784DEBUG [ScheduledTasks:1] 2016-09-27 04:24:30,427 GCInspector.java (line 118) GC for ParNew: 21 ms for 1 collections, 9696523920 used; max is 16760438784DEBUG [ScheduledTasks:1] 2016-09-27 04:24:33,429 GCInspector.java (line 118) GC for ParNew: 20 ms for 1 collections, 9560497904 used; max is 16760438784DEBUG [ScheduledTasks:1] 2016-09-27 04:24:36,430 GCInspector.java (line 118) GC for ParNew: 19 ms for 1 collections, 9568718352 used; max is 16760438784DEBUG [ScheduledTasks:1] 2016-09-27 04:24:39,431 GCInspector.java (line 118) GC for ParNew: 22 ms for 1 collections, 9496991384 used; max is 16760438784DEBUG [ScheduledTasks:1] 2016-09-27 04:24:42,432 GCInspector.java (line 118) GC for ParNew: 19 ms for 1 collections, 9486433840 used; max is 16760438784DEBUG [ScheduledTasks:1] 2016-09-27 04:24:45,434 GCInspector.java (line 118) GC for ParNew: 19 ms for 1 collections, 9442642688 used; max is 16760438784DEBUG [ScheduledTasks:1] 2016-09-27 04:24:48,435 GCInspector.java (line 118) GC for ParNew: 20 ms for 1 collections, 9548532008 used; max is 16760438784DEBUG [STREAM-IN-/xxx.xxx.98.168] 2016-09-27 04:24:49,756 ConnectionHandler.java (line 244) [Stream #5e1b7f40-8496-11e6-8847-1b88665e430d] Received File (Header (cfId: bf446a90-71c5-3552-a2e5-b1b94dbf86e3, #0, version: jb, estimated keys: 252928, transfer size: 5496759656, compressed?: true), file: /home/cassandra/data_directories/data/keyspace_name1/columnfamily_1/keyspace_name1-columnfamily_1-tmp-jb-54-Data.db)DEBUG [STREAM-OUT-/xxx.xxx.98.168] 2016-09-27 04:24:49,757 ConnectionHandler.java (line 310) [Stream #5e1b7f40-8496-11e6-8847-1b88665e430d] Sending Received (bf446a90-71c5-3552-a2e5-b1b94dbf86e3, #0)ERROR [STREAM-OUT-/xxx.xxx.98.168] 2016-09-27 04:24:49,759 StreamSession.java (line 461) [Stream #5e1b7f40-8496-11e6-8847-1b88665e430d] Streaming error occurredjava.io.IOException: Connection timed out at sun.nio.ch.FileDispatcherImpl.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) at sun.nio.ch.IOUtil.write(IOUtil.java:65) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487) at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:44) at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:339) at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:311) at java.lang.Thread.run(Thread.java:745)DEBUG [STREAM-OUT-/xxx.xxx.98.168] 2016-09-27 04:24:49,764 ConnectionHandler.java (line 104) [Stream #5e1b7f40-8496-11e6-8847-1b88665e430d] Closing stream connection handler on /xxx.xxx.98.168 INFO [STREAM-OUT-/xxx.xxx.98.168] 2016-09-27 04:24:49,764 StreamResultFuture.java (line 186) [Stream #5e1b7f40-8496-11e6-8847-1b88665e430d] Session with /xxx.xxx.98.168 is completeERROR [STREAM-OUT-/xxx.xxx.98.168] 2016-09-27 04:24:49,764 StreamSession.java (line 461) [Stream #5e1b7f40-8496-11e6-8847-1b88665e430d] Streaming error occurredjava.io.IOException: Broken pipe at sun.nio.ch.FileDispatcherImpl.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) at sun.nio.ch.IOUtil.write(IOUtil.java:65) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487) at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:44) at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:339) at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:319) at java.lang.Thread.run(Thread.java:745)DEBUG [STREAM-IN-/xxx.xxx.98.168] 2016-09-27 04:24:49,909 ConnectionHandler.java (line 244) [Stream #5e1b7f40-8496-11e6-8847-1b88665e430d] Received File (Header (cfId: 68af9ee0-96f8-3b1d-a418-e5ae844f2cc2, #3, version: jb, estimated keys: 4736, transfer size: 2306880, compressed?: true), file: /home/cassandra/data_directories/data/keyspace_name1/archiving_metadata/keyspace_name1-archiving_metadata-tmp-jb-27-Data.db)ERROR [STREAM-IN-/xxx.xxx.98.168] 2016-09-27 04:24:49,909 StreamSession.java (line 461) [Stream #5e1b7f40-8496-11e6-8847-1b88665e430d] Streaming error occurredjava.lang.RuntimeException: Outgoing stream handler has been closed at org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:126) at org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:524) at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:413) at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:245) at java.lang.Thread.run(Thread.java:745)* I checked with our network operations team , they have confirmed network is stable and no network hiccups. I have set 'streaming_socket_timeout_in_ms: 86400000' (24 hours) as suggested in datastax blog - https://support.datastax.com/hc/en-us/articles/206502913-FAQ-How-to-reduce-the-impact-of-streaming-errors-or-failures and ran 'nodetool rebuild' one node at a time but was of NO USE . Still we are getting above exception. Can someone please help me in debugging and fixing this. Thanks, techpyaasa
