Check ifconfig for dripped tpc messages. Let's rule out your network. all the best,
Sebastián On May 27, 2016 10:45 AM, "George Sigletos" <sigle...@textkernel.nl> wrote: > Hello, > > No there is no version mix. The first stack traces were indeed from > 2.1.13. Then I upgraded all nodes to 2.1.14. Still getting the same errors > > > On Fri, May 27, 2016 at 4:39 PM, Eric Evans <john.eric.ev...@gmail.com> > wrote: > >> From the various stacktraces in this thread, it's obvious you are >> mixing versions 2.1.13 and 2.1.14. Topology changes like this aren't >> supported with mixed Cassandra versions. Sometimes it will work, >> sometimes it won't (and it will definitely not work in this instance). >> >> You should either upgrade your 2.1.13 nodes to 2.1.14 first, or add >> the new nodes using 2.1.13, and upgrade after. >> >> On Fri, May 27, 2016 at 8:41 AM, George Sigletos <sigle...@textkernel.nl> >> wrote: >> >> >>>> ERROR [STREAM-IN-/192.168.1.141] 2016-05-26 09:08:05,027 >> >>>> StreamSession.java:505 - [Stream >> #74c57bc0-231a-11e6-a698-1b05ac77baf9] >> >>>> Streaming error occurred >> >>>> java.lang.RuntimeException: Outgoing stream handler has been closed >> >>>> at >> >>>> >> org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:138) >> >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >> >>>> at >> >>>> >> org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:568) >> >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >> >>>> at >> >>>> >> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:457) >> >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >> >>>> at >> >>>> >> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:263) >> >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >> >>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_79] >> >>>> >> >>>> And this is from the source node: >> >>>> >> >>>> ERROR [STREAM-OUT-/172.31.22.104] 2016-05-26 11:08:05,097 >> >>>> StreamSession.java:505 - [Stream >> #74c57bc0-231a-11e6-a698-1b05ac77baf9] >> >>>> Streaming error occurred >> >>>> java.io.IOException: Broken pipe >> >>>> at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) >> >>>> ~[na:1.7.0_79] >> >>>> at sun.nio.ch.FileChannelImpl.transferToDirectly(Unknown >> Source) >> >>>> ~[na:1.7.0_79] >> >>>> at sun.nio.ch.FileChannelImpl.transferTo(Unknown Source) >> >>>> ~[na:1.7.0_79] >> >>>> at >> >>>> >> org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:84) >> >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >> >>>> at >> >>>> >> org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:88) >> >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >> >>>> at >> >>>> >> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:49) >> >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >> >>>> at >> >>>> >> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:41) >> >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >> >>>> at >> >>>> >> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:45) >> >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >> >>>> at >> >>>> >> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:358) >> >>>> [apache-cassandra-2.1.14.jar:2.1.14] >> >>>> at >> >>>> >> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:330) >> >>>> [apache-cassandra-2.1.14.jar:2.1.14] >> >> >> >>>>>>>>>>> ERROR [STREAM-IN-/192.168.1.140] 2016-05-24 22:44:57,704 >> >>>>>>>>>>> StreamSession.java:620 - [Stream >> #2c290460-20d4-11e6-930f-1b05ac77baf9] >> >>>>>>>>>>> Remote peer 192.168.1.140 failed stream session. >> >>>>>>>>>>> ERROR [STREAM-OUT-/192.168.1.140] 2016-05-24 22:44:57,705 >> >>>>>>>>>>> StreamSession.java:505 - [Stream >> #2c290460-20d4-11e6-930f-1b05ac77baf9] >> >>>>>>>>>>> Streaming error occurred >> >>>>>>>>>>> java.io.IOException: Connection timed out >> >>>>>>>>>>> at sun.nio.ch.FileDispatcherImpl.write0(Native Method) >> >>>>>>>>>>> ~[na:1.7.0_79] >> >>>>>>>>>>> at sun.nio.ch.SocketDispatcher.write(Unknown Source) >> >>>>>>>>>>> ~[na:1.7.0_79] >> >>>>>>>>>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown >> >>>>>>>>>>> Source) ~[na:1.7.0_79] >> >>>>>>>>>>> at sun.nio.ch.IOUtil.write(Unknown Source) >> ~[na:1.7.0_79] >> >>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.write(Unknown Source) >> >>>>>>>>>>> ~[na:1.7.0_79] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> org.apache.cassandra.io.util.DataOutputStreamAndChannel.write(DataOutputStreamAndChannel.java:48) >> >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:44) >> >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:351) >> >>>>>>>>>>> [apache-cassandra-2.1.13.jar:2.1.13] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:323) >> >>>>>>>>>>> [apache-cassandra-2.1.13.jar:2.1.13] >> >>>>>>>>>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_79] >> >>>>>>>>>>> INFO [STREAM-IN-/192.168.1.140] 2016-05-24 22:44:58,625 >> >>>>>>>>>>> StreamResultFuture.java:180 - [Stream >> #2c290460-20d4-11e6-930f-1b05ac77baf9] >> >>>>>>>>>>> Session with /192.168.1.140 is complete >> >>>>>>>>>>> WARN [STREAM-IN-/192.168.1.140] 2016-05-24 22:44:58,627 >> >>>>>>>>>>> StreamResultFuture.java:207 - [Stream >> #2c290460-20d4-11e6-930f-1b05ac77baf9] >> >>>>>>>>>>> Stream failed >> >>>>>>>>>>> ERROR [RMI TCP Connection(24)-127.0.0.1] 2016-05-24 >> 22:44:58,628 >> >>>>>>>>>>> StorageService.java:1075 - Error while rebuilding node >> >>>>>>>>>>> org.apache.cassandra.streaming.StreamException: Stream failed >> >>>>>>>>>>> at >> >>>>>>>>>>> >> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) >> >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> com.google.common.util.concurrent.Futures$4.run(Futures.java:1172) >> >>>>>>>>>>> ~[guava-16.0.jar:na] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) >> >>>>>>>>>>> ~[guava-16.0.jar:na] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) >> >>>>>>>>>>> ~[guava-16.0.jar:na] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) >> >>>>>>>>>>> ~[guava-16.0.jar:na] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) >> >>>>>>>>>>> ~[guava-16.0.jar:na] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:208) >> >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:184) >> >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:415) >> >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> org.apache.cassandra.streaming.StreamSession.sessionFailed(StreamSession.java:621) >> >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:475) >> >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:256) >> >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13] >> >>>>>>>>>>> at java.lang.Thread.run(Unknown Source) ~[na:1.7.0_79] >> >>>>>>>>>>> ERROR [STREAM-OUT-/192.168.1.140] 2016-05-24 22:44:58,629 >> >>>>>>>>>>> StreamSession.java:505 - [Stream >> #2c290460-20d4-11e6-930f-1b05ac77baf9] >> >>>>>>>>>>> Streaming error occurred >> >>>>>>>>>>> java.io.IOException: Broken pipe >> >>>>>>>>>>> at sun.nio.ch.FileDispatcherImpl.write0(Native Method) >> >>>>>>>>>>> ~[na:1.7.0_79] >> >>>>>>>>>>> at sun.nio.ch.SocketDispatcher.write(Unknown Source) >> >>>>>>>>>>> ~[na:1.7.0_79] >> >>>>>>>>>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown >> >>>>>>>>>>> Source) ~[na:1.7.0_79] >> >>>>>>>>>>> at sun.nio.ch.IOUtil.write(Unknown Source) >> ~[na:1.7.0_79] >> >>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.write(Unknown Source) >> >>>>>>>>>>> ~[na:1.7.0_79] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> org.apache.cassandra.io.util.DataOutputStreamAndChannel.write(DataOutputStreamAndChannel.java:48) >> >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:44) >> >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:351) >> >>>>>>>>>>> [apache-cassandra-2.1.13.jar:2.1.13] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:331) >> >>>>>>>>>>> [apache-cassandra-2.1.13.jar:2.1.13] >> >>>>>>>>>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_79] >> >> >> >> -- >> Eric Evans >> john.eric.ev...@gmail.com >> > >
