Thank you very much Paulo On Aug 5, 2016 17:31, "Paulo Motta" <pauloricard...@gmail.com> wrote:
> https://issues.apache.org/jira/browse/CASSANDRA-11840 > > increase streaming_socket_timeout to 86400000 or upgrade to > cassandra-2.1.15. > > 2016-08-05 12:28 GMT-03:00 Jean Carlo <jean.jeancar...@gmail.com>: > >> >> Hello Paulo, >> >> Thx for your fast replay. >> >> You are right about that node, I did not see it that fast. In this node >> we have errors of .SocketTimeoutException: null >> >> >> >> ERROR [STREAM-IN-/192.168.0.146] 2016-08-04 19:10:59,456 >> StreamSession.java:505 - [Stream #06c02460-5a5e-11e6-8e9a-a5bf51981ad8] >> Streaming error occurred >> java.net.SocketTimeoutException: null >> at >> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:211) >> ~[na:1.8.0_60] >> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) >> ~[na:1.8.0_60] >> at >> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385) >> ~[na:1.8.0_60] >> at org.apache.cassandra.streaming.messages.StreamMessage. >> deserialize(StreamMessage.java:51) ~[apache-cassandra-2.1.14.jar:2.1.14] >> at org.apache.cassandra.streaming.ConnectionHandler$IncomingMes >> sageHandler.run(ConnectionHandler.java:257) >> ~[apache-cassandra-2.1.14.jar:2.1.14] >> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] >> >> We have this error with at least 5 more nodes in the same log. But the >> error doesn't say that much >> >> >> Saludos >> >> Jean Carlo >> >> "The best way to predict the future is to invent it" Alan Kay >> >> On Fri, Aug 5, 2016 at 5:16 PM, Paulo Motta <pauloricard...@gmail.com> >> wrote: >> >>> you need to check 192.168.0.36/10.234.86.36 for streaming ERRORS >>> >>> 2016-08-05 12:08 GMT-03:00 Jean Carlo <jean.jeancar...@gmail.com>: >>> >>>> Hi Paulo >>>> >>>> I found the lines, we got an exception "Outgoing stream handler has >>>> been closed" these are >>>> >>>> ERROR [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,772 >>>> StreamSession.java:621 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40] >>>> Remote peer 192.168.0.36 failed stream session. >>>> INFO [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,772 >>>> StreamResultFuture.java:180 - [Stream >>>> #c4e79260-5a46-11e6-9993-e11d93fd5b40] >>>> Session with /192.168.0.36 is complete >>>> WARN [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,773 >>>> StreamResultFuture.java:207 - [Stream >>>> #c4e79260-5a46-11e6-9993-e11d93fd5b40] >>>> Stream failed >>>> ERROR [StreamReceiveTask:107995] 2016-08-04 16:55:53,782 >>>> StreamReceiveTask.java:183 - Error applying streamed data: >>>> java.lang.RuntimeException: Outgoing stream handler has been closed >>>> at >>>> org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:138) >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>>> at >>>> org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:697) >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>>> at >>>> org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:653) >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>>> at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletio >>>> nRunnable.run(StreamReceiveTask.java:179) >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>>> at >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>> [na:1.8.0_60] >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>> [na:1.8.0_60] >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>> [na:1.8.0_60] >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>> [na:1.8.0_60] >>>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] >>>> >>>> ERROR [StreamReceiveTask:107995] 2016-08-04 16:55:53,782 >>>> StreamSession.java:505 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40] >>>> Streaming error occurred >>>> java.lang.RuntimeException: Outgoing stream handler has been closed >>>> at >>>> org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:138) >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>>> at >>>> org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:697) >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>>> at >>>> org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:653) >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>>> at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletio >>>> nRunnable.run(StreamReceiveTask.java:179) >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>>> at >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>> [na:1.8.0_60] >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>> [na:1.8.0_60] >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>> [na:1.8.0_60] >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>> [na:1.8.0_60] >>>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] >>>> >>>> I al looking through the changes files to see if it is a bug fixed in >>>> the 2.1.15 >>>> >>>> >>>> >>>> Saludos >>>> >>>> Jean Carlo >>>> >>>> "The best way to predict the future is to invent it" Alan Kay >>>> >>>> On Fri, Aug 5, 2016 at 4:24 PM, Paulo Motta <pauloricard...@gmail.com> >>>> wrote: >>>> >>>>> It seems you have a streaming error, look for ERROR statement in the >>>>> streaming classes before that which may give you a more specific root >>>>> cause. In any case, I'd suggest you to upgrade to 2.1.15 as there were a >>>>> couple of streaming fixes on this version that might help. >>>>> >>>>> 2016-08-05 11:15 GMT-03:00 Jean Carlo <jean.jeancar...@gmail.com>: >>>>> >>>>>> >>>>>> Hi guys, Doing a repair I got this error for 2 tokenranges. >>>>>> >>>>>> ERROR [Thread-2499244] 2016-08-04 20:05:24,288 >>>>>> StorageService.java:3068 - Repair session >>>>>> 41e4bab0-5a63-11e6-9993-e11d93fd5b40 >>>>>> for range (487410372471205090,492009442088088379] failed with error >>>>>> org.apache.cassandra.exceptions.RepairExcep >>>>>> tion: [repair #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv, >>>>>> (487410372471205090,492009442088088379]] Sync failed between / >>>>>> 192.168.0.144 and /192.168.0.37 >>>>>> java.util.concurrent.ExecutionException: java.lang.RuntimeException: >>>>>> org.apache.cassandra.exceptions.RepairException: [repair >>>>>> #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv, >>>>>> (487410372471205090,492009442088088379]] Sync failed between / >>>>>> 192.168.0.144 and /192.168.0.37 >>>>>> at java.util.concurrent.FutureTask.report(FutureTask.java:122) >>>>>> [na:1.8.0_60] >>>>>> at java.util.concurrent.FutureTask.get(FutureTask.java:192) >>>>>> [na:1.8.0_60] >>>>>> at org.apache.cassandra.service.S >>>>>> torageService$4.runMayThrow(StorageService.java:3059) >>>>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>>>>> at org.apache.cassandra.utils.Wra >>>>>> ppedRunnable.run(WrappedRunnable.java:28) >>>>>> [apache-cassandra-2.1.14.jar:2.1.14] >>>>>> at java.util.concurrent.Executors >>>>>> $RunnableAdapter.call(Executors.java:511) [na:1.8.0_60] >>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>>> [na:1.8.0_60] >>>>>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] >>>>>> Caused by: java.lang.RuntimeException: >>>>>> org.apache.cassandra.exceptions.RepairException: >>>>>> [repair #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv, >>>>>> (487410372471205090,492009442088088379]] Sync failed between / >>>>>> 10.234.72.144 and >>>>>> /10.234.86.37 >>>>>> at com.google.common.base.Throwab >>>>>> les.propagate(Throwables.java:160) ~[guava-16.0.jar:na] >>>>>> at org.apache.cassandra.utils.Wra >>>>>> ppedRunnable.run(WrappedRunnable.java:32) >>>>>> [apache-cassandra-2.1.14.jar:2.1.14] >>>>>> at java.util.concurrent.Executors >>>>>> $RunnableAdapter.call(Executors.java:511) [na:1.8.0_60] >>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>>> [na:1.8.0_60] >>>>>> at java.util.concurrent.ThreadPoo >>>>>> lExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_60] >>>>>> at java.util.concurrent.ThreadPoo >>>>>> lExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_60] >>>>>> ... 1 common frames omitted >>>>>> Caused by: org.apache.cassandra.exceptions.RepairException: [repair >>>>>> #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv, >>>>>> (487410372471205090,492009442088088379]] Sync failed between / >>>>>> 192.168.0.144 and /192.168.0.37 >>>>>> at org.apache.cassandra.repair.Re >>>>>> pairSession.syncComplete(RepairSession.java:223) >>>>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>>>>> at org.apache.cassandra.service.A >>>>>> ctiveRepairService.handleMessage(ActiveRepairService.java:422) >>>>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>>>>> at org.apache.cassandra.repair.Re >>>>>> pairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:134) >>>>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>>>>> at org.apache.cassandra.net.Messa >>>>>> geDeliveryTask.run(MessageDeliveryTask.java:64) >>>>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>>>>> ... 3 common frames omitted >>>>>> >>>>>> Is it a network problem or we are having a bug ? There is nothing >>>>>> else in the log of cassandra more than >>>>>> Sync failed between. >>>>>> >>>>>> We are having cassandra 2.1.14 >>>>>> >>>>>> Best regards >>>>>> >>>>>> Jean Carlo >>>>>> >>>>>> "The best way to predict the future is to invent it" Alan Kay >>>>>> >>>>> >>>>> >>>> >>> >> >
