Hello Paulo,
Thx for your fast replay.
You are right about that node, I did not see it that fast. In this node we
have errors of .SocketTimeoutException: null
ERROR [STREAM-IN-/192.168.0.146] 2016-08-04 19:10:59,456
StreamSession.java:505 - [Stream #06c02460-5a5e-11e6-8e9a-a5bf51981ad8]
Streaming error occurred
java.net.SocketTimeoutException: null
at
sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:211)
~[na:1.8.0_60]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
~[na:1.8.0_60]
at
java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
~[na:1.8.0_60]
at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:51)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:257)
~[apache-cassandra-2.1.14.jar:2.1.14]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
We have this error with at least 5 more nodes in the same log. But the
error doesn't say that much
Saludos
Jean Carlo
"The best way to predict the future is to invent it" Alan Kay
On Fri, Aug 5, 2016 at 5:16 PM, Paulo Motta <pauloricard...@gmail.com>
wrote:
> you need to check 192.168.0.36/10.234.86.36 for streaming ERRORS
>
> 2016-08-05 12:08 GMT-03:00 Jean Carlo <jean.jeancar...@gmail.com>:
>
>> Hi Paulo
>>
>> I found the lines, we got an exception "Outgoing stream handler has been
>> closed" these are
>>
>> ERROR [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,772
>> StreamSession.java:621 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40]
>> Remote peer 192.168.0.36 failed stream session.
>> INFO [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,772
>> StreamResultFuture.java:180 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40]
>> Session with /192.168.0.36 is complete
>> WARN [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,773
>> StreamResultFuture.java:207 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40]
>> Stream failed
>> ERROR [StreamReceiveTask:107995] 2016-08-04 16:55:53,782
>> StreamReceiveTask.java:183 - Error applying streamed data:
>> java.lang.RuntimeException: Outgoing stream handler has been closed
>> at
>> org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:138)
>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>> at
>> org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:697)
>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>> at
>> org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:653)
>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>> at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletio
>> nRunnable.run(StreamReceiveTask.java:179) ~[apache-cassandra-2.1.14.jar:
>> 2.1.14]
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> [na:1.8.0_60]
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> [na:1.8.0_60]
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> [na:1.8.0_60]
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> [na:1.8.0_60]
>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
>>
>> ERROR [StreamReceiveTask:107995] 2016-08-04 16:55:53,782
>> StreamSession.java:505 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40]
>> Streaming error occurred
>> java.lang.RuntimeException: Outgoing stream handler has been closed
>> at
>> org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:138)
>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>> at
>> org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:697)
>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>> at
>> org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:653)
>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>> at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletio
>> nRunnable.run(StreamReceiveTask.java:179) ~[apache-cassandra-2.1.14.jar:
>> 2.1.14]
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> [na:1.8.0_60]
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> [na:1.8.0_60]
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> [na:1.8.0_60]
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> [na:1.8.0_60]
>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
>>
>> I al looking through the changes files to see if it is a bug fixed in the
>> 2.1.15
>>
>>
>>
>> Saludos
>>
>> Jean Carlo
>>
>> "The best way to predict the future is to invent it" Alan Kay
>>
>> On Fri, Aug 5, 2016 at 4:24 PM, Paulo Motta <pauloricard...@gmail.com>
>> wrote:
>>
>>> It seems you have a streaming error, look for ERROR statement in the
>>> streaming classes before that which may give you a more specific root
>>> cause. In any case, I'd suggest you to upgrade to 2.1.15 as there were a
>>> couple of streaming fixes on this version that might help.
>>>
>>> 2016-08-05 11:15 GMT-03:00 Jean Carlo <jean.jeancar...@gmail.com>:
>>>
>>>>
>>>> Hi guys, Doing a repair I got this error for 2 tokenranges.
>>>>
>>>> ERROR [Thread-2499244] 2016-08-04 20:05:24,288 StorageService.java:3068
>>>> - Repair session 41e4bab0-5a63-11e6-9993-e11d93fd5b40 for range
>>>> (487410372471205090,492009442088088379] failed with error
>>>> org.apache.cassandra.exceptions.RepairExcep
>>>> tion: [repair #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv,
>>>> (487410372471205090,492009442088088379]] Sync failed between /
>>>> 192.168.0.144 and /192.168.0.37
>>>> java.util.concurrent.ExecutionException: java.lang.RuntimeException:
>>>> org.apache.cassandra.exceptions.RepairException: [repair
>>>> #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv,
>>>> (487410372471205090,492009442088088379]] Sync failed between /
>>>> 192.168.0.144 and /192.168.0.37
>>>> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>>>> [na:1.8.0_60]
>>>> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>>>> [na:1.8.0_60]
>>>> at
>>>> org.apache.cassandra.service.StorageService$4.runMayThrow(StorageService.java:3059)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>> at
>>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>>>> [apache-cassandra-2.1.14.jar:2.1.14]
>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>> [na:1.8.0_60]
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>> [na:1.8.0_60]
>>>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
>>>> Caused by: java.lang.RuntimeException:
>>>> org.apache.cassandra.exceptions.RepairException:
>>>> [repair #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv,
>>>> (487410372471205090,492009442088088379]] Sync failed between /
>>>> 10.234.72.144 and
>>>> /10.234.86.37
>>>> at com.google.common.base.Throwables.propagate(Throwables.java:160)
>>>> ~[guava-16.0.jar:na]
>>>> at
>>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
>>>> [apache-cassandra-2.1.14.jar:2.1.14]
>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>> [na:1.8.0_60]
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>> [na:1.8.0_60]
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> ~[na:1.8.0_60]
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> ~[na:1.8.0_60]
>>>> ... 1 common frames omitted
>>>> Caused by: org.apache.cassandra.exceptions.RepairException: [repair
>>>> #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv,
>>>> (487410372471205090,492009442088088379]] Sync failed between /
>>>> 192.168.0.144 and /192.168.0.37
>>>> at
>>>> org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:223)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>> at org.apache.cassandra.service.ActiveRepairService.handleMessa
>>>> ge(ActiveRepairService.java:422) ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>> at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(
>>>> RepairMessageVerbHandler.java:134) ~[apache-cassandra-2.1.14.jar:
>>>> 2.1.14]
>>>> at
>>>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>> ... 3 common frames omitted
>>>>
>>>> Is it a network problem or we are having a bug ? There is nothing else
>>>> in the log of cassandra more than
>>>> Sync failed between.
>>>>
>>>> We are having cassandra 2.1.14
>>>>
>>>> Best regards
>>>>
>>>> Jean Carlo
>>>>
>>>> "The best way to predict the future is to invent it" Alan Kay
>>>>
>>>
>>>
>>
>
