[
https://issues.apache.org/jira/browse/CASSANDRA-12905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15692941#comment-15692941
]
Nir Zilka commented on CASSANDRA-12905:
---------------------------------------
Hi,
I decommissioned 4 nodes from when i opened the ticket, so total of 11 nodes
currently running.
nodes list is: 10.101.123.44 , 10.101.123.46 , 10.101.123.62 , 10.101.123.36-39
, 10.101.123.50, 10.101.123.53-55
I started yesterday evening (23th) repair and catched untill now the following
errors:
this is from the machine i ran the "nodetool repair" (10.101.123.55), 11 errors:
---- 1 ----
ERROR [RepairJobTask:23] 2016-11-24 00:14:18,271 RepairSession.java:290 -
[repair #02b2bb40-b1ba-11e6-a5e3-61594f2eacfb] Session completed with the
following error
org.apache.cassandra.exceptions.RepairException: [repair
#02b2bb40-b1ba-11e6-a5e3-61594f2eacfb on radius/radius_log_user,
[(5374555084199702397,5375907191203787693],
(5375907191203787693,5394663393980481083],
(-4608172686811846999,-4606297045741692934],
(343174083518476177,369373536835281005],
(-789474545083012335,-787318207609633720],
(-810527869609466298,-789474545083012335],
(7618946177842619265,7622080038542632561],
(8643901858335832297,8668152326416369637],
(332360446019658863,336878679529361124],
(-8627477550934015644,-8626441922324893605],
(-2577622939633507215,-2575565821746779005],
(336878679529361124,343174083518476177],
(-3519645500583421982,-3517776702806344635],
(-2583420055923926426,-2577622939633507215],
(3383191969840261164,3387991562336214241],
(5394663393980481083,5411354651959353112],
(3387991562336214241,3390236953123836921],
(-705797240279985098,-700707451972711675],
(-5709906583533929520,-5707271395346631992],
(1178486765398617360,1178624428305220059],
(-3520515159422145033,-3519645500583421982]]] Sync failed between
/10.101.123.54 and /10.101.123.50
at
org.apache.cassandra.repair.RemoteSyncTask.syncComplete(RemoteSyncTask.java:67)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:211)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:462)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:165)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_111]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_111]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
ERROR [RepairJobTask:23] 2016-11-24 00:14:18,272 RepairRunnable.java:261 -
Repair session 02b2bb40-b1ba-11e6-a5e3-61594f2eacfb for range
[(5374555084199702397,5375907191203787693],
(5375907191203787693,5394663393980481083],
(-4608172686811846999,-4606297045741692934],
(343174083518476177,369373536835281005],
(-789474545083012335,-787318207609633720],
(-810527869609466298,-789474545083012335],
(7618946177842619265,7622080038542632561],
(8643901858335832297,8668152326416369637],
(332360446019658863,336878679529361124],
(-8627477550934015644,-8626441922324893605],
(-2577622939633507215,-2575565821746779005],
(336878679529361124,343174083518476177],
(-3519645500583421982,-3517776702806344635],
(-2583420055923926426,-2577622939633507215],
(3383191969840261164,3387991562336214241],
(5394663393980481083,5411354651959353112],
(3387991562336214241,3390236953123836921],
(-705797240279985098,-700707451972711675],
(-5709906583533929520,-5707271395346631992],
(1178486765398617360,1178624428305220059],
(-3520515159422145033,-3519645500583421982]] failed with error [repair
#02b2bb40-b1ba-11e6-a5e3-61594f2eacfb on radius/radius_log_user,
[(5374555084199702397,5375907191203787693],
(5375907191203787693,5394663393980481083],
(-4608172686811846999,-4606297045741692934],
(343174083518476177,369373536835281005],
(-789474545083012335,-787318207609633720],
(-810527869609466298,-789474545083012335],
(7618946177842619265,7622080038542632561],
(8643901858335832297,8668152326416369637],
(332360446019658863,336878679529361124],
(-8627477550934015644,-8626441922324893605],
(-2577622939633507215,-2575565821746779005],
(336878679529361124,343174083518476177],
(-3519645500583421982,-3517776702806344635],
(-2583420055923926426,-2577622939633507215],
(3383191969840261164,3387991562336214241],
(5394663393980481083,5411354651959353112],
(3387991562336214241,3390236953123836921],
(-705797240279985098,-700707451972711675],
(-5709906583533929520,-5707271395346631992],
(1178486765398617360,1178624428305220059],
(-3520515159422145033,-3519645500583421982]]] Sync failed between
/10.101.123.54 and /10.101.123.50
org.apache.cassandra.exceptions.RepairException: [repair
#02b2bb40-b1ba-11e6-a5e3-61594f2eacfb on radius/radius_log_user,
[(5374555084199702397,5375907191203787693],
(5375907191203787693,5394663393980481083],
(-4608172686811846999,-4606297045741692934],
(343174083518476177,369373536835281005],
(-789474545083012335,-787318207609633720],
(-810527869609466298,-789474545083012335],
(7618946177842619265,7622080038542632561],
(8643901858335832297,8668152326416369637],
(332360446019658863,336878679529361124],
(-8627477550934015644,-8626441922324893605],
(-2577622939633507215,-2575565821746779005],
(336878679529361124,343174083518476177],
(-3519645500583421982,-3517776702806344635],
(-2583420055923926426,-2577622939633507215],
(3383191969840261164,3387991562336214241],
(5394663393980481083,5411354651959353112],
(3387991562336214241,3390236953123836921],
(-705797240279985098,-700707451972711675],
(-5709906583533929520,-5707271395346631992],
(1178486765398617360,1178624428305220059],
(-3520515159422145033,-3519645500583421982]]] Sync failed between
/10.101.123.54 and /10.101.123.50
at
org.apache.cassandra.repair.RemoteSyncTask.syncComplete(RemoteSyncTask.java:67)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:211)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:462)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:165)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_111]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_111]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
----- 2 -----
ERROR [RepairJobTask:26] 2016-11-24 03:41:30,346 RepairSession.java:290 -
[repair #02ed5341-b1ba-11e6-a5e3-61594f2eacfb] Session completed with the
following error
org.apache.cassandra.exceptions.RepairException: [repair
#02ed5341-b1ba-11e6-a5e3-61594f2eacfb on radius/radius_log_user,
[(-1335941665601891325,-1321221006770475620],
(-7656219523738612247,-7655796196565388324],
(7609334541554158623,7611123200966944805],
(4459027606150691982,4474972746587738233],
(6002584461649025600,6007853196169892926],
(3573722541167411217,3580784726724553369],
(-5464989069710829887,-5459049126444114128],
(-5459049126444114128,-5449173436278793738],
(-1642684260736153313,-1631913262033950119],
(-4419311717241338507,-4408863717427302952],
(-4005546528790223229,-4000695816476140849],
(714787271143415911,718355352027453636],
(-7655796196565388324,-7646812566394513626],
(-1631913262033950119,-1626800922236335484]]] Sync failed between
/10.101.123.37 and /10.101.123.54
at
org.apache.cassandra.repair.RemoteSyncTask.syncComplete(RemoteSyncTask.java:67)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:211)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:462)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:165)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_111]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_111]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
ERROR [RepairJobTask:26] 2016-11-24 03:41:30,347 RepairRunnable.java:261 -
Repair session 02ed5341-b1ba-11e6-a5e3-61594f2eacfb for range
[(-1335941665601891325,-1321221006770475620],
(-7656219523738612247,-7655796196565388324],
(7609334541554158623,7611123200966944805],
(4459027606150691982,4474972746587738233],
(6002584461649025600,6007853196169892926],
(3573722541167411217,3580784726724553369],
(-5464989069710829887,-5459049126444114128],
(-5459049126444114128,-5449173436278793738],
(-1642684260736153313,-1631913262033950119],
(-4419311717241338507,-4408863717427302952],
(-4005546528790223229,-4000695816476140849],
(714787271143415911,718355352027453636],
(-7655796196565388324,-7646812566394513626],
(-1631913262033950119,-1626800922236335484]] failed with error [repair
#02ed5341-b1ba-11e6-a5e3-61594f2eacfb on radius/radius_log_user,
[(-1335941665601891325,-1321221006770475620],
(-7656219523738612247,-7655796196565388324],
(7609334541554158623,7611123200966944805],
(4459027606150691982,4474972746587738233],
(6002584461649025600,6007853196169892926],
(3573722541167411217,3580784726724553369],
(-5464989069710829887,-5459049126444114128],
(-5459049126444114128,-5449173436278793738],
(-1642684260736153313,-1631913262033950119],
(-4419311717241338507,-4408863717427302952],
(-4005546528790223229,-4000695816476140849],
(714787271143415911,718355352027453636],
(-7655796196565388324,-7646812566394513626],
(-1631913262033950119,-1626800922236335484]]] Sync failed between
/10.101.123.37 and /10.101.123.54
org.apache.cassandra.exceptions.RepairException: [repair
#02ed5341-b1ba-11e6-a5e3-61594f2eacfb on radius/radius_log_user,
[(-1335941665601891325,-1321221006770475620],
(-7656219523738612247,-7655796196565388324],
(7609334541554158623,7611123200966944805],
(4459027606150691982,4474972746587738233],
(6002584461649025600,6007853196169892926],
(3573722541167411217,3580784726724553369],
(-5464989069710829887,-5459049126444114128],
(-5459049126444114128,-5449173436278793738],
(-1642684260736153313,-1631913262033950119],
(-4419311717241338507,-4408863717427302952],
(-4005546528790223229,-4000695816476140849],
(714787271143415911,718355352027453636],
(-7655796196565388324,-7646812566394513626],
(-1631913262033950119,-1626800922236335484]]] Sync failed between
/10.101.123.37 and /10.101.123.54
at
org.apache.cassandra.repair.RemoteSyncTask.syncComplete(RemoteSyncTask.java:67)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:211)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:462)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:165)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_111]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_111]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
----- 3 -----
ERROR [StreamReceiveTask:37] 2016-11-24 03:51:05,643 StreamSession.java:533 -
[Stream #93a1a410-b1e5-11e6-a5e3-61594f2eacfb] Streaming error occurred on
session with peer 10.101.123.37
ERROR [STREAM-IN-/10.101.123.37:7000] 2016-11-24 03:51:05,952
StreamSession.java:533 - [Stream #93a1a410-b1e5-11e6-a5e3-61594f2eacfb]
Streaming error occurred on session with peer 10.101.123.37
java.net.SocketException: End-of-stream reached
at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:69)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:292)
~[apache-cassandra-3.9.0.jar:3.9.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
----- 4 -----
ERROR [STREAM-IN-/10.101.123.54:7000] 2016-11-24 03:57:34,878
StreamSession.java:641 - [Stream #93a574a2-b1e5-11e6-a5e3-61594f2eacfb] Remote
peer 10.101.123.54 failed stream session.
ERROR [StreamReceiveTask:38] 2016-11-24 03:57:53,161 StreamSession.java:533 -
[Stream #93a574a2-b1e5-11e6-a5e3-61594f2eacfb] Streaming error occurred on
session with peer 10.101.123.54
----- 5 -----
ERROR [STREAM-IN-/10.101.123.53:7000] 2016-11-24 05:34:55,782
StreamSession.java:641 - [Stream #1233c743-b1e1-11e6-a5e3-61594f2eacfb] Remote
peer 10.101.123.53 failed stream session.
ERROR [RepairJobTask:27] 2016-11-24 05:34:55,786 RepairSession.java:290 -
[repair #02ded451-b1ba-11e6-a5e3-61594f2eacfb] Session completed with the
following error
org.apache.cassandra.streaming.StreamException: Stream failed
at
org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
~[apache-cassandra-3.9.0.jar:3.9.0]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
~[guava-18.0.jar:na]
at
org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:216)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:192)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:439)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.StreamSession.sessionFailed(StreamSession.java:642)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:494)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:298)
~[apache-cassandra-3.9.0.jar:3.9.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
ERROR [RepairJobTask:27] 2016-11-24 05:34:55,787 RepairRunnable.java:261 -
Repair session 02ded451-b1ba-11e6-a5e3-61594f2eacfb for range
[(4336322345184841912,4366163876979388475],
(-7933579975044282977,-7914796416580058518],
(1839892671254449208,1841457557372605571],
(1936854791639141672,1938001536070334116],
(1164510091499688902,1178486765398617360],
(-8158056158024454734,-8152122708824144179],
(-1849807793309054700,-1843766048765777468],
(-6348948024326794761,-6335446313814509922],
(8533993130815635284,8557700176962818555],
(-2638878446128576989,-2630418219576054523],
(8564018824386122995,8570162500655087152],
(8570162500655087152,8580710178918673615],
(8557700176962818555,8564018824386122995],
(8140631388675412405,8147625422662430664],
(4213947160008781931,4214821777286722249],
(-8152122708824144179,-8152031883182314791],
(8582180673436608178,8586996658256378648],
(4366163876979388475,4369191621839595605],
(8481593436612459797,8494752028882049497],
(2990658207661239276,2997421858321140810],
(4207239196271256582,4213947160008781931],
(6071207062879792046,6078438592707528242],
(-7906539059355975104,-7901946461742072348],
(-8403286291733434230,-8396456143995118801],
(8046054504348116945,8048439946921860060],
(-1785584523196265270,-1774397730504967785],
(-5740174840775988273,-5709906583533929520],
(-8166983564376894748,-8158056158024454734],
(8580710178918673615,8582180673436608178],
(-7914796416580058518,-7906539059355975104],
(-6349776557775176357,-6348948024326794761],
(-2575565821746779005,-2565520330427629056],
(8295018274402639934,8299841907912314103]] failed with error Stream failed
org.apache.cassandra.streaming.StreamException: Stream failed
at
org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
~[apache-cassandra-3.9.0.jar:3.9.0]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
~[guava-18.0.jar:na]
at
org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:216)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:192)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:439)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.StreamSession.sessionFailed(StreamSession.java:642)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:494)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:298)
~[apache-cassandra-3.9.0.jar:3.9.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
----- 6 -----
ERROR [StreamReceiveTask:36] 2016-11-24 05:45:04,687 StreamSession.java:533 -
[Stream #1233c743-b1e1-11e6-a5e3-61594f2eacfb] Streaming error occurred on
session with peer 10.101.123.53
java.lang.RuntimeException: Outgoing stream handler has been closed
at
org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:145)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:706)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:662)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:246)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[na:1.8.0_111]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_111]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
----- 7 -----
ERROR [StreamReceiveTask:81] 2016-11-24 08:53:45,489 StreamSession.java:533 -
[Stream #8708c770-b20c-11e6-a5e3-61594f2eacfb] Streaming error occurred on
session with peer 10.101.123.37
org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out -
received only 0 responses.
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:457)
~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.Keyspace.lambda$apply$74(Keyspace.java:466)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_111]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
~[apache-cassandra-3.9.0.jar:3.9.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
ERROR [RepairJobTask:22] 2016-11-24 08:53:45,493 RepairSession.java:290 -
[repair #03204a20-b1ba-11e6-a5e3-61594f2eacfb] Session completed with the
following error
org.apache.cassandra.streaming.StreamException: Stream failed
at
org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
~[apache-cassandra-3.9.0.jar:3.9.0]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
~[guava-18.0.jar:na]
at
org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:216)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:192)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:439)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:542)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:251)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_111]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_111]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
ERROR [RepairJobTask:22] 2016-11-24 08:53:45,494 RepairRunnable.java:261 -
Repair session 03204a20-b1ba-11e6-a5e3-61594f2eacfb for range
[(7904416027548030517,7904644339187931494],
(1992268616217213531,2011387685575474580],
(609927302916152607,627634447892460792],
(-7646812566394513626,-7644383923352462018],
(-4008413746138462176,-4005546528790223229],
(6817850835888923053,6818203283758294007],
(6534614382736965520,6538035742767863193],
(8508956293872390588,8512374867520547024],
(-1922781003874961453,-1922011619066347023],
(-1272836872170180568,-1271613201248661065],
(1976999083225979197,1992268616217213531],
(-2405793572890717179,-2402070345340440541],
(6533344633441688698,6534614382736965520],
(2765121196827852596,2787683730312124458],
(6538035742767863193,6548726064957823711],
(-7644383923352462018,-7642224985092838839],
(3171894574675724789,3172900306782399653],
(2197711694648372887,2206781329223605896],
(6818203283758294007,6818927700000354155],
(4297703982421974284,4305423041633973359],
(6530186721828199637,6533344633441688698],
(-8756481129378507914,-8750331462660878734],
(-7642224985092838839,-7639477375842151880]] failed with error Stream failed
org.apache.cassandra.streaming.StreamException: Stream failed
at
org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
~[apache-cassandra-3.9.0.jar:3.9.0]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
~[guava-18.0.jar:na]
at
org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:216)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:192)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:439)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:542)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:251)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_111]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_111]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
ERROR [STREAM-IN-/10.101.123.37:7000] 2016-11-24 08:53:46,317
StreamSession.java:533 - [Stream #8708c770-b20c-11e6-a5e3-61594f2eacfb]
Streaming error occurred on session with peer 10.101.123.37
java.net.SocketException: End-of-stream reached
at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:69)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:292)
~[apache-cassandra-3.9.0.jar:3.9.0]
------ 8 ------
ERROR [StreamReceiveTask:41] 2016-11-24 09:17:14,677 StreamSession.java:533 -
[Stream #1262c680-b1e1-11e6-a5e3-61594f2eacfb] Streaming error occurred on
session with peer 10.101.123.54
org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out -
received only 0 responses.
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:457)
~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.Keyspace.lambda$apply$74(Keyspace.java:466)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_111]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
~[apache-cassandra-3.9.0.jar:3.9.0]
ERROR [STREAM-IN-/10.101.123.54:7000] 2016-11-24 09:17:15,017
StreamSession.java:533 - [Stream #1262c680-b1e1-11e6-a5e3-61594f2eacfb]
Streaming error occurred on session with peer 10.101.123.54
java.net.SocketException: End-of-stream reached
at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:69)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:292)
~[apache-cassandra-3.9.0.jar:3.9.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
----- 9 -----
ERROR [StreamReceiveTask:72] 2016-11-24 09:47:02,248 StreamSession.java:533 -
[Stream #b0813591-b204-11e6-a5e3-61594f2eacfb] Streaming error occurred on
session with peer 10.101.123.54
org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out -
received only 0 responses.
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:457)
~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.Keyspace.lambda$apply$74(Keyspace.java:466)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_111]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
~[apache-cassandra-3.9.0.jar:3.9.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
INFO [StreamReceiveTask:72] 2016-11-24 09:47:02,249
StreamResultFuture.java:188 - [Stream #b0813591-b204-11e6-a5e3-61594f2eacfb]
Session with /10.101.123.54 is complete
WARN [StreamReceiveTask:72] 2016-11-24 09:47:02,249
StreamResultFuture.java:215 - [Stream #b0813591-b204-11e6-a5e3-61594f2eacfb]
Stream failed
WARN [RepairJobTask:25] 2016-11-24 09:47:02,265 RepairJob.java:153 - [repair
#031881f1-b1ba-11e6-a5e3-61594f2eacfb] radius_log_user sync failed
WARN [MutationStage-1] 2016-11-24 09:47:02,266
AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
Thread[MutationStage-1,5,main]: {}
java.lang.NullPointerException: null
ERROR [RepairJobTask:25] 2016-11-24 09:47:02,275 RepairSession.java:290 -
[repair #031881f1-b1ba-11e6-a5e3-61594f2eacfb] Session completed with the
following error
org.apache.cassandra.streaming.StreamException: Stream failed
at
org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
~[apache-cassandra-3.9.0.jar:3.9.0]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
~[guava-18.0.jar:na]
at
org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:216)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:192)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:439)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:542)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:251)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_111]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_111]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
ERROR [RepairJobTask:25] 2016-11-24 09:47:02,276 RepairRunnable.java:261 -
Repair session 031881f1-b1ba-11e6-a5e3-61594f2eacfb for range
[(-1774397730504967785,-1762758309398474120],
(8147625422662430664,8150788098883189595],
(2403112104956782810,2404720327411004342],
(6078438592707528242,6082354808100608599],
(-1370323679961738923,-1365059347020395178],
(7993733835100003655,8015005513539374688],
(-1857508277235383784,-1849807793309054700],
(-1370928608225827748,-1370323679961738923],
(8150788098883189595,8164821449219626475],
(8679953928760920469,8689453066878026377],
(-3426998630338216696,-3420776174229205444],
(3571513977484531056,3573722541167411217],
(2395920478339673197,2403112104956782810],
(7741141583060715116,7743151848268597969],
(5294878291570266517,5298658264359247830],
(3533995683509843155,3536422023436685141],
(3529213866575892196,3533995683509843155],
(8015005513539374688,8019140942924181546]] failed with error Stream failed
org.apache.cassandra.streaming.StreamException: Stream failed
at
org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
~[apache-cassandra-3.9.0.jar:3.9.0]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
~[guava-18.0.jar:na]
at
org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:216)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:192)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:439)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:542)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:251)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_111]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_111]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
----- 10 -----
ERROR [StreamReceiveTask:84] 2016-11-24 10:01:35,962 StreamSession.java:533 -
[Stream #14bf2f71-b215-11e6-a5e3-61594f2eacfb] Streaming error occurred on
session with peer 10.101.123.50
org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out -
received only 0 responses.
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:457)
~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.Keyspace.lambda$apply$74(Keyspace.java:466)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_111]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
~[apache-cassandra-3.9.0.jar:3.9.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
ERROR [RepairJobTask:26] 2016-11-24 10:01:35,966 RepairSession.java:290 -
[repair #032e05c0-b1ba-11e6-a5e3-61594f2eacfb] Session completed with the
following error
org.apache.cassandra.streaming.StreamException: Stream failed
at
org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
~[apache-cassandra-3.9.0.jar:3.9.0]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
~[guava-18.0.jar:na]
at
org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:216)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:192)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:439)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:542)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:251)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_111]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_111]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
ERROR [RepairJobTask:26] 2016-11-24 10:01:35,966 RepairRunnable.java:261 -
Repair session 032e05c0-b1ba-11e6-a5e3-61594f2eacfb for range
[(4787371764889471292,4791275868383167575],
(7660338039579725082,7682519547658080680],
(-7888038863789003009,-7883335201927605979],
(7551090506670172891,7557064621539106213],
(4203593558336671108,4204711535586868916],
(7682529946942857131,7700547538244194596],
(7545341063171693903,7547704657293476474],
(7547704657293476474,7551090506670172891],
(6833522370653895113,6851489965988389441],
(1281248143248933867,1283892626126598347],
(7682519547658080680,7682529946942857131],
(7712069670755918582,7713757639857850164],
(-2751798030355460938,-2750674055801082756],
(1275546448184112943,1281248143248933867],
(2787683730312124458,2789980706087291658],
(7700547538244194596,7712069670755918582],
(4731219540005886245,4735221596252133393],
(-2753266270806071514,-2751798030355460938],
(-4340445172605159261,-4330574594611691677]] failed with error Stream failed
org.apache.cassandra.streaming.StreamException: Stream failed
at
org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
~[apache-cassandra-3.9.0.jar:3.9.0]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
~[guava-18.0.jar:na]
at
org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:216)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:192)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:439)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:542)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:251)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_111]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_111]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
ERROR [STREAM-IN-/10.101.123.50:7000] 2016-11-24 10:01:36,713
StreamSession.java:533 - [Stream #14bf2f71-b215-11e6-a5e3-61594f2eacfb]
Streaming error occurred on session with peer 10.101.123.50
java.net.SocketException: End-of-stream reached
at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:69)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:292)
~[apache-cassandra-3.9.0.jar:3.9.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
----- 11 -----
ERROR [StreamReceiveTask:82] 2016-11-24 11:20:34,030 StreamSession.java:533 -
[Stream #873a10a2-b20c-11e6-a5e3-61594f2eacfb] Streaming error occurred on
session with peer 10.101.123.44
org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out -
received only 0 responses.
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:457)
~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.Keyspace.lambda$apply$74(Keyspace.java:466)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_111]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
~[apache-cassandra-3.9.0.jar:3.9.0]
--------------------------------
errors from node 10.101.123.46 :
system.log.2016-11-23_1838:ERROR [StreamReceiveTask:183] 2016-11-23
18:36:33,983 StreamSession.java:533 - [Stream
#b9289f40-b186-11e6-8990-19a24d4ea05e] Streaming error occurred on session with
peer 10.101.123.53
system.log.2016-11-23_1848:ERROR [STREAM-IN-/10.101.123.53:7000] 2016-11-23
18:43:59,827 StreamSession.java:533 - [Stream
#b9289f40-b186-11e6-8990-19a24d4ea05e] Streaming error occurred on session with
peer 10.101.123.53
---------------------------------
errors from node 10.101.123.36 :
ERROR [StreamReceiveTask:126] 2016-11-23 20:33:23,594 StreamSession.java:533 -
[Stream #b8914140-b186-11e6-81f0-a9fa8defd0bb] Streaming error occurred on
session with peer 10.101.123.37
---------------------------------
errors from node 10.101.123.37 :
system.log.2016-11-24_0945:ERROR [STREAM-IN-/10.101.123.36:43970] 2016-11-23
20:33:23,595 StreamSession.java:641 - [Stream
#b8914140-b186-11e6-81f0-a9fa8defd0bb] Remote peer 10.101.123.36 failed stream
session.
system.log.2016-11-24_0945:ERROR [StreamReceiveTask:13] 2016-11-24 03:41:27,162
Keyspace.java:516 - Unknown exception caught while attempting to update
MaterializedView! radius.radius_log_user
system.log.2016-11-24_0945:ERROR [StreamReceiveTask:13] 2016-11-24 03:41:27,169
StreamSession.java:533 - [Stream #93a1a410-b1e5-11e6-b34a-11869245b6ff]
Streaming error occurred on session with peer 10.101.123.54
system.log.2016-11-24_0945:ERROR [STREAM-OUT-/10.101.123.54:7000] 2016-11-24
03:41:30,343 StreamSession.java:533 - [Stream
#93a1a410-b1e5-11e6-b34a-11869245b6ff] Streaming error occurred on session with
peer 10.101.123.54
system.log.2016-11-24_0945:ERROR [STREAM-IN-/10.101.123.55:46259] 2016-11-24
03:51:05,653 StreamSession.java:641 - [Stream
#93a1a410-b1e5-11e6-a5e3-61594f2eacfb] Remote peer 10.101.123.55 failed stream
session.
system.log.2016-11-24_0945:ERROR [StreamReceiveTask:12] 2016-11-24 03:51:18,183
Keyspace.java:516 - Unknown exception caught while attempting to update
MaterializedView! radius.radius_log_user
system.log.2016-11-24_0945:ERROR [StreamReceiveTask:12] 2016-11-24 03:51:18,191
StreamSession.java:533 - [Stream #93a1a410-b1e5-11e6-a5e3-61594f2eacfb]
Streaming error occurred on session with peer 10.101.123.55
system.log.2016-11-24_0945:ERROR [STREAM-IN-/10.101.123.55:53963] 2016-11-24
08:53:45,490 StreamSession.java:641 - [Stream
#8708c770-b20c-11e6-a5e3-61594f2eacfb] Remote peer 10.101.123.55 failed stream
session.
system.log.2016-11-24_0945:ERROR [StreamReceiveTask:19] 2016-11-24 09:45:19,554
StreamSession.java:533 - [Stream #8708c770-b20c-11e6-a5e3-61594f2eacfb]
Streaming error occurred on session with peer 10.101.123.55
---------------------------------
errors from node 10.101.123.50 :
system.log:ERROR [STREAM-IN-/10.101.123.55:51504] 2016-11-24 10:01:35,962
StreamSession.java:641 - [Stream #14bf2f71-b215-11e6-a5e3-61594f2eacfb] Remote
peer 10.101.123.55 failed stream session.
system.log:ERROR [StreamReceiveTask:29] 2016-11-24 10:17:21,273
StreamSession.java:533 - [Stream #14bf2f71-b215-11e6-a5e3-61594f2eacfb]
Streaming error occurred on session with peer 10.101.123.55
system.log.2016-11-24_0943:ERROR [STREAM-IN-/10.101.123.54:40081] 2016-11-24
00:14:17,660 StreamSession.java:641 - [Stream
#dadda440-b1be-11e6-91dc-6d83e11e0103] Remote peer 10.101.123.54 failed stream
session.
system.log.2016-11-24_0943:ERROR [StreamReceiveTask:5] 2016-11-24 01:43:07,057
StreamSession.java:533 - [Stream #dadda440-b1be-11e6-91dc-6d83e11e0103]
Streaming error occurred on session with peer 10.101.123.54
---------------------------------
errors from node 10.101.123.53 :
system.log:ERROR [StreamReceiveTask:11] 2016-11-24 05:34:55,782
StreamSession.java:533 - [Stream #1233c743-b1e1-11e6-a5e3-61594f2eacfb]
Streaming error occurred on session with peer 10.101.123.55
system.log:ERROR [STREAM-IN-/10.101.123.55:51141] 2016-11-24 05:34:56,268
StreamSession.java:533 - [Stream #1233c743-b1e1-11e6-a5e3-61594f2eacfb]
Streaming error occurred on session with peer 10.101.123.55
system.log:ERROR [STREAM-IN-/10.101.123.54:7000] 2016-11-24 06:47:39,392
StreamSession.java:641 - [Stream #1265acb0-b1e1-11e6-8dc2-79786d1a296a] Remote
peer 10.101.123.54 failed stream session.
system.log:ERROR [StreamReceiveTask:12] 2016-11-24 07:29:10,812
StreamSession.java:533 - [Stream #1265acb0-b1e1-11e6-8dc2-79786d1a296a]
Streaming error occurred on session with peer 10.101.123.54
system.log.2016-11-23_1933:ERROR [STREAM-IN-/10.101.123.46:53169] 2016-11-23
18:36:33,983 StreamSession.java:641 - [Stream
#b9289f40-b186-11e6-8990-19a24d4ea05e] Remote peer 10.101.123.46 failed stream
session.
---------------------------------
errors from node 10.101.123.54 :
system.log:ERROR [StreamReceiveTask:7] 2016-11-24 00:14:17,661
StreamSession.java:533 - [Stream #dadda440-b1be-11e6-91dc-6d83e11e0103]
Streaming error occurred on session with peer 10.101.123.50
system.log:ERROR [STREAM-IN-/10.101.123.50:7000] 2016-11-24 00:14:18,508
StreamSession.java:533 - [Stream #dadda440-b1be-11e6-91dc-6d83e11e0103]
Streaming error occurred on session with peer 10.101.123.50
system.log:ERROR [STREAM-IN-/10.101.123.37:42462] 2016-11-24 03:41:42,022
StreamSession.java:641 - [Stream #93a1a410-b1e5-11e6-b34a-11869245b6ff] Remote
peer 10.101.123.37 failed stream session.
system.log:ERROR [StreamReceiveTask:18] 2016-11-24 03:57:34,863
Keyspace.java:516 - Unknown exception caught while attempting to update
MaterializedView! radius.radius_log_user
system.log:ERROR [StreamReceiveTask:18] 2016-11-24 03:57:34,871
StreamSession.java:533 - [Stream #93a574a2-b1e5-11e6-a5e3-61594f2eacfb]
Streaming error occurred on session with peer 10.101.123.55
system.log:ERROR [STREAM-IN-/10.101.123.55:42230] 2016-11-24 03:57:35,179
StreamSession.java:533 - [Stream #93a574a2-b1e5-11e6-a5e3-61594f2eacfb]
Streaming error occurred on session with peer 10.101.123.55
system.log:ERROR [StreamReceiveTask:19] 2016-11-24 06:47:39,391
StreamSession.java:533 - [Stream #1265acb0-b1e1-11e6-8dc2-79786d1a296a]
Streaming error occurred on session with peer 10.101.123.53
system.log:ERROR [STREAM-IN-/10.101.123.53:50120] 2016-11-24 06:47:40,371
StreamSession.java:533 - [Stream #1265acb0-b1e1-11e6-8dc2-79786d1a296a]
Streaming error occurred on session with peer 10.101.123.53
system.log:ERROR [STREAM-IN-/10.101.123.55:51183] 2016-11-24 09:17:14,678
StreamSession.java:641 - [Stream #1262c680-b1e1-11e6-a5e3-61594f2eacfb] Remote
peer 10.101.123.55 failed stream session.
system.log:ERROR [StreamReceiveTask:20] 2016-11-24 09:32:22,594
StreamSession.java:533 - [Stream #1262c680-b1e1-11e6-a5e3-61594f2eacfb]
Streaming error occurred on session with peer 10.101.123.55
system.log:ERROR [STREAM-IN-/10.101.123.55:54255] 2016-11-24 09:47:02,249
StreamSession.java:641 - [Stream #b0813591-b204-11e6-a5e3-61594f2eacfb] Remote
peer 10.101.123.55 failed stream session.
system.log.2016-11-23_1754:ERROR [STREAM-IN-/10.101.123.46:55942] 2016-11-23
17:52:43,889 StreamSession.java:641 - [Stream
#b9289f40-b186-11e6-8990-19a24d4ea05e] Remote peer 10.101.123.46 failed stream
session.
Thanks,
Nir.
> streaming issues with 3.9 (repair)
> ----------------------------------
>
> Key: CASSANDRA-12905
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12905
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: centos 6.7 x86_64
> Reporter: Nir Zilka
> Fix For: 3.9
>
>
> Hello,
> I performed two upgrades to the current cluster (currently 15 nodes, 1 DC,
> private VLAN),
> first it was 2.2.5.1 and repair worked flawlessly,
> second upgrade was to 3.0.9 (with upgradesstables) and also repair worked
> well,
> then i upgraded 2 weeks ago to 3.9 - and the repair problems started.
> there are several errors types from the system.log (different nodes) :
> - Sync failed between /xxx.xxx.xxx.xxx and /xxx.xxx.xxx.xxx
> - Streaming error occurred on session with peer xxx.xxx.xxx.xxx Operation
> timed out - received only 0 responses
> - Remote peer xxx.xxx.xxx.xxx failed stream session
> - Session completed with the following error
> org.apache.cassandra.streaming.StreamException: Stream failed
> ----
> i use 3.9 default configuration with the cluster settings adjustments (3
> seeds, GossipingPropertyFileSnitch).
> streaming_socket_timeout_in_ms is the default (86400000).
> i'm afraid from consistency problems while i'm not performing repair.
> Any ideas?
> Thanks,
> Nir.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
