[
https://issues.apache.org/jira/browse/CASSANDRA-14470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494571#comment-16494571
]
Kurt Greaves commented on CASSANDRA-14470:
------------------------------------------
Given version and that you appear to be using vnodes I wouldn't be surprised if
CASSANDRA-14332 is at play here. You can confirm if on the repairing nodes you
are seeing LOTS of Validation compactions occuring simultaneously.
We haven't released 3.11.3 yet, but you can try apply that one line patch on
CASSANDRA-14332 and see if it helps.
> Repair validation failed/unable to create merkle tree
> -----------------------------------------------------
>
> Key: CASSANDRA-14470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14470
> Project: Cassandra
> Issue Type: Bug
> Reporter: Harry Hough
> Priority: Major
>
> I had trouble repairing with a full repair across all nodes and keyspaces so
> I swapped to doing table by table. This table will not repair even after
> scrub/restart of all nodes. I am using command:
> {code:java}
> nodetool repair -full -seq keyspace table
> {code}
> {code:java}
> [2018-05-25 19:26:36,525] Repair session 0198ee50-6050-11e8-a3b7-9d0793eab507
> for range [(165598500763544933,166800441975877433],
> (-5455068259072262254,-5445777107512274819],
> (-4614366950466274594,-4609359222424798148],
> (3417371506258365094,3421921915575816226],
> (5221788898381458942,5222846663270250559],
> (3421921915575816226,3429175540277204991],
> (3276484330153091115,3282213186258578546],
> (-3306169730424140596,-3303439264231406101],
> (5228704360821395206,5242415853745535023],
> (5808045095951939338,5808562658315740708],
> (-3303439264231406101,-3302592736123212969]] finished (progress: 1%)
> [2018-05-25 19:27:23,848] Repair session 0180f980-6050-11e8-a3b7-9d0793eab507
> for range [(-8495158945319933291,-8482949618583319581],
> (1803296697741516342,1805330812863783941],
> (8633191319643427141,8637771071728131257],
> (2214097236323810344,2218253238829661319],
> (8637771071728131257,8639627594735133685],
> (2195525904029414718,2214097236323810344],
> (-8500127431270773970,-8495158945319933291],
> (7151693083782264341,7152162989417914407],
> (-8482949618583319581,-8481973749935314249]] finished (progress: 1%)
> [2018-05-25 19:30:32,590] Repair session 01ac9d62-6050-11e8-a3b7-9d0793eab507
> for range [(7887346492105510731,7893062759268864220],
> (-153277717939330979,-151986584968539220],
> (-6351665356961460262,-6336288442758847669],
> (7881942012672602731,7887346492105510731],
> (-5884528383037906783,-5878097817437987368],
> (6054625594262089428,6060773114960761336],
> (-6354401100436622515,-6351665356961460262],
> (3358411934943460772,3363367777663817876],
> (6255644242745576360,6278718135193665575],
> (-6321106762570843270,-6316788220143151823],
> (1754319239259058661,1759314644652031521],
> (7893062759268864220,7894890594190784729],
> (-8012293411840276426,-8011781808288431224]] failed with error [repair
> #01ac9d62-6050-11e8-a3b7-9d0793eab507 on keyspace/table,
> [(7887346492105510731,7893062759268864220],
> (-153277717939330979,-151986584968539220],
> (-6351665356961460262,-6336288442758847669],
> (7881942012672602731,7887346492105510731],
> (-5884528383037906783,-5878097817437987368],
> (6054625594262089428,6060773114960761336],
> (-6354401100436622515,-6351665356961460262],
> (3358411934943460772,3363367777663817876],
> (6255644242745576360,6278718135193665575],
> (-6321106762570843270,-6316788220143151823],
> (1754319239259058661,1759314644652031521],
> (7893062759268864220,7894890594190784729],
> (-8012293411840276426,-8011781808288431224]]] Validation failed in
> /192.168.8.64 (progress: 1%)
> [2018-05-25 19:30:38,744] Repair session 01ab16c1-6050-11e8-a3b7-9d0793eab507
> for range [(4474598255414218354,4477186372547790770],
> (-8368931070988054567,-8367389908801757978],
> (4445104759712094068,4445123832517144036],
> (6749641233379918040,6749879473217708908],
> (717627050679001698,729408043324000761],
> (8984622403893999385,8990662643404904110],
> (4457612694557846994,4474598255414218354],
> (5589049422573545528,5593079877787783784],
> (3609693317839644945,3613727999875360405],
> (8499016262183246473,8504603366117127178],
> (-5421277973540712245,-5417725796037372830],
> (5586405751301680690,5589049422573545528],
> (-2611069890590917549,-2603911539353128123],
> (2424772330724108233,2427564448454334730],
> (3172651438220766183,3175226710613527829],
> (4445123832517144036,4457612694557846994],
> (-6827531712183440570,-6800863837312326365],
> (5593079877787783784,5596020904874304252],
> (716705770783505310,717627050679001698],
> (115377252345874298,119626359210683992],
> (239394377432130766,240250561347730054]] failed with error [repair
> #01ab16c1-6050-11e8-a3b7-9d0793eab507 on keyspace/table,
> [(4474598255414218354,4477186372547790770],
> (-8368931070988054567,-8367389908801757978],
> (4445104759712094068,4445123832517144036],
> (6749641233379918040,6749879473217708908],
> (717627050679001698,729408043324000761],
> (8984622403893999385,8990662643404904110],
> (4457612694557846994,4474598255414218354],
> (5589049422573545528,5593079877787783784],
> (3609693317839644945,3613727999875360405],
> (8499016262183246473,8504603366117127178],
> (-5421277973540712245,-5417725796037372830],
> (5586405751301680690,5589049422573545528],
> (-2611069890590917549,-2603911539353128123],
> (2424772330724108233,2427564448454334730],
> (3172651438220766183,3175226710613527829],
> (4445123832517144036,4457612694557846994],
> (-6827531712183440570,-6800863837312326365],
> (5593079877787783784,5596020904874304252],
> (716705770783505310,717627050679001698],
> (115377252345874298,119626359210683992],
> (239394377432130766,240250561347730054]]] Validation failed in
> /192.168.8.63 (progress: 1%)
> [2018-05-25 19:31:49,787] Repair session 01a4ae20-6050-11e8-a3b7-9d0793eab507
> for range [(-2541759376733803975,-2534654569942446346],
> (5879245607426320709,5880012885546321040],
> (-6369551868880447648,-6359409984081717656],
> (-6599114937188060013,-6597469275333616279],
> (-5074096572632539578,-5067488659471711472],
> (-6379754598016153113,-6369551868880447648],
> (2064405355459946002,2071996664850745669],
> (-2534654569942446346,-2517719430302560572],
> (7881309182913674059,7881942012672602731],
> (-2544088936726049385,-2541759376733803975],
> (2279496339605311864,2281121064700207175],
> (7872992433920056063,7881309182913674059],
> (2062114659748646544,2064405355459946002],
> (-2150878401005443227,-2148033787477253835],
> (-1741268532521628862,-1723492194304925672],
> (-2148033787477253835,-2148008030576152684],
> (2274175180327961853,2279496339605311864]] failed with error [repair
> #01a4ae20-6050-11e8-a3b7-9d0793eab507 on keyspace/table,
> [(-2541759376733803975,-2534654569942446346],
> (5879245607426320709,5880012885546321040],
> (-6369551868880447648,-6359409984081717656],
> (-6599114937188060013,-6597469275333616279],
> (-5074096572632539578,-5067488659471711472],
> (-6379754598016153113,-6369551868880447648],
> (2064405355459946002,2071996664850745669],
> (-2534654569942446346,-2517719430302560572],
> (7881309182913674059,7881942012672602731],
> (-2544088936726049385,-2541759376733803975],
> (2279496339605311864,2281121064700207175],
> (7872992433920056063,7881309182913674059],
> (2062114659748646544,2064405355459946002],
> (-2150878401005443227,-2148033787477253835],
> (-1741268532521628862,-1723492194304925672],
> (-2148033787477253835,-2148008030576152684],
> (2274175180327961853,2279496339605311864]]] Validation failed in
> /192.168.8.64 (progress: 1%)
> [2018-05-25 19:31:49,845] Repair session 01c26f52-6050-11e8-a3b7-9d0793eab507
> for range [(-6336288442758847669,-6327494039552357362],
> (-6596499651894591521,-6570651311582753946],
> (-6597469275333616279,-6596499651894591521],
> (2057770067222008303,2062114659748646544],
> (-5870054111151365631,-5835304364517776345],
> (-3812151910311844467,-3802006636037441627],
> (-2619800330042834297,-2615481117037091603],
> (4808940926778034213,4810350864294758856],
> (-7508256920307222829,-7506372018227268626],
> (-7104590653728972577,-7104546570237712729],
> (3158009800098518496,3172651438220766183],
> (-2615481117037091603,-2611069890590917549],
> (-5878097817437987368,-5870054111151365631],
> (-2547658065527858190,-2544088936726049385],
> (232652608016417486,239394377432130766],
> (3154311195118940026,3158009800098518496]] failed with error [repair
> #01c26f52-6050-11e8-a3b7-9d0793eab507 on keyspace/table,
> [(-6336288442758847669,-6327494039552357362],
> (-6596499651894591521,-6570651311582753946],
> (-6597469275333616279,-6596499651894591521],
> (2057770067222008303,2062114659748646544],
> (-5870054111151365631,-5835304364517776345],
> (-3812151910311844467,-3802006636037441627],
> (-2619800330042834297,-2615481117037091603],
> (4808940926778034213,4810350864294758856],
> (-7508256920307222829,-7506372018227268626],
> (-7104590653728972577,-7104546570237712729],
> (3158009800098518496,3172651438220766183],
> (-2615481117037091603,-2611069890590917549],
> (-5878097817437987368,-5870054111151365631],
> (-2547658065527858190,-2544088936726049385],
> (232652608016417486,239394377432130766],
> (3154311195118940026,3158009800098518496]]] Validation failed in
> /192.168.10.63 (progress: 1%)
> [2018-05-25 19:31:50,027] Repair session 01b3f061-6050-11e8-a3b7-9d0793eab507
> for range [(2424051311739332070,2424772330724108233],
> (6848066208555197,10521229928033262],
> (992385332284940308,1000066900542109637],
> (4418797036920007266,4421783585221695744],
> (-5417725796037372830,-5412149532100548404],
> (178766242164281045,191217736969025363],
> (-3802006636037441627,-3796416071827586080],
> (5683533739750457455,5688298632819249302],
> (3653327414143088744,3655860906328373441],
> (3655860906328373441,3657219071532471378],
> (5746716543928841040,5753897313199191356],
> (-7506372018227268626,-7477180353912675682],
> (1911795960615895165,1921474545637686707],
> (4421783585221695744,4445104759712094068],
> (-4428987737460108139,-4413904067417968038],
> (5680321325075541449,5683533739750457455]] failed with error [repair
> #01b3f061-6050-11e8-a3b7-9d0793eab507 on keyspace/table,
> [(2424051311739332070,2424772330724108233],
> (6848066208555197,10521229928033262],
> (992385332284940308,1000066900542109637],
> (4418797036920007266,4421783585221695744],
> (-5417725796037372830,-5412149532100548404],
> (178766242164281045,191217736969025363],
> (-3802006636037441627,-3796416071827586080],
> (5683533739750457455,5688298632819249302],
> (3653327414143088744,3655860906328373441],
> (3655860906328373441,3657219071532471378],
> (5746716543928841040,5753897313199191356],
> (-7506372018227268626,-7477180353912675682],
> (1911795960615895165,1921474545637686707],
> (4421783585221695744,4445104759712094068],
> (-4428987737460108139,-4413904067417968038],
> (5680321325075541449,5683533739750457455]]] Validation failed in
> /192.168.10.63 (progress: 1%)
> [2018-05-25 19:31:50,065] Repair session 01d226c2-6050-11e8-a3b7-9d0793eab507
> for range [(731483217573828589,749016052425471844],
> (3349217091766639630,3355743728768043539],
> (8297509817744988677,8299811671851037140],
> (-1080064213437365415,-1067683134584617984],
> (-8988387420898594746,-8988256206650322851],
> (-1083473978088553649,-1080064213437365415],
> (-7068314886788869981,-7062826172876507507],
> (8299811671851037140,8306379796303668520],
> (-8500393685425499630,-8500127431270773970],
> (9077374236600850244,9080101637323836166],
> (9080101637323836166,9095536755598180114],
> (-2759657072078827823,-2750629632199441038],
> (-7938459356954944009,-7933123149264580832],
> (1759642905348136701,1772996641768793656],
> (-2788441126655538224,-2774970527117004032],
> (-7070810217579746608,-7068314886788869981],
> (-7959560447639828128,-7938459356954944009],
> (-7679921498492428955,-7664015662435807775]] failed with error [repair
> #01d226c2-6050-11e8-a3b7-9d0793eab507 on keyspace/table,
> [(731483217573828589,749016052425471844],
> (3349217091766639630,3355743728768043539],
> (8297509817744988677,8299811671851037140],
> (-1080064213437365415,-1067683134584617984],
> (-8988387420898594746,-8988256206650322851],
> (-1083473978088553649,-1080064213437365415],
> (-7068314886788869981,-7062826172876507507],
> (8299811671851037140,8306379796303668520],
> (-8500393685425499630,-8500127431270773970],
> (9077374236600850244,9080101637323836166],
> (9080101637323836166,9095536755598180114],
> (-2759657072078827823,-2750629632199441038],
> (-7938459356954944009,-7933123149264580832],
> (1759642905348136701,1772996641768793656],
> (-2788441126655538224,-2774970527117004032],
> (-7070810217579746608,-7068314886788869981],
> (-7959560447639828128,-7938459356954944009],
> (-7679921498492428955,-7664015662435807775]]] Validation failed in
> /192.168.8.63 (progress: 2%)
> [2018-05-25 19:32:24,797] Repair session 01aff8c0-6050-11e8-a3b7-9d0793eab507
> for range [(119626359210683992,128454334208965433],
> (6169854579148936152,6189260921105966960],
> (8460580156771389602,8466680988634247357],
> (10521229928033262,11278848941988721],
> (6165215300562655515,6169854579148936152],
> (191217736969025363,212964375650430729],
> (-5297146550802223153,-5294434130239676253],
> (6189260921105966960,6193074220809370652],
> (-655425716305023073,-647730635946823030]] failed with error [repair
> #01aff8c0-6050-11e8-a3b7-9d0793eab507 on keyspace/table,
> [(119626359210683992,128454334208965433],
> (6169854579148936152,6189260921105966960],
> (8460580156771389602,8466680988634247357],
> (10521229928033262,11278848941988721],
> (6165215300562655515,6169854579148936152],
> (191217736969025363,212964375650430729],
> (-5297146550802223153,-5294434130239676253],
> (6189260921105966960,6193074220809370652],
> (-655425716305023073,-647730635946823030]]] Validation failed in
> /192.168.10.63 (progress: 2%)
> [2018-05-25 19:32:24,873] Repair session 0199d8b1-6050-11e8-a3b7-9d0793eab507
> for range [(2708724319719658573,2710986923384204956],
> (6278718135193665575,6281813004301666161],
> (-8025315476660819134,-8015410683496661099],
> (2516704840921371424,2519633614752918103],
> (2519633614752918103,2526922953145276348],
> (8641102301927501454,8641256970223193109],
> (8643632109719583963,8645181823655307237],
> (-8015410683496661099,-8012293411840276426],
> (1368548173174048881,1373330457443776421],
> (5550121777767121,6848066208555197],
> (8641256970223193109,8643632109719583963],
> (-4201893423037098789,-4196287665648271477],
> (2692054381245703566,2708724319719658573],
> (-4208139091663389178,-4201893423037098789],
> (6281813004301666161,6282606461503930756],
> (-3470325001213070915,-3465759276556337455],
> (-4196287665648271477,-4185162268982289501],
> (-5006305410789315624,-5000646423000423501],
> (2714363942918413158,2722577239100121227],
> (5692402142504566885,5693342630493279303],
> (2710986923384204956,2714363942918413158],
> (5688298632819249302,5692402142504566885]] failed with error [repair
> #0199d8b1-6050-11e8-a3b7-9d0793eab507 on keyspace/table,
> [(2708724319719658573,2710986923384204956],
> (6278718135193665575,6281813004301666161],
> (-8025315476660819134,-8015410683496661099],
> (2516704840921371424,2519633614752918103],
> (2519633614752918103,2526922953145276348],
> (8641102301927501454,8641256970223193109],
> (8643632109719583963,8645181823655307237],
> (-8015410683496661099,-8012293411840276426],
> (1368548173174048881,1373330457443776421],
> (5550121777767121,6848066208555197],
> (8641256970223193109,8643632109719583963],
> (-4201893423037098789,-4196287665648271477],
> (2692054381245703566,2708724319719658573],
> (-4208139091663389178,-4201893423037098789],
> (6281813004301666161,6282606461503930756],
> (-3470325001213070915,-3465759276556337455],
> (-4196287665648271477,-4185162268982289501],
> (-5006305410789315624,-5000646423000423501],
> (2714363942918413158,2722577239100121227],
> (5692402142504566885,5693342630493279303],
> (2710986923384204956,2714363942918413158],
> (5688298632819249302,5692402142504566885]]] Validation failed in
> /192.168.8.65 (progress: 2%)
> Exception occurred during clean-up.
> java.lang.reflect.UndeclaredThrowableException
> Cassandra has shutdown.
> error: [2018-05-25 19:36:47,652] JMX connection closed. You should check
> server log for repair status of keyspace keyspace(Subsequent keyspaces are
> not going to be repaired).
> -- StackTrace --
> May 25, 2018 7:36:47 PM ClientCommunicatorAdmin Checker-run
> WARNING: Failed to check connection: java.net.SocketException: Connection
> reset
> java.io.IOException: [2018-05-25 19:36:47,652] JMX connection closed. You
> should check server log for repair status of keyspace keyspace(Subsequent
> keyspaces are not going to be repaired).
> at
> org.apache.cassandra.tools.RepairRunner.handleConnectionFailed(RepairRunner.java:98)
> at
> org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:86)
> at
> javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:275)
> at
> javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:352)
> at
> javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:337)
> at
> javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:248)
> at
> javax.management.remote.rmi.RMIConnector.sendNotification(RMIConnector.java:441)
> at
> javax.management.remote.rmi.RMIConnector.access$1200(RMIConnector.java:121)
> at
> javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.gotIOException(RMIConnector.java:1531)
> at
> com.sun.jmx.remote.internal.ClientCommunicatorAdmin$Checker.run(ClientCommunicatorAdmin.java:199)
> at java.lang.Thread.run(Thread.java:748)
> May 25, 2018 7:36:47 PM ClientCommunicatorAdmin Checker-run
> WARNING: stopping
> {code}
> Here is the log on one of the nodes where validation fails.
> {code:java}
> INFO [AntiEntropyStage:1] 2018-05-25 19:23:10,548 Validator.java:281 -
> [repair #01cf67a1-6050-11e8-a3b7-9d0793eab507] Sending completed merkle tree
> to /192.168.10.65 for pr$
> INFO [AntiEntropyStage:1] 2018-05-25 19:26:17,161 Validator.java:281 -
> [repair #01828020-6050-11e8-a3b7-9d0793eab507] Sending completed merkle tree
> to /192.168.10.65 for pr$
> INFO [AntiEntropyStage:1] 2018-05-25 19:26:23,909 Validator.java:281 -
> [repair #019dd051-6050-11e8-a3b7-9d0793eab507] Sending completed merkle tree
> to /192.168.10.65 for pr$
> INFO [AntiEntropyStage:1] 2018-05-25 19:28:15,118 Validator.java:281 -
> [repair #01c52e71-6050-11e8-a3b7-9d0793eab507] Sending completed merkle tree
> to /192.168.10.65 for pr$
> INFO [GossipTasks:1] 2018-05-25 19:30:23,087 Gossiper.java:1034 -
> InetAddress /192.168.10.65 is now DOWN
> INFO [HANDSHAKE-/192.168.10.65] 2018-05-25 19:30:31,093
> OutboundTcpConnection.java:560 - Handshaking version with /192.168.10.65
> INFO [HANDSHAKE-/192.168.10.65] 2018-05-25 19:30:31,281
> OutboundTcpConnection.java:560 - Handshaking version with /192.168.10.65
> INFO [RequestResponseStage-4] 2018-05-25 19:30:31,320 Gossiper.java:1019 -
> InetAddress /192.168.10.65 is now UP
> INFO [RequestResponseStage-3] 2018-05-25 19:30:31,320 Gossiper.java:1019 -
> InetAddress /192.168.10.65 is now UP
> INFO [RequestResponseStage-2] 2018-05-25 19:30:31,320 Gossiper.java:1019 -
> InetAddress /192.168.10.65 is now UP
> INFO [RequestResponseStage-1] 2018-05-25 19:30:31,320 Gossiper.java:1019 -
> InetAddress /192.168.10.65 is now UP
> INFO [RequestResponseStage-5] 2018-05-25 19:30:31,320 Gossiper.java:1019 -
> InetAddress /192.168.10.65 is now UP
> INFO [AntiEntropyStage:1] 2018-05-25 19:30:49,172 Validator.java:281 -
> [repair #01860291-6050-11e8-a3b7-9d0793eab507] Sending completed merkle tree
> to /192.168.10.65 for pr$
> INFO [HANDSHAKE-/192.168.10.65] 2018-05-25 19:30:49,188
> OutboundTcpConnection.java:560 - Handshaking version with /192.168.10.65
> INFO [HANDSHAKE-/192.168.10.65] 2018-05-25 19:30:54,188
> OutboundTcpConnection.java:569 - Cannot handshake version with /192.168.10.65
> INFO [HANDSHAKE-/192.168.10.65] 2018-05-25 19:30:54,188
> OutboundTcpConnection.java:560 - Handshaking version with /192.168.10.65
> INFO [HANDSHAKE-/192.168.10.65] 2018-05-25 19:30:59,188
> OutboundTcpConnection.java:569 - Cannot handshake version with /192.168.10.65
> INFO [GossipTasks:1] 2018-05-25 19:31:03,247 Gossiper.java:1034 -
> InetAddress /192.168.10.65 is now DOWN
> INFO [HANDSHAKE-/192.168.10.65] 2018-05-25 19:31:10,250
> OutboundTcpConnection.java:560 - Handshaking version with /192.168.10.65
> INFO [HANDSHAKE-/192.168.10.65] 2018-05-25 19:31:12,237
> OutboundTcpConnection.java:560 - Handshaking version with /192.168.10.65
> INFO [RequestResponseStage-7] 2018-05-25 19:31:12,712 Gossiper.java:1019 -
> InetAddress /192.168.10.65 is now UP
> INFO [RequestResponseStage-9] 2018-05-25 19:31:12,712 Gossiper.java:1019 -
> InetAddress /192.168.10.65 is now UP
> INFO [RequestResponseStage-13] 2018-05-25 19:31:12,712 Gossiper.java:1019 -
> InetAddress /192.168.10.65 is now UP
> INFO [GossipTasks:1] 2018-05-25 19:31:37,252 Gossiper.java:1034 -
> InetAddress /192.168.10.65 is now DOWN
> INFO [HANDSHAKE-/192.168.10.65] 2018-05-25 19:31:45,254
> OutboundTcpConnection.java:560 - Handshaking version with /192.168.10.65
> INFO [HANDSHAKE-/192.168.10.65] 2018-05-25 19:31:48,759
> OutboundTcpConnection.java:560 - Handshaking version with /192.168.10.65
> ERROR [ValidationExecutor:7] 2018-05-25 19:31:49,021 Validator.java:268 -
> Failed creating a merkle tree for [repair
> #01c26f52-6050-11e8-a3b7-9d0793eab507 on keyspace/$
> ERROR [ValidationExecutor:7] 2018-05-25 19:31:49,022 CassandraDaemon.java:228
> - Exception in thread
> Thread[ValidationExecutor:7,1,main]java.lang.RuntimeException: Parent repair
> session with id = 0103da40-6050-11e8-a3b7-9d0793eab507 has failed. at
> org.apache.cassandra.service.ActiveRepairService.getParentRepairSession(ActiveRepairService.java:412)
> ~[apache-cassandra-3.11.2.jar:3.11.2] at
> org.apache.cassandra.db.compaction.CompactionManager.getSSTablesToValidate(CompactionManager.java:1459)
> ~[apache-cassandra-3.11.2.jar:3.11.2] at
> org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1366)
> ~[apache-cassandra-3.11.2.jar:3.11.2] at
> org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:86)
> ~[apache-cassandra-3.11.2.jar:3.11.2] at
> org.apache.cassandra.db.compaction.CompactionManager$13.call(CompactionManager.java:955)
> ~[apache-cassandra-3.11.2.jar:3.11.2] at
> java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_171]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[na:1.8.0_171] at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [na:1.8.0_171] at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
> [apache-cassandra-3.11.2.jar:3.11.2] at
> java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_171
> ]INFO [RequestResponseStage-2] 2018-05-25 19:31:49,025 Gossiper.java:1019 -
> InetAddress /192.168.10.65 is now UP
> INFO [RequestResponseStage-3] 2018-05-25 19:31:49,025 Gossiper.java:1019 -
> InetAddress /192.168.10.65 is now UP
> INFO [RequestResponseStage-1] 2018-05-25 19:31:49,039 Gossiper.java:1019 -
> InetAddress /192.168.10.65 is now UP
> ERROR [ValidationExecutor:7] 2018-05-25 19:31:49,817 Validator.java:268 -
> Failed creating a merkle tree for [repair
> #01b3f061-6050-11e8-a3b7-9d0793eab507 on keyspace/$
> ERROR [ValidationExecutor:7] 2018-05-25 19:31:49,817 CassandraDaemon.java:228
> - Exception in thread
> Thread[ValidationExecutor:7,1,main]java.lang.RuntimeException: Parent repair
> session with id = 0103da40-6050-11e8-a3b7-9d0793eab507 has failed. at
> org.apache.cassandra.service.ActiveRepairService.getParentRepairSession(ActiveRepairService.java:412)
> ~[apache-cassandra-3.11.2.jar:3.11.2] at
> org.apache.cassandra.db.compaction.CompactionManager.getSSTablesToValidate(CompactionManager.java:1459)
> ~[apache-cassandra-3.11.2.jar:3.11.2] at
> org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1366)
> ~[apache-cassandra-3.11.2.jar:3.11.2] at
> org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:86)
> ~[apache-cassandra-3.11.2.jar:3.11.2] at
> org.apache.cassandra.db.compaction.CompactionManager$13.call(CompactionManager.java:955)
> ~[apache-cassandra-3.11.2.jar:3.11.2] at
> java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_171]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[na:1.8.0_171] at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [na:1.8.0_171] at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
> [apache-cassandra-3.11.2.jar:3.11.2] at
> java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_171]
> {code}
> 192.168.10.65 is the node where I started the repair. It looks like this node
> goes down before the merkle tree creation failure occurs? The debug log on
> the repair node is full of the below and doesn't help me much.
> {code:java}
> DEBUG [RepairJobTask:11] 2018-05-25 19:25:49,646 MerkleTree.java:295 - (17)
> Hashing sub-ranges [#<TreeRange (8732300281801533308,8732300339552037321]
> depth=18>, #<TreeRange (8732300339552037321,8732300397302541334] depth=18>]
> for #<TreeRange (8732300281801533308,8732300397302541334] depth=17> divided
> by midpoint 8732300339552037321
> DEBUG [RepairJobTask:11] 2018-05-25 19:25:49,647 MerkleTree.java:311 - (17)
> Inconsistent digest on left sub-range #<TreeRange
> (8732300281801533308,8732300339552037321] depth=18>: [#<Leaf
> [16cd9a47184232c7ed028a9d4546332d8a8c8ff83526f8f274997592eecc722d]>, #<Leaf
> [97b8f2fd61c1130ed2cfd8c2db52afdea59b2cabb5f39aed967cf8f2539f08b8]>]
> DEBUG [RepairJobTask:11] 2018-05-25 19:25:49,647 MerkleTree.java:333 - (17)
> Inconsistent digest on right sub-range #<TreeRange
> (8732300339552037321,8732300397302541334] depth=18>: [#<Leaf
> [fcf6daa5b5124a1e099e7776475aff22f2befedd88dc7b3e4277b92fd3115833]>, #<Leaf
> []>]
> DEBUG [RepairJobTask:11] 2018-05-25 19:25:49,647 MerkleTree.java:346 - (17)
> Fully inconsistent range [#<TreeRange
> (8732300281801533308,8732300339552037321] depth=18>, #<TreeRange
> (8732300339552037321,8732300397302541334] depth=18>]
> DEBUG [RepairJobTask:11] 2018-05-25 19:25:49,647 MerkleTree.java:346 - (16)
> Fully inconsistent range [#<TreeRange
> (8732300166300525283,8732300281801533308] depth=17>, #<TreeRange
> (8732300281801533308,8732300397302541334] depth=17>]
> DEBUG [RepairJobTask:11] 2018-05-25 19:25:49,647 MerkleTree.java:346 - (15)
> Fully inconsistent range [#<TreeRange
> (8732299935298509232,8732300166300525283] depth=16>, #<TreeRange
> (8732300166300525283,8732300397302541334] depth=16>]
> DEBUG [RepairJobTask:11] 2018-05-25 19:25:49,647 MerkleTree.java:346 - (14)
> Fully inconsistent range [#<TreeRange
> (8732299473294477131,8732299935298509232] depth=15>, #<TreeRange
> (8732299935298509232,8732300397302541334] depth=15>]
> DEBUG [RepairJobTask:11] 2018-05-25 19:25:49,647 MerkleTree.java:346 - (13)
> Fully inconsistent range [#<TreeRange
> (8732298549286412929,8732299473294477131] depth=14>, #<TreeRange
> (8732299473294477131,8732300397302541334] depth=14>]
> {code}
> Really at a loss of how to repair this table at this point.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
