Sorry I may be wrong about the cause - didn't see -full Mea culpa, its early here and I'm not awake
-- Jeff Jirsa > On Sep 18, 2017, at 7:01 AM, Steinmaurer, Thomas > <thomas.steinmau...@dynatrace.com> wrote: > > Hi Jeff, > > understood. That’s quite a change then coming from 2.1 from an operational > POV. > > Thanks again. > > Thomas > > From: Jeff Jirsa [mailto:jji...@gmail.com] > Sent: Montag, 18. September 2017 15:56 > To: user@cassandra.apache.org > Subject: Re: Multi-node repair fails after upgrading to 3.0.14 > > The command you're running will cause anticompaction and the range borders > for all instances at the same time > > Since only one repair session can anticompact any given sstable, it's almost > guaranteed to fail > > Run it on one instance at a time > > > -- > Jeff Jirsa > > > On Sep 18, 2017, at 1:11 AM, Steinmaurer, Thomas > <thomas.steinmau...@dynatrace.com> wrote: > > Hi Alex, > > I now ran nodetool repair –full –pr keyspace cfs on all nodes in parallel and > this may pop up now: > > 0.176.38.128 (progress: 1%) > [2017-09-18 07:59:17,145] Some repair failed > [2017-09-18 07:59:17,151] Repair command #3 finished in 0 seconds > error: Repair job has failed with the error message: [2017-09-18 > 07:59:17,145] Some repair failed > -- StackTrace -- > java.lang.RuntimeException: Repair job has failed with the error message: > [2017-09-18 07:59:17,145] Some repair failed > at > org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:115) > at > org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77) > at > com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:583) > at > com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:533) > at > com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452) > at > com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108) > > 2017-09-18 07:59:17 repair finished > > > If running the above nodetool call sequentially on all nodes, repair finishes > without printing a stack trace. > > The error message and stack trace isn’t really useful here. Any further > ideas/experiences? > > Thanks, > Thomas > > From: Alexander Dejanovski [mailto:a...@thelastpickle.com] > Sent: Freitag, 15. September 2017 11:30 > To: user@cassandra.apache.org > Subject: Re: Multi-node repair fails after upgrading to 3.0.14 > > Right, you should indeed add the "--full" flag to perform full repairs, and > you can then keep the "-pr" flag. > > I'd advise to monitor the status of your SSTables as you'll probably end up > with a pool of SSTables marked as repaired, and another pool marked as > unrepaired which won't be compacted together (hence the suggestion of running > subrange repairs). > Use sstablemetadata to check on the "Repaired at" value for each. 0 means > unrepaired and any other value (a timestamp) means the SSTable has been > repaired. > I've had behaviors in the past where running "-pr" on the whole cluster would > still not mark all SSTables as repaired, but I can't say if that behavior has > changed in latest versions. > > Having separate pools of SStables that cannot be compacted means that you > might have tombstones that don't get evicted due to partitions living in both > states (repaired/unrepaired). > > To sum up the recommendations : > - Run a full repair with both "--full" and "-pr" and check that SSTables are > properly marked as repaired > - Use a tight repair schedule to avoid keeping partitions for too long in > both repaired and unrepaired state > - Switch to subrange repair if you want to fully avoid marking SSTables as > repaired (which you don't need anyway since you're not using incremental > repairs). If you wish to do this, you'll have to mark back all your sstables > to unrepaired, using nodetool sstablerepairedset. > > Cheers, > > On Fri, Sep 15, 2017 at 10:27 AM Steinmaurer, Thomas > <thomas.steinmau...@dynatrace.com> wrote: > Hi Alex, > > thanks a lot. Somehow missed that incremental repairs are the default now. > > We have been happy with full repair so far, cause data what we currently > manually invoke for being prepared is a small (~1GB or even smaller). > > So I guess with full repairs across all nodes, we still can stick with the > partition range (-pr) option, but with 3.0 we additionally have to provide > the –full option, right? > > Thanks again, > Thomas > > From: Alexander Dejanovski [mailto:a...@thelastpickle.com] > Sent: Freitag, 15. September 2017 09:45 > To: user@cassandra.apache.org > Subject: Re: Multi-node repair fails after upgrading to 3.0.14 > > Hi Thomas, > > in 2.1.18, the default repair mode was full repair while since 2.2 it is > incremental repair. > So running "nodetool repair -pr" since your upgrade to 3.0.14 doesn't trigger > the same operation. > > Incremental repair cannot run on more than one node at a time on a cluster, > because you risk to have conflicts with sessions trying to anticompact and > run validation compactions on the same SSTables (which will make the > validation phase fail, like your logs are showing). > Furthermore, you should never use "-pr" with incremental repair because it is > useless in that mode, and won't properly perform anticompaction on all nodes. > > If you were happy with full repairs in 2.1.18, I'd suggest to stick with > those in 3.0.14 as well because there are still too many caveats with > incremental repairs that should hopefully be fixed in 4.0+. > Note that full repair will also trigger anticompaction and mark SSTables as > repaired in your release of Cassandra, and only full subrange repairs are the > only flavor that will skip anticompaction. > > You will need some tooling to help with subrange repairs though, and I'd > recommend to use Reaper which handles automation for you : > http://cassandra-reaper.io/ > > If you decide to stick with incremental repairs, first perform a rolling > restart of your cluster to make sure no repair session still runs, and run > "nodetool repair" on a single node at a time. Move on to the next node only > when nodetool or the logs show that repair is over (which will include the > anticompaction phase). > > Cheers, > > > > On Fri, Sep 15, 2017 at 8:42 AM Steinmaurer, Thomas > <thomas.steinmau...@dynatrace.com> wrote: > Hello, > > we are currently in the process of upgrading from 2.1.18 to 3.0.14. After > upgrading a few test environments, we start to see some suspicious log > entries regarding repair issues. > > We have a cron job on all nodes basically executing the following repair call > on a daily basis: > > nodetool repair –pr <list of CFs> > > This gets started on all nodes at the same time. While this has worked with > 2.1.18 (at least we haven’t seen anything suspicious in Cassandra log), with > 3.0.14 we get something similar like that on all nodes (see below; IP > addresses and KS/CF faked). > > Any pointers are appreciated. Thanks. > Thomas > > > INFO [Thread-2941] 2017-09-15 03:00:28,036 RepairSession.java:224 - [repair > #071f81e0-99c2-11e7-91dc-6132f5fe5fb0] new session: will sync /FAKE.33.64, > /FAKE.35.153, /FAKE.34.171 on range > [(8195393703879512303,8196334842725538685], > (8166975326273137878,8182604850967732931], > (-7246799942440641887,-7227869626613009045], > (-8371707510273823988,-8365977215604569699], > (-141862581573028594,-140310864869418908], > (3732113975108886193,3743105867152786342], > (4998127507903069087,5008922734235607550], > (-5115827291264930140,-5111054924035590372], > (-2475342271852943287,-2447285553369030332], > (-8318606053827235336,-8308721754886697230], > (-5208900659917654871,-5202385837264015269], > (6618737991399272130,6623100721269775102], > (-4650650128572424858,-4650260492494258461], > (1886545362164970333,1886646959491599822], > (-4511817721998311568,-4507491187192881115], > (8114903118676615937,8132992506844206601], > (6224957219376301858,6304379125732293904], > (-3460547504877234383,-3459262416082517136], > (-167838948111369123,-141862581573028594], > (481579232521229473,491242114841289497], > (4052464144722307684,4059745901618136723], > (1659668187498418295,1679582585970705122], > (-1118922763210109192,-1093766915505652874], > (7504365235878319341,7526188885210185292], > (-79866884352549492,-77667207866300333], > (8151204058820798561,8154760186218662205], > (-1040398370287131739,-1033770179677543189], > (3767057277953758442,3783780844370292025], > (-6491678058233994892,-6487797181789288329], > (-916868210769480248,-907141794196269524], > (-9005441616028750657,-9002220258513351832], > (8183526518331102304,8186908810225025483], > (-5685737903527826627,-5672136154194382932], > (4976122621177738811,4987871287137312689], > (6051670147160447042,6051686987147911650], > (-1161640137086921883,-1159172734746043158], > (6895951547735922309,6899152466544114890], > (-3357667382515377172,-3356304907368646189], > (-5370953856683870319,-5345971445444542485], > (3824272999898372667,3829315045986248983], > (8132992506844206601,8149858096109302285], > (3975126143101303723,3980729378827590597], > (-956691623200349709,-946602525018301692], > (-82499927325251331,-79866884352549492], > (3952144214544622998,3955602392726495936], > (8154760186218662205,8157079055586089583], > (3840595196718778916,3866458971850198755], > (-1066905024007783341,-1055954824488508260], > (-7252356975874511782,-7246799942440641887], > (-810612946397276081,-792189809286829222], > (4964519403172053705,4970446606512414858], > (-5380038118840759647,-5370953856683870319], > (-3221630728515706463,-3206856875356976885], > (-1193448110686154165,-1161640137086921883], > (-3356304907368646189,-3346460884208327912], > (3466596314109623830,3468144326691723333], > (-9050241313548454460,-9005441616028750657], > (402227699082311580,407458511300218383]] for XXX.[YYY, ZZZ] > INFO [Repair#1:1] 2017-09-15 03:00:28,419 RepairJob.java:172 - [repair > #071f81e0-99c2-11e7-91dc-6132f5fe5fb0] Requesting merkle trees for YYY (to > [/FAKE.35.153, /FAKE.34.171, /FAKE.33.64]) > INFO [Thread-2941] 2017-09-15 03:00:28,434 RepairSession.java:224 - [repair > #075d2720-99c2-11e7-91dc-6132f5fe5fb0] new session: will sync /FAKE.33.64, > /FAKE.35.57, /FAKE.34.171 on range > [(-5410955131843184047,-5390722609201388849], > (-2429793939970389370,-2402273315769352748], > (8085575576842594575,8086965740279021106], > (-8802193901675845653,-8790472027607832351], > (-3900412470120874591,-3892641480459306647], > (5455804264750818305,5465037357825542970], > (4930767198829659527,4939587074207662799], > (8086965740279021106,8087442741329154201], > (-8933201045321260661,-8926445549049070674], > (-4841328524165418854,-4838895482794593338], > (628107265570603622,682509946926464280], > (7043245467621414187,7055126022831789025], > (624871765540463735,627374995781897409], > (9219228482330263660,9221294940422311559], > (-2335215188301493066,-2315034243278984017], > (-6216599212198827632,-6211460136507414133], > (-3276490559558850323,-3273110814046238767], > (7204991007334459472,7214826985711309418], > (1815809811279373566,1846961604192445001], > (8743912118048160970,8751518028513315549], > (-9204701745739426439,-9200185935622985719], > (7926527126882050773,7941554683778488797], > (-1307707180308444994,-1274682085495751899], > (8354147540115782875,8358523989614737607], > (-5418282332713406631,-5415077779309282099], > (2436459402559272117,2441988676982099299], > (3718536403335860823,3722612994041485455], > (613072756805503750,624871765540463735], > (3348338086790700840,3352211222405547568], > (7941554683778488797,7949493423692701952], > (-319434281654804937,-312029192727845573], > (6560852500852746428,6577389024948220270], > (3468872066705970665,3469611600245714638], > (-4691666557374653357,-4691655196509961176], > (1517867403379415135,1524487506534693145], > (-7422802504145504981,-7402342710717945820], > (1766173970230627147,1798177111825664424], > (-4402244320098942276,-4399211415161403124], > (-6024004998426219783,-6013031695922198269], > (-6930256177103656090,-6926607739769548590], > (-3578196517591043633,-3561058036097453011], > (-7923695406605667073,-7885733750925254770], > (-8063817427378208986,-8061632046535266293], > (-7341733690813664665,-7287847431444491429], > (-8327456680139422800,-8322944736494135341], > (885662136143255295,898069608029365525], > (-5555349213881020405,-5555145874347997903], > (-4895525453201761515,-4892963646364151947], > (4925572085497096406,4926057814710187377], > (7785700473508314561,7813617525484553901], > (-1598850926292630432,-1593204162764888475], > (-2583030464997858536,-2581717714589673598], > (7780476570449984013,7785700473508314561], > (-3558045406807923433,-3550343462935144866], > (7571721681971063633,7578715430118769516], > (8865905084309486257,8879414598472867800], > (-4984170160639031182,-4973560491294201055], > (-5469249030580263062,-5432196900285376733], > (-340636581410346966,-324449230085298889], > (9120471287581648922,9127588925404102522], > (8366987539912250171,8368772126617619299], > (-9125573241533654210,-9123870950381500195], > (1193377366975272040,1205407211929215537], > (8879414598472867800,8900714457425949896], > (7055126022831789025,7065799572034111661], > (-1588707445511226686,-1539910209795101605], > (6990922382177634221,7007948980474566617]] for XXX.[YYY, ZZZ] > INFO [Thread-2941] 2017-09-15 03:00:28,778 RepairSession.java:224 - [repair > #0791a4a0-99c2-11e7-91dc-6132f5fe5fb0] new session: will sync /FAKE.33.64, > /FAKE.35.57, /FAKE.34.90 on range > [(-3078252446668567055,-3069777768104388887], > (3648099278098819362,3665843214181043507], > (5797678959887947223,5820452778888079981], > (-2642545928835445644,-2631494184435884123], > (-1612062874902452641,-1610692114449831135], > (-1707971415501754602,-1692411707392102715], > (291977720343160936,302780457185309086], > (-8155767539645304082,-8140296327332660530], > (5553475010776119070,5565820507856584918], > (3560286576304809625,3566372473455883512], > (5226123540640728306,5228400303590993685], > (3421729237821036215,3428685863718017921], > (1227377289103207735,1240348980976161342], > (3246531324407742399,3247845316596819400], > (5304633131413824443,5306083993155890840], > (-8619514398407319973,-8606439320645645548], > (5911017071945481872,5912872298340550901], > (-5637986723275926091,-5632042588860715397], > (-7132684921710202809,-7132365563419969085], > (8246342224027692193,8283424185055442516], > (-4044458382788374703,-4036186919446747968], > (-5613445605139257263,-5596969946550242654], > (1083135773045625799,1084813234330856986], > (-8140296327332660530,-8138240730730282332], > (2996936679493130945,3002752531484603869], > (-4490677154018906600,-4487029511866186581], > (8456154651893349894,8463606226270547769], > (-2656313914110802764,-2642545928835445644], > (7193156006022031982,7197068598281426786], > (8318481021259042556,8345872746994439720], > (-1692411707392102715,-1673006435564531865], > (1097191643473406518,1104630053604629743], > (-3613177001569108681,-3599878627600515791], > (-1662308889719802294,-1656482847672279831], > (-6811080679276087557,-6804940650782637386], > (7392275851743093948,7393304112887579707], > (4240513765329086484,4257716503015670881], > (-5036265422304136579,-5029426061972234042], > (-4230577892781670592,-4228053239714848467], > (-8159151100170546488,-8155767539645304082], > (-5492523660879132140,-5478464594772026508], > (3421125671176312710,3421729237821036215], > (5262611881402902860,5275876653180871852], > (1577722798587790899,1613215617842269344], > (6666156845317387896,6676801654642938276], > (5155814764198615180,5168412530555049843], > (-6824229014637940118,-6819748162411093540], > (-3666684985038092211,-3613177001569108681], > (-8874623295866980549,-8872063750661463000], > (-3599878627600515791,-3599733923716673512], > (-5960723066215554398,-5949787783010678970], > (258158907638948329,259127284829404565], > (6661762711488840391,6666156845317387896], > (-6367100347601878936,-6363911697345955296], > (-6250542414915159381,-6229550501272299394], > (-6917261891898915460,-6904807338337882298], > (-7449919126757571330,-7438527202807402113], > (-6904807338337882298,-6881332882762615208], > (-7942161058117423218,-7938222136384107977], > (781504901963705642,791302021433124535], > (6527741860207715339,6537050054260012793], > (7186200312153798494,7187207897161667549]] for XXX.[YYY, ZZZ] > INFO [Thread-2941] 2017-09-15 03:00:28,942 RepairSession.java:224 - [repair > #07aaaae0-99c2-11e7-91dc-6132f5fe5fb0] new session: will sync /FAKE.33.64, > /FAKE.35.153, /FAKE.34.90 on range > [(3075097238958732453,3095612227388458535], > (-3791204388712221856,-3785200705970537637], > (8410714452405602873,8417924634749091961], > (5114137550939144346,5121098642749542165], > (2740032807677652551,2746972622905345611], > (2197312222134575375,2220347677927119914], > (4864988928009437565,4869276508135102594], > (-3451050463660365008,-3450091362870187260], > (7424782039380072835,7472858232992173041], > (-5338516343760774634,-5325637469358550877], > (2731359217194343571,2734544646630861913], > (382200642783712616,390184326306122373], > (430251086609083397,431821763463501369], > (-5338895526845396808,-5338516343760774634], > (-2677272836643942252,-2671813555202977248], > (-2132202487362637132,-2128425620095084071], > (2165546227633605752,2174360971273026818], > (2763720958753148061,2764848409944217454], > (3538741049060169597,3547917038270802802], > (-6741608677619654195,-6731111079555719429], > (2764848409944217454,2779298419000989854], > (374883901669627753,382200642783712616], > (-2706148319144852162,-2694312255222799319], > (7132646356350938192,7136352706932176819], > (194821142331236531,228318513249291325], > (-2047674613927860008,-2032759017860426681], > (-2032759017860426681,-2011077831196488705], > (7472858232992173041,7477981949405760270], > (431821763463501369,449911816694531926], > (4347942912351042626,4349624658068191606], > (311672118330363382,320380777495445406], > (3099877433733294763,3112419968835610433], > (-7182182373680571317,-7177421545891675761], > (-6742074278675667310,-6741608677619654195], > (-2770159943814078393,-2757633164368231752], > (3095612227388458535,3099877433733294763], > (-3741816028443553809,-3741630281022069399], > (-738951315633674297,-727109613092919188], > (4851195187346961355,4864988928009437565], > (-769229207399684543,-744901822064683937], > (-4143548408671617625,-4141029975015318194], > (-7957562451432593113,-7952882742232241654], > (-511680378144663402,-494227476671052554], > (-832115128813031544,-819699210830328959], > (6784361874104123161,6792435444931864417], > (-8284973647739186524,-8256379601422884439], > (-5321596150219749918,-5309942391653520721], > (4723799060589319973,4734570102349898748], > (5077208816402461873,5091797538025862046], > (7487792874019513006,7489186250794409815], > (-1647123563569159073,-1620470309759457440], > (2779298419000989854,2784870878309140225], > (320380777495445406,344020839234233467], > (3112419968835610433,3119057615014897057], > (-8506504972343997648,-8502433630545238869], > (-2810980092222625726,-2810271001960433176], > (344020839234233467,347093216302465566], > (-8545779729891984163,-8519832061775919840], > (2633165392330429557,2680549740474631393], > (6391794517057168460,6431615673158447973], > (6440766036098716417,6458083094779720657], > (-8735147357564473077,-8730522574908969951], > (7484355688408489651,7487792874019513006], > (4325444310827485194,4347942912351042626], > (2220347677927119914,2223568835084606157], > (6802149418704156481,6819713717509862346], > (4827439895247379322,4851195187346961355], > (-4016529712384875309,-4009393333251988044], > (2018021481024026660,2049270980733207626]] for XXX.[YYY, ZZZ] > ERROR [ValidationExecutor:3] 2017-09-15 03:00:29,471 > ActiveRepairService.java:554 - Cannot start multiple repair sessions over the > same sstables > ERROR [ValidationExecutor:3] 2017-09-15 03:00:29,471 Validator.java:268 - > Failed creating a merkle tree for [repair > #071f81e0-99c2-11e7-91dc-6132f5fe5fb0 on XXX/YYY, > [(8195393703879512303,8196334842725538685], > (8166975326273137878,8182604850967732931], > (-7246799942440641887,-7227869626613009045], > (-8371707510273823988,-8365977215604569699], > (-141862581573028594,-140310864869418908], > (3732113975108886193,3743105867152786342], > (4998127507903069087,5008922734235607550], > (-5115827291264930140,-5111054924035590372], > (-2475342271852943287,-2447285553369030332], > (-8318606053827235336,-8308721754886697230], > (-5208900659917654871,-5202385837264015269], > (6618737991399272130,6623100721269775102], > (-4650650128572424858,-4650260492494258461], > (1886545362164970333,1886646959491599822], > (-4511817721998311568,-4507491187192881115], > (8114903118676615937,8132992506844206601], > (6224957219376301858,6304379125732293904], > (-3460547504877234383,-3459262416082517136], > (-167838948111369123,-141862581573028594], > (481579232521229473,491242114841289497], > (4052464144722307684,4059745901618136723], > (1659668187498418295,1679582585970705122], > (-1118922763210109192,-1093766915505652874], > (7504365235878319341,7526188885210185292], > (-79866884352549492,-77667207866300333], > (8151204058820798561,8154760186218662205], > (-1040398370287131739,-1033770179677543189], > (3767057277953758442,3783780844370292025], > (-6491678058233994892,-6487797181789288329], > (-916868210769480248,-907141794196269524], > (-9005441616028750657,-9002220258513351832], > (8183526518331102304,8186908810225025483], > (-5685737903527826627,-5672136154194382932], > (4976122621177738811,4987871287137312689], > (6051670147160447042,6051686987147911650], > (-1161640137086921883,-1159172734746043158], > (6895951547735922309,6899152466544114890], > (-3357667382515377172,-3356304907368646189], > (-5370953856683870319,-5345971445444542485], > (3824272999898372667,3829315045986248983], > (8132992506844206601,8149858096109302285], > (3975126143101303723,3980729378827590597], > (-956691623200349709,-946602525018301692], > (-82499927325251331,-79866884352549492], > (3952144214544622998,3955602392726495936], > (8154760186218662205,8157079055586089583], > (3840595196718778916,3866458971850198755], > (-1066905024007783341,-1055954824488508260], > (-7252356975874511782,-7246799942440641887], > (-810612946397276081,-792189809286829222], > (4964519403172053705,4970446606512414858], > (-5380038118840759647,-5370953856683870319], > (-3221630728515706463,-3206856875356976885], > (-1193448110686154165,-1161640137086921883], > (-3356304907368646189,-3346460884208327912], > (3466596314109623830,3468144326691723333], > (-9050241313548454460,-9005441616028750657], > (402227699082311580,407458511300218383]]], /FAKE.33.64 (see log for details) > INFO [AntiEntropyStage:1] 2017-09-15 03:00:29,473 RepairSession.java:176 - > [repair #071f81e0-99c2-11e7-91dc-6132f5fe5fb0] Received merkle tree for YYY > from /FAKE.33.64 > ERROR [Repair#1:1] 2017-09-15 03:00:29,492 CassandraDaemon.java:207 - > Exception in thread Thread[Repair#1:1,5,RMI Runtime] > com.google.common.util.concurrent.UncheckedExecutionException: > org.apache.cassandra.exceptions.RepairException: [repair > #071f81e0-99c2-11e7-91dc-6132f5fe5fb0 on XXX/YYY, > [(8195393703879512303,8196334842725538685], > (8166975326273137878,8182604850967732931], > (-7246799942440641887,-7227869626613009045], > (-8371707510273823988,-8365977215604569699], > (-141862581573028594,-140310864869418908], > (3732113975108886193,3743105867152786342], > (4998127507903069087,5008922734235607550], > (-5115827291264930140,-5111054924035590372], > (-2475342271852943287,-2447285553369030332], > (-8318606053827235336,-8308721754886697230], > (-5208900659917654871,-5202385837264015269], > (6618737991399272130,6623100721269775102], > (-4650650128572424858,-4650260492494258461], > (1886545362164970333,1886646959491599822], > (-4511817721998311568,-4507491187192881115], > (8114903118676615937,8132992506844206601], > (6224957219376301858,6304379125732293904], > (-3460547504877234383,-3459262416082517136], > (-167838948111369123,-141862581573028594], > (481579232521229473,491242114841289497], > (4052464144722307684,4059745901618136723], > (1659668187498418295,1679582585970705122], > (-1118922763210109192,-1093766915505652874], > (7504365235878319341,7526188885210185292], > (-79866884352549492,-77667207866300333], > (8151204058820798561,8154760186218662205], > (-1040398370287131739,-1033770179677543189], > (3767057277953758442,3783780844370292025], > (-6491678058233994892,-6487797181789288329], > (-916868210769480248,-907141794196269524], > (-9005441616028750657,-9002220258513351832], > (8183526518331102304,8186908810225025483], > (-5685737903527826627,-5672136154194382932], > (4976122621177738811,4987871287137312689], > (6051670147160447042,6051686987147911650], > (-1161640137086921883,-1159172734746043158], > (6895951547735922309,6899152466544114890], > (-3357667382515377172,-3356304907368646189], > (-5370953856683870319,-5345971445444542485], > (3824272999898372667,3829315045986248983], > (8132992506844206601,8149858096109302285], > (3975126143101303723,3980729378827590597], > (-956691623200349709,-946602525018301692], > (-82499927325251331,-79866884352549492], > (3952144214544622998,3955602392726495936], > (8154760186218662205,8157079055586089583], > (3840595196718778916,3866458971850198755], > (-1066905024007783341,-1055954824488508260], > (-7252356975874511782,-7246799942440641887], > (-810612946397276081,-792189809286829222], > (4964519403172053705,4970446606512414858], > (-5380038118840759647,-5370953856683870319], > (-3221630728515706463,-3206856875356976885], > (-1193448110686154165,-1161640137086921883], > (-3356304907368646189,-3346460884208327912], > (3466596314109623830,3468144326691723333], > (-9050241313548454460,-9005441616028750657], > (402227699082311580,407458511300218383]]] Validation failed in /FAKE.33.64 > at > com.google.common.util.concurrent.Futures.wrapAndThrowUnchecked(Futures.java:1525) > ~[guava-18.0.jar:na] > at > com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1511) > ~[guava-18.0.jar:na] > at org.apache.cassandra.repair.RepairJob.run(RepairJob.java:160) > ~[apache-cassandra-3.0.14.jar:3.0.14] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > ~[na:1.8.0_102] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > ~[na:1.8.0_102] > at > org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79) > ~[apache-cassandra-3.0.14.jar:3.0.14] > at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_102] > Caused by: org.apache.cassandra.exceptions.RepairException: [repair > #071f81e0-99c2-11e7-91dc-6132f5fe5fb0 on XXX/YYY, > [(8195393703879512303,8196334842725538685], > (8166975326273137878,8182604850967732931], > (-7246799942440641887,-7227869626613009045], > (-8371707510273823988,-8365977215604569699], > (-141862581573028594,-140310864869418908], > (3732113975108886193,3743105867152786342], > (4998127507903069087,5008922734235607550], > (-5115827291264930140,-5111054924035590372], > (-2475342271852943287,-2447285553369030332], > (-8318606053827235336,-8308721754886697230], > (-5208900659917654871,-5202385837264015269], > (6618737991399272130,6623100721269775102], > (-4650650128572424858,-4650260492494258461], > (1886545362164970333,1886646959491599822], > (-4511817721998311568,-4507491187192881115], > (8114903118676615937,8132992506844206601], > (6224957219376301858,6304379125732293904], > (-3460547504877234383,-3459262416082517136], > (-167838948111369123,-141862581573028594], > (481579232521229473,491242114841289497], > (4052464144722307684,4059745901618136723], > (1659668187498418295,1679582585970705122], > (-1118922763210109192,-1093766915505652874], > (7504365235878319341,7526188885210185292], > (-79866884352549492,-77667207866300333], > (8151204058820798561,8154760186218662205], > (-1040398370287131739,-1033770179677543189], > (3767057277953758442,3783780844370292025], > (-6491678058233994892,-6487797181789288329], > (-916868210769480248,-907141794196269524], > (-9005441616028750657,-9002220258513351832], > (8183526518331102304,8186908810225025483], > (-5685737903527826627,-5672136154194382932], > (4976122621177738811,4987871287137312689], > (6051670147160447042,6051686987147911650], > (-1161640137086921883,-1159172734746043158], > (6895951547735922309,6899152466544114890], > (-3357667382515377172,-3356304907368646189], > (-5370953856683870319,-5345971445444542485], > (3824272999898372667,3829315045986248983], > (8132992506844206601,8149858096109302285], > (3975126143101303723,3980729378827590597], > (-956691623200349709,-946602525018301692], > (-82499927325251331,-79866884352549492], > (3952144214544622998,3955602392726495936], > (8154760186218662205,8157079055586089583], > (3840595196718778916,3866458971850198755], > (-1066905024007783341,-1055954824488508260], > (-7252356975874511782,-7246799942440641887], > (-810612946397276081,-792189809286829222], > (4964519403172053705,4970446606512414858], > (-5380038118840759647,-5370953856683870319], > (-3221630728515706463,-3206856875356976885], > (-1193448110686154165,-1161640137086921883], > (-3356304907368646189,-3346460884208327912], > (3466596314109623830,3468144326691723333], > (-9050241313548454460,-9005441616028750657], > (402227699082311580,407458511300218383]]] Validation failed in /FAKE.33.64 > at > org.apache.cassandra.repair.ValidationTask.treesReceived(ValidationTask.java:68) > ~[apache-cassandra-3.0.14.jar:3.0.14] > at > org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:178) > ~[apache-cassandra-3.0.14.jar:3.0.14] > at > org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:486) > ~[apache-cassandra-3.0.14.jar:3.0.14] > at > org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:164) > ~[apache-cassandra-3.0.14.jar:3.0.14] > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) > ~[apache-cassandra-3.0.14.jar:3.0.14] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_102] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > ~[na:1.8.0_102] > ... 4 common frames omitted > WARN [RepairJobTask:2] 2017-09-15 03:00:29,493 RepairJob.java:153 - [repair > #071f81e0-99c2-11e7-91dc-6132f5fe5fb0] YYY sync failed > ERROR [RepairJobTask:2] 2017-09-15 03:00:29,498 RepairSession.java:277 - > [repair #071f81e0-99c2-11e7-91dc-6132f5fe5fb0] Session completed with the > following error > org.apache.cassandra.exceptions.RepairException: [repair > #071f81e0-99c2-11e7-91dc-6132f5fe5fb0 on XXX/YYY, > [(8195393703879512303,8196334842725538685], > (8166975326273137878,8182604850967732931], > (-7246799942440641887,-7227869626613009045], > (-8371707510273823988,-8365977215604569699], > (-141862581573028594,-140310864869418908], > (3732113975108886193,3743105867152786342], > (4998127507903069087,5008922734235607550], > (-5115827291264930140,-5111054924035590372], > (-2475342271852943287,-2447285553369030332], > (-8318606053827235336,-8308721754886697230], > (-5208900659917654871,-5202385837264015269], > (6618737991399272130,6623100721269775102], > (-4650650128572424858,-4650260492494258461], > (1886545362164970333,1886646959491599822], > (-4511817721998311568,-4507491187192881115], > (8114903118676615937,8132992506844206601], > (6224957219376301858,6304379125732293904], > (-3460547504877234383,-3459262416082517136], > (-167838948111369123,-141862581573028594], > (481579232521229473,491242114841289497], > (4052464144722307684,4059745901618136723], > (1659668187498418295,1679582585970705122], > (-1118922763210109192,-1093766915505652874], > (7504365235878319341,7526188885210185292], > (-79866884352549492,-77667207866300333], > (8151204058820798561,8154760186218662205], > (-1040398370287131739,-1033770179677543189], > (3767057277953758442,3783780844370292025], > (-6491678058233994892,-6487797181789288329], > (-916868210769480248,-907141794196269524], > (-9005441616028750657,-9002220258513351832], > (8183526518331102304,8186908810225025483], > (-5685737903527826627,-5672136154194382932], > (4976122621177738811,4987871287137312689], > (6051670147160447042,6051686987147911650], > (-1161640137086921883,-1159172734746043158], > (6895951547735922309,6899152466544114890], > (-3357667382515377172,-3356304907368646189], > (-5370953856683870319,-5345971445444542485], > (3824272999898372667,3829315045986248983], > (8132992506844206601,8149858096109302285], > (3975126143101303723,3980729378827590597], > (-956691623200349709,-946602525018301692], > (-82499927325251331,-79866884352549492], > (3952144214544622998,3955602392726495936], > (8154760186218662205,8157079055586089583], > (3840595196718778916,3866458971850198755], > (-1066905024007783341,-1055954824488508260], > (-7252356975874511782,-7246799942440641887], > (-810612946397276081,-792189809286829222], > (4964519403172053705,4970446606512414858], > (-5380038118840759647,-5370953856683870319], > (-3221630728515706463,-3206856875356976885], > (-1193448110686154165,-1161640137086921883], > (-3356304907368646189,-3346460884208327912], > (3466596314109623830,3468144326691723333], > (-9050241313548454460,-9005441616028750657], > (402227699082311580,407458511300218383]]] Validation failed in /FAKE.33.64 > at > org.apache.cassandra.repair.ValidationTask.treesReceived(ValidationTask.java:68) > ~[apache-cassandra-3.0.14.jar:3.0.14] > at > org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:178) > ~[apache-cassandra-3.0.14.jar:3.0.14] > at > org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:486) > ~[apache-cassandra-3.0.14.jar:3.0.14] > at > org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:164) > ~[apache-cassandra-3.0.14.jar:3.0.14] > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) > ~[apache-cassandra-3.0.14.jar:3.0.14] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_102] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > ~[na:1.8.0_102] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_102] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_102] > at > org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79) > [apache-cassandra-3.0.14.jar:3.0.14] > at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_102] > The contents of this e-mail are intended for the named addressee only. It > contains information that may be confidential. Unless you are the named > addressee or an authorized designee, you may not copy or use it, or disclose > it to anyone else. If you received it in error please notify us immediately > and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) > is a company registered in Linz whose registered office is at 4040 Linz, > Austria, Freistädterstraße 313 > -- > ----------------- > Alexander Dejanovski > France > @alexanderdeja > > Consultant > Apache Cassandra Consulting > http://www.thelastpickle.com > The contents of this e-mail are intended for the named addressee only. It > contains information that may be confidential. Unless you are the named > addressee or an authorized designee, you may not copy or use it, or disclose > it to anyone else. If you received it in error please notify us immediately > and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) > is a company registered in Linz whose registered office is at 4040 Linz, > Austria, Freistädterstraße 313 > -- > ----------------- > Alexander Dejanovski > France > @alexanderdeja > > Consultant > Apache Cassandra Consulting > http://www.thelastpickle.com > The contents of this e-mail are intended for the named addressee only. It > contains information that may be confidential. Unless you are the named > addressee or an authorized designee, you may not copy or use it, or disclose > it to anyone else. If you received it in error please notify us immediately > and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) > is a company registered in Linz whose registered office is at 4040 Linz, > Austria, Freistädterstraße 313 > The contents of this e-mail are intended for the named addressee only. It > contains information that may be confidential. Unless you are the named > addressee or an authorized designee, you may not copy or use it, or disclose > it to anyone else. If you received it in error please notify us immediately > and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) > is a company registered in Linz whose registered office is at 4040 Linz, > Austria, Freistädterstraße 313
