hi Hannu,

I think some of the repairs are hanging there. please restart all the nodes
in the  cluster and start the repair


Thanks
Nanda

On Wed, Jan 3, 2018 at 9:35 AM, Hannu Kröger <hkro...@gmail.com> wrote:

> Additional notes:
>
> 1) If I run the repair just on those tables, it works fine
> 2) Those tables are empty
>
> Hannu
>
> > On 3 Jan 2018, at 18:23, Hannu Kröger <hkro...@gmail.com> wrote:
> >
> > Hello,
> >
> > Situation is as follows:
> >
> > Repair was started on node X on this keyspace with —full —pr. Repair
> fails on node Y.
> >
> > Node Y has debug logging on (DEBUG on org.apache.cassandra) and I’m
> looking at the debug.log. I see following messages related to this repair
> request:
> >
> > -----------
> > DEBUG [AntiEntropyStage:1] 2018-01-02 17:52:12,530
> RepairMessageVerbHandler.java:114 - Validating 
> ValidationRequest{gcBefore=1511473932}
> org.apache.cassandra.repair.messages.ValidationRequest@5a17430c
> > DEBUG [ValidationExecutor:4] 2018-01-02 17:52:12,531
> StorageService.java:3321 - Forcing flush on keyspace mykeyspace, CF mytable
> > DEBUG [MemtablePostFlush:54] 2018-01-02 17:52:12,531
> ColumnFamilyStore.java:954 - forceFlush requested but everything is clean
> in mytable
> > ERROR [ValidationExecutor:4] 2018-01-02 17:52:12,532 Validator.java:268
> - Failed creating a merkle tree for [repair 
> #1df000a0-effa-11e7-8361-b7c9edfbfc33
> on mykeyspace/mytable, [(6917529027641081856,-9223372036854775808]]], /
> 123.123.123.123 (see log for details)
> > -----------
> >
> > then the same about another table and after that which indicates that
> repair “master” has told to abort basically, right?
> >
> > -----------
> > DEBUG [AntiEntropyStage:1] 2018-01-02 17:52:12,563
> RepairMessageVerbHandler.java:142 - Got anticompaction request
> AnticompactionRequest{parentRepairSession=1de949e0-effa-11e7-8361-b7c9edfbfc33}
> org.apache.cassandra.repair.messages.AnticompactionRequest@5dc8be
> > ea
> > ERROR [AntiEntropyStage:1] 2018-01-02 17:52:12,563
> RepairMessageVerbHandler.java:168 - Got error, removing parent repair
> session
> > ERROR [AntiEntropyStage:1] 2018-01-02 17:52:12,564
> CassandraDaemon.java:228 - Exception in thread Thread[AntiEntropyStage:1,5,
> main]
> > java.lang.RuntimeException: java.lang.RuntimeException: Parent repair
> session with id = 1de949e0-effa-11e7-8361-b7c9edfbfc33 has failed.
> >        at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(
> RepairMessageVerbHandler.java:171) ~[apache-cassandra-3.11.0.jar:3.11.0]
> >        at 
> > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> >        at 
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_111]
> >        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_111]
> >        at 
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_111]
> >        at 
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_111]
> >        at org.apache.cassandra.concurrent.NamedThreadFactory.
> lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
> [apache-cassandra-3.11.0.jar:3.11.0]
> >        at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_111]
> > Caused by: java.lang.RuntimeException: Parent repair session with id =
> 1de949e0-effa-11e7-8361-b7c9edfbfc33 has failed.
> >        at org.apache.cassandra.service.ActiveRepairService.
> getParentRepairSession(ActiveRepairService.java:409)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> >        at org.apache.cassandra.service.ActiveRepairService.
> doAntiCompaction(ActiveRepairService.java:444)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> >        at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(
> RepairMessageVerbHandler.java:143) ~[apache-cassandra-3.11.0.jar:3.11.0]
> >        ... 7 common frames omitted
> > -----------
> >
> > But that is almost all in the log and I don’t really see what the
> original problem here is.
> >
> > Cassandra flushes the table to start building merkle tree and on next
> millisecond it already fails the repair but without proper exception or
> error logging about the problem.
> >
> > Cassandra version is the 3.11.0.
> >
> > Any ideas?
> >
> > Cheers,
> > Hannu
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>


-- 
Thanks & Regards,
Nanda Kishore

Reply via email to