[
https://issues.apache.org/jira/browse/CASSANDRA-9133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philip Thompson updated CASSANDRA-9133:
---------------------------------------
Description:
Cassandra V:2.01.4
We are running repairs on nodes to run scrub on them later to get around issue
(CASSANDRA-9126). While running nodetool repair on nodes, getting the
validationexception error. We are seeing bunch of these validation exceptions
on lot of nodes. Any help is appreciated?
{code}
java.lang.RuntimeException: org.apache.cassandra.exceptions.RepairException:
[repair #b3ee6050-dd7a-11e4-adac-172f3292c10d on ....,
(-1729382256910270568,-1537228672809129408]] Validation failed in /........
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.cassandra.exceptions.RepairException: [repair
#b3ee6050-dd7a-11e4-adac-172f3292c10d on ....,
(-1729382256910270568,-1537228672809129408]] Validation failed in .....
at
org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:161)
at
org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:228)
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:59)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
{code}
Also, seeing the following exceptions:{code}
ERROR [ValidationExecutor:33] 2015-04-08 00:16:45,103 Validator.java (line 242)
Failed creating a merkle tree for [repair #bf8d13c0-dd7a-11e4-80ff-810270bd0450
on ......, (-8262604116349070008,-8070450532247928848]], /.... (see log for
details)
ERROR [ValidationExecutor:33] 2015-04-08 00:16:45,103 CassandraDaemon.java
(line 258) Exception in thread Thread[ValidationExecutor:33,1,main]
java.lang.AssertionError: row DecoratedKey(-8232281172744762721,
354036383831303736375f3330335f31355f474d4543454f) received out of order wrt
DecoratedKey(-8070630157486006754,
3540383834383134305f3336315f31355f474d4543454f)
at org.apache.cassandra.repair.Validator.add(Validator.java:136)
at
org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:806)
at
org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:63)
at
org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:404)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745) {code}
was:
Cassandra V:2.01.4
We are running repairs on nodes to run scrub on them later to get around issue
(CASSANDRA-9126). While running nodetool repair on nodes, getting the
validationexception error. We are seeing bunch of these validation exceptions
on lot of nodes. Any help is appreciated?
java.lang.RuntimeException: org.apache.cassandra.exceptions.RepairException:
[repair #b3ee6050-dd7a-11e4-adac-172f3292c10d on ....,
(-1729382256910270568,-1537228672809129408]] Validation failed in /........
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.cassandra.exceptions.RepairException: [repair
#b3ee6050-dd7a-11e4-adac-172f3292c10d on ....,
(-1729382256910270568,-1537228672809129408]] Validation failed in .....
at
org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:161)
at
org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:228)
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:59)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
Also, seeing the following exceptions:
ERROR [ValidationExecutor:33] 2015-04-08 00:16:45,103 Validator.java (line 242)
Failed creating a merkle tree for [repair #bf8d13c0-dd7a-11e4-80ff-810270bd0450
on ......, (-8262604116349070008,-8070450532247928848]], /.... (see log for
details)
ERROR [ValidationExecutor:33] 2015-04-08 00:16:45,103 CassandraDaemon.java
(line 258) Exception in thread Thread[ValidationExecutor:33,1,main]
java.lang.AssertionError: row DecoratedKey(-8232281172744762721,
354036383831303736375f3330335f31355f474d4543454f) received out of order wrt
DecoratedKey(-8070630157486006754,
3540383834383134305f3336315f31355f474d4543454f)
at org.apache.cassandra.repair.Validator.add(Validator.java:136)
at
org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:806)
at
org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:63)
at
org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:404)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
> Getting validationexception while repairing nodes
> -------------------------------------------------
>
> Key: CASSANDRA-9133
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9133
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: srinivasu gottipati
> Priority: Blocker
> Fix For: 2.0.15
>
>
> Cassandra V:2.01.4
> We are running repairs on nodes to run scrub on them later to get around
> issue (CASSANDRA-9126). While running nodetool repair on nodes, getting the
> validationexception error. We are seeing bunch of these validation exceptions
> on lot of nodes. Any help is appreciated?
> {code}
> java.lang.RuntimeException: org.apache.cassandra.exceptions.RepairException:
> [repair #b3ee6050-dd7a-11e4-adac-172f3292c10d on ....,
> (-1729382256910270568,-1537228672809129408]] Validation failed in /........
> at com.google.common.base.Throwables.propagate(Throwables.java:160)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.cassandra.exceptions.RepairException: [repair
> #b3ee6050-dd7a-11e4-adac-172f3292c10d on ....,
> (-1729382256910270568,-1537228672809129408]] Validation failed in .....
> at
> org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:161)
> at
> org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:228)
> at
> org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:59)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
> {code}
> Also, seeing the following exceptions:{code}
> ERROR [ValidationExecutor:33] 2015-04-08 00:16:45,103 Validator.java (line
> 242) Failed creating a merkle tree for [repair
> #bf8d13c0-dd7a-11e4-80ff-810270bd0450 on ......,
> (-8262604116349070008,-8070450532247928848]], /.... (see log for details)
> ERROR [ValidationExecutor:33] 2015-04-08 00:16:45,103 CassandraDaemon.java
> (line 258) Exception in thread Thread[ValidationExecutor:33,1,main]
> java.lang.AssertionError: row DecoratedKey(-8232281172744762721,
> 354036383831303736375f3330335f31355f474d4543454f) received out of order wrt
> DecoratedKey(-8070630157486006754,
> 3540383834383134305f3336315f31355f474d4543454f)
> at org.apache.cassandra.repair.Validator.add(Validator.java:136)
> at
> org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:806)
> at
> org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:63)
> at
> org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:404)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745) {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
