[
https://issues.apache.org/jira/browse/CASSANDRA-8120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14171640#comment-14171640
]
Yuki Morishita commented on CASSANDRA-8120:
-------------------------------------------
What option did you used for repair?
And what is your compaction strategy on censored/censored?
> Failed creating a merkle tree during repair.
> --------------------------------------------
>
> Key: CASSANDRA-8120
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8120
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Ubuntu 12.04.5 LTS. 3.2.0-69-virtual.
> Reporter: jason milliron
>
> Seeing this while running nodetool repair on several nodes of our new (not an
> upgrade) 2.1.0 cluster. Multiple keyspaces. Doesn't hang the repair job.
> Storage is SSD ephemeral (EC2). No storage errors in syslog.
> ERROR [ValidationExecutor:39] 2014-10-13 22:20:51,140 Validator.java:242 -
> Failed creating a merkle tree for [repair
> #1523dfa0-5327-11e4-925c-65a001aa6d80 on censored/censored,
> (-4468426774357114003,-4433737571269222163]], /10.37.666.175 (see log for
> details)
> INFO [AntiEntropySessions:37] 2014-10-13 22:20:51,141 RepairSession.java:260
> - [repair #308e2750-5327-11e4-925c-65a001aa6d80] new session: will sync
> ip-10-37-666-175.ec2.internal/10.37.666.175, /10.167.666.137, /10.164.666.81
> on range (6870883176570587821,6871062738154464368] for censored.[censored]
> INFO [AntiEntropyStage:1] 2014-10-13 22:20:51,142 ColumnFamilyStore.java:856
> - Enqueuing flush of censored: 3227760 (0%) on-heap, 0 (0%) off-heap
> INFO [MemtableFlushWriter:2295] 2014-10-13 22:20:51,145 Memtable.java:326 -
> Writing Memtable-censored@2052537453(380369 serialized bytes, 8235 ops, 0%/0%
> of on/off-heap limit)
> ERROR [AntiEntropySessions:34] 2014-10-13 22:20:51,148 RepairSession.java:303
> - [repair #1523dfa0-5327-11e4-925c-65a001aa6d80] session completed with the
> following error
> org.apache.cassandra.exceptions.RepairException: [repair
> #1523dfa0-5327-11e4-925c-65a001aa6d80 on censored/censored,
> (-4468426774357114003,-4433737571269222163]] Validation failed in
> /10.37.666.175
> at
> org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:166)
> ~[apache-cassandra-2.1.0.jar:2.1.0]
> at
> org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:384)
> ~[apache-cassandra-2.1.0.jar:2.1.0]
> at
> org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:123)
> ~[apache-cassandra-2.1.0.jar:2.1.0]
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
> ~[apache-cassandra-2.1.0.jar:2.1.0]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_67]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_67]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
> ERROR [ValidationExecutor:39] 2014-10-13 22:20:51,148
> CassandraDaemon.java:166 - Exception in thread
> Thread[ValidationExecutor:39,1,main]
> java.lang.AssertionError: row DecoratedKey(-4468391015221140553,
> 00040000000100000800000000005abc2d00) received out of order wrt
> DecoratedKey(-4436255891196926644, 0004000000010000080000000000617f4400)
> at org.apache.cassandra.repair.Validator.add(Validator.java:136)
> ~[apache-cassandra-2.1.0.jar:2.1.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:916)
> ~[apache-cassandra-2.1.0.jar:2.1.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:100)
> ~[apache-cassandra-2.1.0.jar:2.1.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:560)
> ~[apache-cassandra-2.1.0.jar:2.1.0]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[na:1.7.0_67]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[na:1.7.0_67]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_67]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
> ERROR [AntiEntropySessions:34] 2014-10-13 22:20:51,149
> CassandraDaemon.java:166 - Exception in thread
> Thread[AntiEntropySessions:34,5,RMI Runtime]
> java.lang.RuntimeException: org.apache.cassandra.exceptions.RepairException:
> [repair #1523dfa0-5327-11e4-925c-65a001aa6d80 on censored/censored,
> (-4468426774357114003,-4433737571269222163]] Validation failed in
> /10.37.666.175
> at com.google.common.base.Throwables.propagate(Throwables.java:160)
> ~[guava-16.0.jar:na]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
> ~[apache-cassandra-2.1.0.jar:2.1.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[na:1.7.0_67]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[na:1.7.0_67]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[na:1.7.0_67]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_67]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
> Caused by: org.apache.cassandra.exceptions.RepairException: [repair
> #1523dfa0-5327-11e4-925c-65a001aa6d80 on censored/censored,
> (-4468426774357114003,-4433737571269222163]] Validation failed in
> /10.37.666.175
> at
> org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:166)
> ~[apache-cassandra-2.1.0.jar:2.1.0]
> at
> org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:384)
> ~[apache-cassandra-2.1.0.jar:2.1.0]
> at
> org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:123)
> ~[apache-cassandra-2.1.0.jar:2.1.0]
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
> ~[apache-cassandra-2.1.0.jar:2.1.0]
> ... 3 common frames omitted
> INFO [MemtableFlushWriter:2295] 2014-10-13 22:20:51,180 Memtable.java:360 -
> Completed flushing
> /ephem/data/censored/censored-f69e4e9049ac11e4bc96bfea6c7ad14a/censored-censored-ka-3185-Data.db
> (126124 bytes) for commitlog position
> ReplayPosition(segmentId=1412194655943, position=28904247)
> INFO [RepairJobTask:3] 2014-10-13 22:20:51,259 RepairJob.java:145 - [repair
> #308e2750-5327-11e4-925c-65a001aa6d80] requesting merkle trees for censored
> (to [/10.167.666.137, /10.164.666.81,
> ip-10-37-666-175.ec2.internal/10.37.666.175])
> INFO [AntiEntropyStage:1] 2014-10-13 22:20:51,351 RepairSession.java:171 -
> [repair #308e2750-5327-11e4-925c-65a001aa6d80] Received merkle tree for
> censored from /10.167.666.137
> INFO [AntiEntropyStage:1] 2014-10-13 22:20:51,446 RepairSession.java:171 -
> [repair #308e2750-5327-11e4-925c-65a001aa6d80] Received merkle tree for
> censored from /10.164.666.81
> INFO [ValidationExecutor:39] 2014-10-13 22:20:51,452 SSTableReader.java:383
> - Opening
> /ephem/data/censored/censored-f69e4e9049ac11e4bc96bfea6c7ad14a/snapshots/308e2750-5327-11e4-925c-65a001aa6d80/censored-censored-ka-3173
> (87123 bytes)
> INFO [ValidationExecutor:39] 2014-10-13 22:20:51,453 SSTableReader.java:383
> - Opening
> /ephem/data/censored/censored-f69e4e9049ac11e4bc96bfea6c7ad14a/snapshots/308e2750-5327-11e4-925c-65a001aa6d80/censored-censored-ka-3181
> (7010 bytes)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
