[
https://issues.apache.org/jira/browse/CASSANDRA-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14877084#comment-14877084
]
Ruggero Marchei commented on CASSANDRA-10375:
---------------------------------------------
I think I hit [#10057|CASSANDRA-10057]. I'm closing this one as duplicated
> nodetool repair fails
> ---------------------
>
> Key: CASSANDRA-10375
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10375
> Project: Cassandra
> Issue Type: Bug
> Components: Core, Tools
> Environment: multi-dc Cassandra 2.2.1 cluster with 6 nodes, 3 per DC
> Reporter: Ruggero Marchei
> Fix For: 2.2.2
>
>
> When I'm running a *nodetool repair* it gets stalled after few seconds:
> {code}[2015-09-19 11:12:13,807] Repair session
> 479ca1c0-5ebf-11e5-9619-3f4813058061 for range
> (40511972970986385,59154612555757611] failed with error [repair
> #479ca1c0-5ebf-11e5-9619-3f4813058061 on static_assets/assets,
> (40511972970986385,59154612555757611]] Validation failed in /10.8.34.113
> (progress: 0%)
> [2015-09-19 11:12:13,812] Repair session 479cc8d1-5ebf-11e5-9619-3f4813058061
> for range (6553929828848556033,6576029219234973671] failed with error [repair
> #479cc8d1-5ebf-11e5-9619-3f4813058061 on static_assets/assets,
> (6553929828848556033,6576029219234973671]] Validation failed in /10.8.34.113
> (progress: 0%)
> {code}
> At the same time I have this exception on another node (I'm not running
> multiple repairs):
> {code}
> ERROR [ValidationExecutor:66] 2015-09-19 11:12:13,825
> CompactionManager.java:1070 - Cannot start multiple repair sessions over the
> same sstables
> ERROR [ValidationExecutor:66] 2015-09-19 11:12:13,826 Validator.java:246 -
> Failed creating a merkle tree for [repair
> #479c2c90-5ebf-11e5-9619-3f4813058061 on static_assets/assets,
> (-2926621365236563900,-2916361392298929067]], /10.8.34.113 (see log for
> details)
> ERROR [ValidationExecutor:66] 2015-09-19 11:12:13,826
> CassandraDaemon.java:183 - Exception in thread
> Thread[ValidationExecutor:66,1,main]
> java.lang.RuntimeException: Cannot start multiple repair sessions over the
> same sstables
> at
> org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1071)
> ~[apache-cassandra-2.2.1.jar:2.2.1]
> at
> org.apache.cassandra.db.compaction.CompactionManager.access$700(CompactionManager.java:94)
> ~[apache-cassandra-2.2.1.jar:2.2.1]
> at
> org.apache.cassandra.db.compaction.CompactionManager$10.call(CompactionManager.java:669)
> ~[apache-cassandra-2.2.1.jar:2.2.1]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_45]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> ERROR [ValidationExecutor:66] 2015-09-19 11:12:13,900
> CompactionManager.java:1070 - Cannot start multiple repair sessions over the
> same sstables
> ERROR [ValidationExecutor:66] 2015-09-19 11:12:13,900 Validator.java:246 -
> Failed creating a merkle tree for [repair
> #479c53a1-5ebf-11e5-9619-3f4813058061 on static_assets/assets,
> (8236929501578674892,8238760988019827700]], /10.8.34.113 (see log for details)
> ERROR [ValidationExecutor:68] 2015-09-19 11:12:13,900
> CompactionManager.java:1070 - Cannot start multiple repair sessions over the
> same sstables
> ERROR [ValidationExecutor:67] 2015-09-19 11:12:13,900
> CompactionManager.java:1070 - Cannot start multiple repair sessions over the
> same sstables
> ERROR [ValidationExecutor:66] 2015-09-19 11:12:13,901
> CassandraDaemon.java:183 - Exception in thread
> Thread[ValidationExecutor:66,1,main]
> java.lang.RuntimeException: Cannot start multiple repair sessions over the
> same sstables
> at
> org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1071)
> ~[apache-cassandra-2.2.1.jar:2.2.1]
> at
> org.apache.cassandra.db.compaction.CompactionManager.access$700(CompactionManager.java:94)
> ~[apache-cassandra-2.2.1.jar:2.2.1]
> at
> org.apache.cassandra.db.compaction.CompactionManager$10.call(CompactionManager.java:669)
> ~[apache-cassandra-2.2.1.jar:2.2.1]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_45]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> ERROR [ValidationExecutor:68] 2015-09-19 11:12:13,901 Validator.java:246 -
> Failed creating a merkle tree for [repair
> #479cc8d1-5ebf-11e5-9619-3f4813058061 on static_assets/assets,
> (6553929828848556033,6576029219234973671]], /10.8.34.113 (see log for details)
> ERROR [ValidationExecutor:67] 2015-09-19 11:12:13,901 Validator.java:246 -
> Failed creating a merkle tree for [repair
> #479ca1c0-5ebf-11e5-9619-3f4813058061 on static_assets/assets,
> (40511972970986385,59154612555757611]], /10.8.34.113 (see log for details)
> ERROR [ValidationExecutor:68] 2015-09-19 11:12:13,901
> CassandraDaemon.java:183 - Exception in thread
> Thread[ValidationExecutor:68,1,main]
> java.lang.RuntimeException: Cannot start multiple repair sessions over the
> same sstables
> at
> org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1071)
> ~[apache-cassandra-2.2.1.jar:2.2.1]
> at
> org.apache.cassandra.db.compaction.CompactionManager.access$700(CompactionManager.java:94)
> ~[apache-cassandra-2.2.1.jar:2.2.1]
> at
> org.apache.cassandra.db.compaction.CompactionManager$10.call(CompactionManager.java:669)
> ~[apache-cassandra-2.2.1.jar:2.2.1]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_45]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> ERROR [ValidationExecutor:67] 2015-09-19 11:12:13,901
> CassandraDaemon.java:183 - Exception in thread
> Thread[ValidationExecutor:67,1,main]
> java.lang.RuntimeException: Cannot start multiple repair sessions over the
> same sstables
> at
> org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1071)
> ~[apache-cassandra-2.2.1.jar:2.2.1]
> at
> org.apache.cassandra.db.compaction.CompactionManager.access$700(CompactionManager.java:94)
> ~[apache-cassandra-2.2.1.jar:2.2.1]
> at
> org.apache.cassandra.db.compaction.CompactionManager$10.call(CompactionManager.java:669)
> ~[apache-cassandra-2.2.1.jar:2.2.1]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_45]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> {code}
> And on the remaining nodes I have something similar to this:
> {code}
> ERROR [Reference-Reaper:1] 2015-09-19 11:12:14,147 Ref.java:187 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@320d7d92) to class
> org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@780334113:/var/lib/cassandra/data/static_assets/assets-ceb62f1056df11e5865db530abf562fa/la-2447-big
> was not released before the reference was garbage collected
> ERROR [Reference-Reaper:1] 2015-09-19 11:12:14,147 Ref.java:187 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@972fce9) to class
> org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@389940435:/var/lib/cassandra/data/static_assets/assets-ceb62f1056df11e5865db530abf562fa/la-6-big
> was not released before the reference was garbage collected
> ERROR [Reference-Reaper:1] 2015-09-19 11:12:14,147 Ref.java:187 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@5a2a0617) to class
> org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@156533865:/var/lib/cassandra/data/static_assets/assets-ceb62f1056df11e5865db530abf562fa/la-4-big
> was not released before the reference was garbage collected
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
