Ruggero Marchei created CASSANDRA-10375:
-------------------------------------------
Summary: nodetool repair fails
Key: CASSANDRA-10375
URL: https://issues.apache.org/jira/browse/CASSANDRA-10375
Project: Cassandra
Issue Type: Bug
Components: Core, Tools
Environment: multi-dc Cassandra 2.2.1 cluster with 6 nodes, 3 per DC
Reporter: Ruggero Marchei
When I'm running a *nodetool repair* it often gets stalled after few seconds:
{code}[2015-09-19 11:12:13,807] Repair session
479ca1c0-5ebf-11e5-9619-3f4813058061 for range
(40511972970986385,59154612555757611] failed with error [repair
#479ca1c0-5ebf-11e5-9619-3f4813058061 on static_assets/assets,
(40511972970986385,59154612555757611]] Validation failed in /10.8.34.113
(progress: 0%)
[2015-09-19 11:12:13,812] Repair session 479cc8d1-5ebf-11e5-9619-3f4813058061
for range (6553929828848556033,6576029219234973671] failed with error [repair
#479cc8d1-5ebf-11e5-9619-3f4813058061 on static_assets/assets,
(6553929828848556033,6576029219234973671]] Validation failed in /10.8.34.113
(progress: 0%)
{code}
At the same time I have this exception on another node:
{code}
ERROR [ValidationExecutor:66] 2015-09-19 11:12:13,825
CompactionManager.java:1070 - Cannot start multiple repair sessions over the
same sstables
ERROR [ValidationExecutor:66] 2015-09-19 11:12:13,826 Validator.java:246 -
Failed creating a merkle tree for [repair #479c2c90-5ebf-11e5-9619-3f4813058061
on static_assets/assets, (-2926621365236563900,-2916361392298929067]],
/10.8.34.113 (see log for details)
ERROR [ValidationExecutor:66] 2015-09-19 11:12:13,826 CassandraDaemon.java:183
- Exception in thread Thread[ValidationExecutor:66,1,main]
java.lang.RuntimeException: Cannot start multiple repair sessions over the same
sstables
at
org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1071)
~[apache-cassandra-2.2.1.jar:2.2.1]
at
org.apache.cassandra.db.compaction.CompactionManager.access$700(CompactionManager.java:94)
~[apache-cassandra-2.2.1.jar:2.2.1]
at
org.apache.cassandra.db.compaction.CompactionManager$10.call(CompactionManager.java:669)
~[apache-cassandra-2.2.1.jar:2.2.1]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
ERROR [ValidationExecutor:66] 2015-09-19 11:12:13,900
CompactionManager.java:1070 - Cannot start multiple repair sessions over the
same sstables
ERROR [ValidationExecutor:66] 2015-09-19 11:12:13,900 Validator.java:246 -
Failed creating a merkle tree for [repair #479c53a1-5ebf-11e5-9619-3f4813058061
on static_assets/assets, (8236929501578674892,8238760988019827700]],
/10.8.34.113 (see log for details)
ERROR [ValidationExecutor:68] 2015-09-19 11:12:13,900
CompactionManager.java:1070 - Cannot start multiple repair sessions over the
same sstables
ERROR [ValidationExecutor:67] 2015-09-19 11:12:13,900
CompactionManager.java:1070 - Cannot start multiple repair sessions over the
same sstables
ERROR [ValidationExecutor:66] 2015-09-19 11:12:13,901 CassandraDaemon.java:183
- Exception in thread Thread[ValidationExecutor:66,1,main]
java.lang.RuntimeException: Cannot start multiple repair sessions over the same
sstables
at
org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1071)
~[apache-cassandra-2.2.1.jar:2.2.1]
at
org.apache.cassandra.db.compaction.CompactionManager.access$700(CompactionManager.java:94)
~[apache-cassandra-2.2.1.jar:2.2.1]
at
org.apache.cassandra.db.compaction.CompactionManager$10.call(CompactionManager.java:669)
~[apache-cassandra-2.2.1.jar:2.2.1]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
ERROR [ValidationExecutor:68] 2015-09-19 11:12:13,901 Validator.java:246 -
Failed creating a merkle tree for [repair #479cc8d1-5ebf-11e5-9619-3f4813058061
on static_assets/assets, (6553929828848556033,6576029219234973671]],
/10.8.34.113 (see log for details)
ERROR [ValidationExecutor:67] 2015-09-19 11:12:13,901 Validator.java:246 -
Failed creating a merkle tree for [repair #479ca1c0-5ebf-11e5-9619-3f4813058061
on static_assets/assets, (40511972970986385,59154612555757611]], /10.8.34.113
(see log for details)
ERROR [ValidationExecutor:68] 2015-09-19 11:12:13,901 CassandraDaemon.java:183
- Exception in thread Thread[ValidationExecutor:68,1,main]
java.lang.RuntimeException: Cannot start multiple repair sessions over the same
sstables
at
org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1071)
~[apache-cassandra-2.2.1.jar:2.2.1]
at
org.apache.cassandra.db.compaction.CompactionManager.access$700(CompactionManager.java:94)
~[apache-cassandra-2.2.1.jar:2.2.1]
at
org.apache.cassandra.db.compaction.CompactionManager$10.call(CompactionManager.java:669)
~[apache-cassandra-2.2.1.jar:2.2.1]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
ERROR [ValidationExecutor:67] 2015-09-19 11:12:13,901 CassandraDaemon.java:183
- Exception in thread Thread[ValidationExecutor:67,1,main]
java.lang.RuntimeException: Cannot start multiple repair sessions over the same
sstables
at
org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1071)
~[apache-cassandra-2.2.1.jar:2.2.1]
at
org.apache.cassandra.db.compaction.CompactionManager.access$700(CompactionManager.java:94)
~[apache-cassandra-2.2.1.jar:2.2.1]
at
org.apache.cassandra.db.compaction.CompactionManager$10.call(CompactionManager.java:669)
~[apache-cassandra-2.2.1.jar:2.2.1]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
{code}
And on the remaining nodes I have something similar to this:
{code}
ERROR [Reference-Reaper:1] 2015-09-19 11:12:14,147 Ref.java:187 - LEAK
DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@320d7d92) to class
org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@780334113:/var/lib/cassandra/data/static_assets/assets-ceb62f1056df11e5865db530abf562fa/la-2447-big
was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2015-09-19 11:12:14,147 Ref.java:187 - LEAK
DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@972fce9)
to class
org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@389940435:/var/lib/cassandra/data/static_assets/assets-ceb62f1056df11e5865db530abf562fa/la-6-big
was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2015-09-19 11:12:14,147 Ref.java:187 - LEAK
DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@5a2a0617) to class
org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@156533865:/var/lib/cassandra/data/static_assets/assets-ceb62f1056df11e5865db530abf562fa/la-4-big
was not released before the reference was garbage collected
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
