Christian Schjolberg created CASSANDRA-11014:
------------------------------------------------
Summary: Repair fails with "not enough bytes"
Key: CASSANDRA-11014
URL: https://issues.apache.org/jira/browse/CASSANDRA-11014
Project: Cassandra
Issue Type: Bug
Environment: 3 node cluster, debian jessie, cassandra 2.2.4
Reporter: Christian Schjolberg
Priority: Blocker
Attachments: scrub-output.txt
After upgrading to 2.2.4, nodetool repair fails every time with the error
message "Not enough bytes". It appears no data is being repaired at all. Here's
some output:
-@cas01:~$ nodetool repair
[2016-01-14 12:00:16,590] Starting repair command #1, repairing keyspace
adsquare with repair options (parallelism: parallel, primary range: false,
incremental: true, job threads: 1, ColumnFamilies: [], dataCenters: [], hosts:
[], # of ranges: 768)
[2016-01-14 12:00:21,935] Repair session 61174f80-bab6-11e5-9fa9-11175757c857
for range (-3942884673882176939,-3929110923969659376] failed with error [repair
#61174f80-bab6-11e5-9fa9-11175757c857 on adsquare/device_lookup,
(-3942884673882176939,-3929110923969659376]] Validation failed in /10.10.100.61
(progress: 0%)
The system.log on the host in question shows
ERROR [ValidationExecutor:2] 2016-01-14 09:58:19,935 CassandraDaemon.java:185 -
Exception in thread Thread[ValidationExecutor:2,1,main]
java.lang.IllegalArgumentException: Not enough bytes
at
org.apache.cassandra.db.composites.AbstractCType.checkRemaining(AbstractCType.java:362)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.db.composites.AbstractCompoundCellNameType.fromByteBuffer(AbstractCompoundCellNameType.java:98)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:381)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:365)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:75)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:52)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:46)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
~[guava-16.0.jar:na]
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
~[guava-16.0.jar:na]
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:169)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:202)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
~[guava-16.0.jar:na]
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
~[guava-16.0.jar:na]
at
com.google.common.collect.Iterators$7.computeNext(Iterators.java:645)
~[guava-16.0.jar:na]
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
~[guava-16.0.jar:na]
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
~[guava-16.0.jar:na]
at
org.apache.cassandra.db.compaction.LazilyCompactedRow.update(LazilyCompactedRow.java:172)
~[apache-cassandra-2.2.4.jar:2.2.4]
at org.apache.cassandra.repair.Validator.rowHash(Validator.java:194)
~[apache-cassandra-2.2.4.jar:2.2.4]
at org.apache.cassandra.repair.Validator.add(Validator.java:143)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1118)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.db.compaction.CompactionManager.access$700(CompactionManager.java:73)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.db.compaction.CompactionManager$10.call(CompactionManager.java:671)
~[apache-cassandra-2.2.4.jar:2.2.4]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_66]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_66]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_66]
Thinking that there might be some corruption I ran nodetool scrub, which fills
the log with statements like the one in the attachment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
