Jason Kania created CASSANDRA-12762:
---------------------------------------
Summary: Cassandra 3.0.9 Fails both compact and repair without
even debug logs
Key: CASSANDRA-12762
URL: https://issues.apache.org/jira/browse/CASSANDRA-12762
Project: Cassandra
Issue Type: Bug
Environment: Debian Jessie current
Reporter: Jason Kania
Priority: Critical
After upgrading from 3.0.7 to 3.0.9, the following exception occurs when trying
to run compaction (previous to the upgrade compaction worked fine):
error:
(/home/circuitwatch/cassandra/data/circuitwatch/edgeTransitionByCircuitId-f5d33310024b11e5bb310d2316086bf7/mb-12063-big-Data.db):
corruption detected, chunk at 345885546 of length 62024.
-- StackTrace --
org.apache.cassandra.io.compress.CorruptBlockException:
(/home/circuitwatch/cassandra/data/circuitwatch/edgeTransitionByCircuitId-f5d33310024b11e5bb310d2316086bf7/mb-12063-big-Data.db):
corruption detected, chunk at 345885546 of length 62024.
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap(CompressedRandomAccessReader.java:202)
at
org.apache.cassandra.io.util.RandomAccessReader.reBuffer(RandomAccessReader.java:111)
at
org.apache.cassandra.io.util.RebufferingInputStream.read(RebufferingInputStream.java:88)
at
org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:66)
at
org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
at
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:404)
at
org.apache.cassandra.db.marshal.AbstractType.readValue(AbstractType.java:406)
at
org.apache.cassandra.db.rows.BufferCell$Serializer.deserialize(BufferCell.java:302)
at
org.apache.cassandra.db.rows.UnfilteredSerializer.readSimpleColumn(UnfilteredSerializer.java:476)
at
org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:454)
at
org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:377)
at
org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:87)
at
org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:65)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.doCompute(SSTableIdentityIterator.java:123)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
at
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:369)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:189)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:158)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:509)
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:369)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129)
at
org.apache.cassandra.db.ColumnIndex$Builder.build(ColumnIndex.java:111)
at
org.apache.cassandra.db.ColumnIndex.writeAndBuildIndex(ColumnIndex.java:52)
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:149)
at
org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125)
at
org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:57)
at
org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:183)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
at
org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:606)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Seeing that there was a corruption, I ran nodetool repair, but it also fails in
a spectacularly poor manner giving no indication of the problem:
error: Repair job has failed with the error message: [2016-10-08 01:25:22,431]
Some repair failed
-- StackTrace --
java.lang.RuntimeException: Repair job has failed with the error message:
[2016-10-08 01:25:22,431] Some repair failed
at
org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:115)
at
org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:583)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:533)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108)
The debug logs contain nothing and are just cut at the time of the error.
Please update the errors in the repair case to communicate something meaningful
as these sorts of errors mean the user is stranded.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
