[
https://issues.apache.org/jira/browse/CASSANDRA-14060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16276191#comment-16276191
]
Jay Zhuang commented on CASSANDRA-14060:
----------------------------------------
bq. when you say most of your corrupt sstables aren't disk issues - to what do
you attribute them?
We don't know the root cause, but it should be software bug. We saw the
{{CorruptSSTableException}} in several nodes at the same time, which causes
multiple nodes down. So I don't think it's a disk issue.
For example just yesterday, 10 nodes were down at the same time (the cluster
has 2 DCs, each DC has 15 nodes, all down nodes are in the same DC, we use
{{local_quorum}}), here is the call stack:
{noformat}
ERROR [SharedPool-Worker-7] 2017-12-02 01:06:51,135
JVMStabilityInspector.java:140 - JVM state determined to be unstable. Exiting
forcefully due to:
org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
/var/data/[keyspace]/[table]/mc-1-big-Data.db
at
org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:356)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:220)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:33)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.transform.UnfilteredRows.isEmpty(UnfilteredRows.java:58)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:67)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:26)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:96)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:296)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:145)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:138)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:134)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:321)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:47)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_121]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
Caused by: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~[na:1.8.0_121]
at java.util.ArrayList.get(ArrayList.java:429) ~[na:1.8.0_121]
at
org.apache.cassandra.db.ClusteringPrefix$Deserializer.deserializeOne(ClusteringPrefix.java:482)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.ClusteringPrefix$Deserializer.deserializeAll(ClusteringPrefix.java:488)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.ClusteringPrefix$Deserializer.deserializeNextClustering(ClusteringPrefix.java:504)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.readNext(UnfilteredDeserializer.java:208)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.computeNext(SSTableIterator.java:143)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:172)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:343)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
... 26 common frames omitted
{noformat}
The similar problem was happened before for another cluster:
{noformat}
ERROR [SharedPool-Worker-4] 2017-11-14 20:14:18,381
JVMStabilityInspector.java:140 - JVM state determined to be unstable. Exiting
forcefully due to:
org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
/var/data/[keyspace]/[table]/mc-128336-big-Data.db
at
org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:356)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.filter.ClusteringIndexNamesFilter$1.hasNext(ClusteringIndexNamesFilter.java:145)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.partitions.AbstractBTreePartition.build(AbstractBTreePartition.java:329)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.partitions.ImmutableBTreePartition.create(ImmutableBTreePartition.java:111)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.partitions.ImmutableBTreePartition.create(ImmutableBTreePartition.java:94)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.SinglePartitionReadCommand.add(SinglePartitionReadCommand.java:816)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndSSTablesInTimestampOrder(SinglePartitionReadCommand.java:770)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:520)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:497)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:359)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:395)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1835)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2520)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_121]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
Caused by: java.lang.ArrayIndexOutOfBoundsException: 94
at
org.apache.cassandra.db.ClusteringPrefix$Deserializer.prepare(ClusteringPrefix.java:425)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.prepareNext(UnfilteredDeserializer.java:170)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.hasNext(UnfilteredDeserializer.java:151)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.computeNext(SSTableIterator.java:140)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:172)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
at
org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:343)
~[apache-cassandra-3.0.14.1.jar:3.0.14.1]
... 16 common frames omitted
{noformat}
The similar problem has happened even before (6 months ago), but at that time,
we were on {{3.0.10}}, so it may be because of CASSANDRA-13004. I will create a
new ticket if there are more findings, any suggestion on debugging and root
cause the corruption issue would be appreciated. Thanks.
This patch would make {{disk_failure_policy}} behave consistent with {{stop}}
and {{die}}.
> Separate CorruptSSTableException and FSError handling policies
> --------------------------------------------------------------
>
> Key: CASSANDRA-14060
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14060
> Project: Cassandra
> Issue Type: Improvement
> Components: Configuration
> Reporter: Jay Zhuang
> Assignee: Jay Zhuang
> Priority: Minor
>
> Currently, if
> [{{disk_failure_policy}}|https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L230]
> is set to {{stop}} (default), StorageService will shutdown for {{FSError}},
> but not {{CorruptSSTableException}}
> [DefaultFSErrorHandler.java:40|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/DefaultFSErrorHandler.java#L40].
> But when we use policy: {{die}}, it has different behave, JVM will be killed
> for both {{FSError}} and {{CorruptSSTableException}}
> [JVMStabilityInspector.java:63|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/utils/JVMStabilityInspector.java#L63]:
> ||{{disk_failure_policy}}|| hit {{FSError}} Exception || hit
> {{CorruptSSTableException}} ||
> |{{stop}}| (/) stop | (x) not stop |
> |{{die}}| (/) die | (/) die |
> We saw {{CorruptSSTableException}} from time to time in our production, but
> mostly it's *not* because of a disk issue. So I would suggest having a
> separate policy for CorruptSSTable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
