[
https://issues.apache.org/jira/browse/CASSANDRA-8337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14216848#comment-14216848
]
Alexander Sterligov commented on CASSANDRA-8337:
------------------------------------------------
Another variant of exception:
{quote}
ERROR [ValidationExecutor:2] 2014-11-19 01:44:14,664 CassandraDaemon.java:153 -
Exception in thread Thread[ValidationExecutor:2,1,main]
org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFException
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:122)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:146)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:125)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:99)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
~[guava-16.0.jar:na]
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
~[guava-16.0.jar:na]
at
com.google.common.collect.Iterators$7.computeNext(Iterators.java:645)
~[guava-16.0.jar:na]
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
~[guava-16.0.jar:na]
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
~[guava-16.0.jar:na]
at
org.apache.cassandra.db.compaction.LazilyCompactedRow.update(LazilyCompactedRow.java:160)
~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.repair.Validator.rowHash(Validator.java:188)
~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.repair.Validator.add(Validator.java:141)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:931)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:97)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:557)
~[apache-cassandra-2.1.2.jar:2.1.2]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_51]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_51]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_51]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
Caused by: java.io.EOFException: null
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:446)
~[na:1.7.0_51]
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:424)
~[na:1.7.0_51]
at
org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:351)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:348)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:327)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:397)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:381)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:75)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:52)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:46)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
~[guava-16.0.jar:na]
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
~[guava-16.0.jar:na]
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:116)
~[apache-cassandra-2.1.2.jar:2.1.2]
... 18 common frames omitted
{quote}
Exception happen almost at the same moment on several nodes.
> mmap underflow during validation compaction
> -------------------------------------------
>
> Key: CASSANDRA-8337
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8337
> Project: Cassandra
> Issue Type: Bug
> Reporter: Alexander Sterligov
> Attachments: thread_dump
>
>
> During full parallel repair I often get errors like the following
> {quote}
> [2014-11-19 01:02:39,355] Repair session 116beaf0-6f66-11e4-afbb-c1c082008cbe
> for range (3074457345618263602,-9223372036854775808] failed with error
> org.apache.cassandra.exceptions.RepairException: [repair
> #116beaf0-6f66-11e4-afbb-c1c082008cbe on iss/target_state_history,
> (3074457345618263602,-9223372036854775808]] Validation failed in
> /95.108.242.19
> {quote}
> At the log of the node there are always same exceptions:
> {quote}
> ERROR [ValidationExecutor:2] 2014-11-19 01:02:10,847
> JVMStabilityInspector.java:94 - JVM state determined to be unstable. Exiting
> forcefully due to:
> org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException:
> mmap segment underflow; remaining is 15 but 47 requested
> at
> org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:1518)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:1385)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.io.sstable.SSTableReader.getPositionsForRanges(SSTableReader.java:1315)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1706)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1694)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:276)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.getScanners(WrappingCompactionStrategy.java:320)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:917)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:97)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:557)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[na:1.7.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[na:1.7.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_51]
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
> Caused by: java.io.IOException: mmap segment underflow; remaining is 15 but
> 47 requested
> at
> org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:135)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:348)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:327)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:1460)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> ... 13 common frames omitted
> {quote}
> Now i'm using die disk_failure_policy to determine such conditions faster,
> but I get them even with stop policy.
> Streams related to host with such exception are hanged. Thread dump is
> attached. Only restart helps.
> After retry I get errors from other nodes.
> scrub doesn't help and report that sstables are ok.
> Sequential repairs doesn't cause such exceptions.
> Load is about 1000 write rps and 50 read rps per node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
