[
https://issues.apache.org/jira/browse/CASSANDRA-11200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15857837#comment-15857837
]
Kurt Greaves commented on CASSANDRA-11200:
------------------------------------------
Saw this on a 3.7 node. The IAE exception occured several times prior to the
SIGBUS. On restart there were a couple corrupt SSTables in sstable_activity,
not sure if it was related. I don't have the core dumps but might try see if I
can reproduce with some user defined compactions.
{code}
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: ERROR
o.a.c.service.CassandraDaemon Exception in thread
Thread[CompactionExecutor:86114,1,main]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:
java.lang.IllegalArgumentException: null
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
java.nio.Buffer.position(Buffer.java:244) ~[na:1.8.0_111]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
org.apache.cassandra.io.compress.LZ4Compressor.uncompress(LZ4Compressor.java:163)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
org.apache.cassandra.io.util.CompressedSegmentedFile$Mmap.readChunk(CompressedSegmentedFile.java:323)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:137)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:19)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache.lambda$new$0(BoundedLocalCache.java:2949)
~[caffeine-2.2.6.jar:na]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$15(BoundedLocalCache.java:1807)
~[caffeine-2.2.6.jar:na]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
~[na:1.8.0_111]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:1805)
~[caffeine-2.2.6.jar:na]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:1788)
~[caffeine-2.2.6.jar:na]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:97)
~[caffeine-2.2.6.jar:na]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:66)
~[caffeine-2.2.6.jar:na]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:215)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:193)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
org.apache.cassandra.io.util.LimitingRebufferer.rebuffer(LimitingRebufferer.java:34)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
org.apache.cassandra.io.util.RandomAccessReader.reBufferAt(RandomAccessReader.java:78)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:220)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
org.apache.cassandra.io.sstable.format.big.BigTableScanner.seekToCurrentRangeStart(BigTableScanner.java:186)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
org.apache.cassandra.io.sstable.format.big.BigTableScanner.access$200(BigTableScanner.java:51)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator.computeNext(BigTableScanner.java:285)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator.computeNext(BigTableScanner.java:265)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
org.apache.cassandra.io.sstable.format.big.BigTableScanner.hasNext(BigTableScanner.java:245)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$2.hasNext(UnfilteredPartitionIterators.java:150)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:72)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:226)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:182)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:82)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_111]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_111]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_111]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_111]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: at
java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
Feb 08 08:46:43 ip-10-224-42-94.ec2.internal cassandra[31038]: #
Feb 08 08:46:43 ip-10-224-42-94.ec2.internal cassandra[31038]: # A fatal error
has been detected by the Java Runtime Environment:
Feb 08 08:46:43 ip-10-224-42-94.ec2.internal cassandra[31038]: #
Feb 08 08:46:43 ip-10-224-42-94.ec2.internal cassandra[31038]: # SIGBUS (0x7)
at pc=0x00007ff4150520d0, pid=1, tid=0x00007ff175b28700
Feb 08 08:46:43 ip-10-224-42-94.ec2.internal cassandra[31038]: #
Feb 08 08:46:43 ip-10-224-42-94.ec2.internal cassandra[31038]: # JRE version:
Java(TM) SE Runtime Environment (8.0_111-b14) (build 1.8.0_111-b14)
Feb 08 08:46:43 ip-10-224-42-94.ec2.internal cassandra[31038]: # Java VM: Java
HotSpot(TM) 64-Bit Server VM (25.111-b14 mixed mode linux-amd64 compressed oops)
Feb 08 08:46:43 ip-10-224-42-94.ec2.internal cassandra[31038]: # Problematic
frame:
Feb 08 08:46:43 ip-10-224-42-94.ec2.internal cassandra[31038]: # v
~StubRoutines::jbyte_disjoint_arraycopy
{code}
> CompactionExecutor thread error brings down JVM in 3.0.3
> --------------------------------------------------------
>
> Key: CASSANDRA-11200
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11200
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: debian jesse latest release, updated Feb. 20th
> Reporter: Jason Kania
> Priority: Critical
>
> When launching Cassandra 3.0.3, with java version "1.8.0_74", Cassandra
> writes the following to the debug file before a segmentation fault occurs
> bringing down the JVM - the problem is repeatable.
> DEBUG [CompactionExecutor:1] 2016-02-20 18:26:16,892 CompactionTask.java:146
> - Compacting (56f677c0-d829-11e5-b23a-25dbd4d727f6)
> [/var/lib/cassandra/data/sensordb/periodicReading/ma-367-big-Data.db:level=0,
> /var/lib/cassandra/data/sensordb/periodicReading/ma-368-big-Data.db:level=0,
> /var/lib/cassandra/data/sensordb/periodicReading/ma-371-big-Data.db:level=0,
> /var/lib/cassandra/data/sensordb/periodicReading/ma-370-big-Data.db:level=0,
> /var/lib/cassandra/data/sensordb/periodicReading/ma-369-big-Data.db:level=0, ]
> The JVM error that occurs is the following:
> \#
> \# A fatal error has been detected by the Java Runtime Environment:
> \#
> \# SIGBUS (0x7) at pc=0x00007fa8a1052150, pid=12179, tid=140361951868672
> \#
> \# JRE version: Java(TM) SE Runtime Environment (8.0_74-b02) (build
> 1.8.0_74-b02)
> \# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.74-b02 mixed mode
> linux-amd64 compressed oops)
> \# Problematic frame:
> \# v ~StubRoutines::jbyte_disjoint_arraycopy
> \#
> \# Core dump written. Default location: /tmp/core or core.12179
> \#
> \# If you would like to submit a bug report, please visit:
> \# http://bugreport.java.com/bugreport/crash.jsp
> \#
> --------------- T H R E A D ---------------
> Current thread (0x00007fa89c56ac20): JavaThread "CompactionExecutor:1"
> daemon [_thread_in_Java, id=12323,
> stack(0x00007fa89043f000,0x00007fa890480000)]
> siginfo: si_signo: 7 (SIGBUS), si_code: 2 (BUS_ADRERR), si_addr:
> 0x00007fa838988002
> Even if all of the files associated with "ma-[NNN]*" are removed, the JVM
> dies with the same error after the next group of "ma-[NNN]*" are eventually
> written out and compacted.
> Though this may be strictly a JVM problem, I have seen the issue in Oracle
> JVM 8.0_65 and 8.0_74 and I raise it in case this problem is due to JNI usage
> of an external compression library or some direct memory usage.
> I have a core dump if that is helpful to anyone.
> Bug CASSANDRA-11201 may also be related although when the exception
> referenced in the bug occurs, the JVM remains alive.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
