Serhat Rıfat Demircan created CASSANDRA-13757: -------------------------------------------------
Summary: Cassandra 3.5.0 JVM Segfault Problem While Repair Job is Running Key: CASSANDRA-13757 URL: https://issues.apache.org/jira/browse/CASSANDRA-13757 Project: Cassandra Issue Type: Bug Environment: Operation System: Debian Jessie Java: Oracle JDK 1.8.0_131 Cassandra: 3.5.0 Reporter: Serhat Rıfat Demircan We got following error while repair job running on our cluster. One of the nodes stop due to following error. {code:java} # # SIGSEGV (0xb) at pc=0x00007fd80a399e70, pid=1305, tid=0x00007fd7ee7c4700 # # JRE version: Java(TM) SE Runtime Environment (8.0_131-b11) (build 1.8.0_131-b11) # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.131-b11 mixed mode linux-amd64 compressed oops) # Problematic frame: # C [liblz4-java3580121503903465201.so+0x5e70] LZ4_decompress_fast+0xd0 # # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again # # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp # The crash happened outside the Java Virtual Machine in native code. # See problematic frame for where to report the bug. # --------------- T H R E A D --------------- Current thread (0x00007fce32dad1b0): JavaThread "CompactionExecutor:9798" daemon [_thread_in_native, id=16879, stack(0x00007fd7ee784000,0x00007fd7ee7c5000)] siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x00007fd450c4d000 Registers: RAX=0x00007fcde6560d32, RBX=0x00007fd450c4cff9, RCX=0x00007fcde6560c7a, RDX=0x00007fcde6560d3e RSP=0x00007fd7ee7c3160, RBP=0x00007fd450c44ae6, RSI=0x00007fcde6562ff8, RDI=0x00000000000000c2 R8 =0x00007fcde6562ff4, R9 =0x00007fcde6563000, R10=0x0000000000000000, R11=0x0000000000000000 R12=0x000000000000000c, R13=0x00007fd4501cd000, R14=0x00007fcde6562ff7, R15=0x00007fcde6562ffb RIP=0x00007fd80a399e70, EFLAGS=0x0000000000010283, CSGSFS=0x0000000000000033, ERR=0x0000000000000004 TRAPNO=0x000000000000000e Top of Stack: (sp=0x00007fd7ee7c3160) 0x00007fd7ee7c3160: 0000000000000008 00007fd81e21c3d0 0x00007fd7ee7c3170: 0000000000000004 0000000000000001 0x00007fd7ee7c3180: 0000000000000002 0000000000000001 0x00007fd7ee7c3190: 0000000000000004 0000000000000004 0x00007fd7ee7c31a0: 0000000000000004 0000000000000004 0x00007fd7ee7c31b0: 0000000000000000 0000000000000000 0x00007fd7ee7c31c0: 0000000000000000 ffffffffffffffff 0x00007fd7ee7c31d0: 0000000000000000 0000000000000001 0x00007fd7ee7c31e0: 0000000000000002 0000000000000003 0x00007fd7ee7c31f0: 00007fd7ee7c32b8 00007fce32dad3a8 0x00007fd7ee7c3200: 0000000000000000 0000000000000000 0x00007fd7ee7c3210: 00007fd4501cd000 00007fcde6553000 0x00007fd7ee7c3220: 0000000000a77ae6 00007fd80a39659d 0x00007fd7ee7c3230: 0000000000000000 00000000dcb8fc9b 0x00007fd7ee7c3240: 00007fd7ee7c32d0 0000000000000000 0x00007fd7ee7c3250: 00000006e5c7e4d8 00007fd7ee7c32b8 0x00007fd7ee7c3260: 00007fce32dad1b0 00007fd81df2099d 0x00007fd7ee7c3270: 00007fd7ee7c32a8 0000000000000000 0x00007fd7ee7c3280: 0000000000010000 0000000000000000 0x00007fd7ee7c3290: 00000006e5c7e528 00007fd81d74df10 0x00007fd7ee7c32a0: 0000000000000000 00000006e5c7e4d8 0x00007fd7ee7c32b0: 00000006f6c7fbf8 00000006f6e957f0 0x00007fd7ee7c32c0: 00000006e5c7e350 00007fd87fffffff 0x00007fd7ee7c32d0: 00000006e5c7e528 00007fd81fa867e0 0x00007fd7ee7c32e0: 00a77ae200010000 0000000000a77ae2 0x00007fd7ee7c32f0: 00000006e5c7e488 000000000112d5f1 0x00007fd7ee7c3300: dcb8fc9b000099ce 0001000000a77ae6 0x00007fd7ee7c3310: 00a814b000a814b4 00000006e5c7e4d8 0x00007fd7ee7c3320: 00000006e5c7e4d8 00000006f6a4df38 0x00007fd7ee7c3330: 0000000600010000 000000067fffffff 0x00007fd7ee7c3340: 0089715800002c8a 0006189d87852057 0x00007fd7ee7c3350: 0000000000000000 00000000e5244e71 Instructions: (pc=0x00007fd80a399e70) 0x00007fd80a399e50: e4 0f 49 83 fc 0f 0f 84 94 00 00 00 4a 8d 14 20 0x00007fd80a399e60: 48 39 f2 0f 87 c0 00 00 00 0f 1f 80 00 00 00 00 0x00007fd80a399e70: 48 8b 0b 48 83 c3 08 48 89 08 48 83 c0 08 48 39 0x00007fd80a399e80: c2 77 ed 48 29 d0 48 89 d1 48 29 c3 0f b7 03 48 Register to memory mapping: RAX=0x00007fcde6560d32 is an unknown value RBX=0x00007fd450c4cff9 is an unknown value RCX=0x00007fcde6560c7a is an unknown value RDX=0x00007fcde6560d3e is an unknown value RSP=0x00007fd7ee7c3160 is pointing into the stack for thread: 0x00007fce32dad1b0 RBP=0x00007fd450c44ae6 is an unknown value RSI=0x00007fcde6562ff8 is an unknown value RDI=0x00000000000000c2 is an unknown value R8 =0x00007fcde6562ff4 is an unknown value R9 =0x00007fcde6563000 is an unknown value R10=0x0000000000000000 is an unknown value R11=0x0000000000000000 is an unknown value R12=0x000000000000000c is an unknown value R13=0x00007fd4501cd000 is an unknown value R14=0x00007fcde6562ff7 is an unknown value R15=0x00007fcde6562ffb is an unknown value Stack: [0x00007fd7ee784000,0x00007fd7ee7c5000], sp=0x00007fd7ee7c3160, free space=252k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) C [liblz4-java3580121503903465201.so+0x5e70] LZ4_decompress_fast+0xd0 Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) J 5084 net.jpountz.lz4.LZ4JNI.LZ4_decompress_fast([BLjava/nio/ByteBuffer;I[BLjava/nio/ByteBuffer;II)I (0 bytes) @ 0x00007fd81df20923 [0x00007fd81df20880+0xa3] J 20311 C2 org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap()V (354 bytes) @ 0x00007fd81fa867e0 [0x00007fd81fa86160+0x680] J 10706 C2 org.apache.cassandra.io.util.RandomAccessReader.reBuffer()V (78 bytes) @ 0x00007fd81ea59c58 [0x00007fd81ea59b20+0x138] J 19996 C2 org.apache.cassandra.io.util.SegmentedFile.createReader(J)Lorg/apache/cassandra/io/util/FileDataInput; (12 bytes) @ 0x00007fd81fa32934 [0x00007fd81fa32540+0x3f4] J 20645 C2 org.apache.cassandra.db.columniterator.AbstractSSTableIterator.<init>(Lorg/apache/cassandra/io/sstable/format/SSTableReader;Lorg/apache/cassandra/io/util/FileDataInput;Lorg/apache/cassandra/db/DecoratedKey;Lorg/apache/cassandra/db/RowIndexEntry;Lorg/apache/cassandra/db/Slices;Lorg/apache/cassandra/db/filter/ColumnFilter;Z)V (423 bytes) @ 0x00007fd81fbf8cec [0x00007fd81fbf8a20+0x2cc] J 20644 C2 org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext()Lorg/apache/cassandra/db/rows/Unfiltered; (120 bytes) @ 0x00007fd81e61b5d0 [0x00007fd81e61b140+0x490] J 9597 C2 org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext()Ljava/lang/Object; (5 bytes) @ 0x00007fd81da8dad4 [0x00007fd81da8daa0+0x34] J 3432 C2 org.apache.cassandra.utils.AbstractIterator.hasNext()Z (80 bytes) @ 0x00007fd81da28f20 [0x00007fd81da28e80+0xa0] J 19755 C2 org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext()Ljava/lang/Object; (9 bytes) @ 0x00007fd81f81d00c [0x00007fd81f81cb60+0x4ac] J 21810 C2 org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext()Lorg/apache/cassandra/db/rows/Unfiltered; (42 bytes) @ 0x00007fd81ef929fc [0x00007fd81ef927a0+0x25c] J 15842 C2 org.apache.cassandra.db.SinglePartitionReadCommand.withSSTablesIterated(Ljava/util/List;Lorg/apache/cassandra/metrics/TableMetrics;)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator; (71 bytes) @ 0x00007fd81ece33b4 [0x00007fd81ece3200+0x1b4] J 21162 C2 org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(Lorg/apache/cassandra/db/ColumnFamilyStore;)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator; (635 bytes) @ 0x00007fd81fdb2ff0 [0x00007fd81fdb2100+0xef0] J 27194 C2 org.apache.cassandra.db.Keyspace.indexPartition(Lorg/apache/cassandra/db/DecoratedKey;Lorg/apache/cassandra/db/ColumnFamilyStore;Ljava/util/Set;)V (274 bytes) @ 0x00007fd820c8b044 [0x00007fd820c8aae0+0x564] J 27422 C2 org.apache.cassandra.index.internal.CollatedViewIndexBuilder.build()V (73 bytes) @ 0x00007fd81f0b5a00 [0x00007fd81f0b5940+0xc0] J 27619 C1 org.apache.cassandra.db.compaction.CompactionManager$12.run()V (56 bytes) @ 0x00007fd82051313c [0x00007fd820512a00+0x73c] J 24483 C2 java.util.concurrent.FutureTask.run()V (126 bytes) @ 0x00007fd81d50faf4 [0x00007fd81d50fa20+0xd4] J 23783 C2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (225 bytes) @ 0x00007fd81fca71a8 [0x00007fd81fca70a0+0x108] J 21390 C2 java.lang.Thread.run()V (17 bytes) @ 0x00007fd81e6d3cb8 [0x00007fd81e6d3c60+0x58] v ~StubRoutines::call_stub {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org