[jira] [Commented] (CASSANDRA-14711) Apache Cassandra 3.2 crashing with exception org.apache.cassandra.db.marshal.TimestampType.compareCustom
[ https://issues.apache.org/jira/browse/CASSANDRA-14711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609190#comment-16609190 ] Saurabh commented on CASSANDRA-14711: - [~jasobrown] - Thanks for your response. We are in a process of planning the upgrade but as it is a Prod it will take time. We have started seeing this issue just a few days back and trying to fix it. There were no changes from Application code/DB changes. As per the hr_err log file (attached), I can see a lot of threads in Blocked status and also 100% used HEAP regions. I have tried increasing the -Xms - 4G -> 8G -> 16G -Xmx - 4G -> 8G -> 16G but this didnot thelp much but just delayed the crash. Something is pinning up in the memory but the cassandra logs does not show any OOM errors too. > Apache Cassandra 3.2 crashing with exception > org.apache.cassandra.db.marshal.TimestampType.compareCustom > > > Key: CASSANDRA-14711 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14711 > Project: Cassandra > Issue Type: Bug >Reporter: Saurabh >Priority: Major > Attachments: hs_err_pid32069.log > > > Hi Team, > I am using Apache Cassandra 3.2 with Java 1.8.0_161-b12.. > Issue: > Cassandra is continuously crashing with generating an HEAP dump log. There > are no errors reported in system.log OR Debug.log. > Exception in hs_err_PID.log: > # Problematic frame: > # J 8283 C2 > org.apache.cassandra.db.marshal.TimestampType.compareCustom(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I > (6 bytes) @ 0x2b7d3d417fb4 [0x2b7d3d417c80+0x334] > Java Threads: ( => current thread ) > 0x2b7da57924a0 JavaThread "MemtableReclaimMemory:52" daemon > [_thread_blocked, id=117880, stack(0x2b7d917ff000,0x2b7d9184)] > 0x2b7d39f6a9e0 JavaThread "PerDiskMemtableFlushWriter_0:52" daemon > [_thread_blocked, id=117879, stack(0x2b7e4ea94000,0x2b7e4ead5000)] > 0x2b7d39d0f520 JavaThread "MemtablePostFlush:53" daemon > [_thread_blocked, id=117878, stack(0x2b7e407dd000,0x2b7e4081e000)] > 0x2b7df31a9150 JavaThread "MemtableFlushWriter:52" daemon > [_thread_blocked, id=117877, stack(0x2b7e406d9000,0x2b7e4071a000)] > 0x2b7e53e60110 JavaThread "RMI TCP Connection(1795)-127.0.0.1" daemon > : > : > lot of threads in BLOCKED status > Other Threads: > 0x2b7d38de5ea0 VMThread [stack: 0x2b7d8208d000,0x2b7d8218d000] > [id=32098] > 0x2b7d38fa9de0 WatcherThread [stack: > 0x2b7d88ee9000,0x2b7d88fe9000] [id=32108] > VM state:not at safepoint (normal execution) > VM Mutex/Monitor currently owned by a thread: None > Heap: > garbage-first heap total 8388608K, used 6791168K [0x0003c000, > 0x0003c0404000, 0x0007c000) > region size 4096K, 785 young (3215360K), 55 survivors (225280K) > Metaspace used 40915K, capacity 42044K, committed 42368K, reserved 1087488K > class space used 4429K, capacity 4646K, committed 4736K, reserved 1048576K > Heap Regions: (Y=young(eden), SU=young(survivor), HS=humongous(starts), > HC=humongous(continues), CS=collection set, F=free, TS=gc time stamp, > PTAMS=previous top-at-mark-start, NTAMS=next top-at-mark-start) > AC 0 O TS 0 PTAMS 0x0003c040 NTAMS 0x0003c040 space 4096K, > 100% used [0x0003c000, 0x0003c040) > AC 0 O TS 0 PTAMS 0x0003c080 NTAMS 0x0003c080 space 4096K, > 100% used [0x0003c040, 0x0003c080) > AC 0 O TS 9 PTAMS 0x0003c080 NTAMS 0x0003c080 space 4096K, > 100% used [0x0003c080, 0x0003c0c0) > AC 0 O TS 11 PTAMS 0x0003c0c0 NTAMS 0x0003c0c0 space 4096K, > 100% used [0x0003c0c0, 0x0003c100) > AC 0 O TS 11 PTAMS 0x0003c100 NTAMS 0x0003c100 space 4096K, > 100% used [0x0003c100, 0x0003c140) > AC 0 O TS 11 PTAMS 0x0003c140 NTAMS 0x0003c140 space 4096K, > 100% used [0x0003c140, 0x0003c180) > : > : > lot of such messages -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14711) Apache Cassandra 3.2 crashing with exception org.apache.cassandra.db.marshal.TimestampType.compareCustom
[ https://issues.apache.org/jira/browse/CASSANDRA-14711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608828#comment-16608828 ] Saurabh commented on CASSANDRA-14711: - Cassandra config: java -ea -XX:+UseThreadPriorities -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103 -XX:+AlwaysPreTouch -XX:-UseBiasedLocking -XX:+UseTLAB -XX:+ResizeTLAB -XX:+PerfDisableSharedMem -Djava.net.preferIPv4Stack=true -Xms8G -Xmx16G -XX:+CMSClassUnloadingEnabled -XX:+UseG1GC -XX:G1RSetUpdatingPauseTimePercent=5 -XX:MaxGCPauseMillis=500 -XX:InitiatingHeapOccupancyPercent=70 -XX:ParallelGCThreads=10 -XX:ConcGCThreads=3 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -XX:+PrintPromotionFailure -XX:PrintFLSStatistics=1 -Xloggc:/data/cassandra/log/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M -XX:CompileCommandFile=/data/tools/repository/apache-cassandra-3.2/conf/hotspot_compiler -javaagent:/data/tools/repository/apache-cassandra-3.2/lib/jamm-0.3.0.jar -Djava.net.preferIPv4Stack=true -Dcassandra.jmx.local.port=7199 -XX:+DisableExplicitGC -Djava.library.path=/data/tools/repository/apache-cassandra-3.2/lib/sigar-bin -Dcassandra.max_queued_native_transport_requests=4096 -Dlogback.configurationFile=logback.xml -Dcas andra.logdir=/data/tools/repository/apache-cassandra-3.2/logs -Dcassandra.storagedir=/data/tools/repository/apache-cassandra-3.2/data -cp /data/tools/repository/apache-cassandra-3.2/conf > Apache Cassandra 3.2 crashing with exception > org.apache.cassandra.db.marshal.TimestampType.compareCustom > > > Key: CASSANDRA-14711 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14711 > Project: Cassandra > Issue Type: Bug >Reporter: Saurabh >Priority: Minor > Attachments: hs_err_pid32069.log > > > Hi Team, > I am using Apache Cassandra 3.2 with Java 1.8.0_161-b12.. > Issue: > Cassandra is continuously crashing with generating an HEAP dump log. There > are no errors reported in system.log OR Debug.log. > Exception in hs_err_PID.log: > # Problematic frame: > # J 8283 C2 > org.apache.cassandra.db.marshal.TimestampType.compareCustom(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I > (6 bytes) @ 0x2b7d3d417fb4 [0x2b7d3d417c80+0x334] > Java Threads: ( => current thread ) > 0x2b7da57924a0 JavaThread "MemtableReclaimMemory:52" daemon > [_thread_blocked, id=117880, stack(0x2b7d917ff000,0x2b7d9184)] > 0x2b7d39f6a9e0 JavaThread "PerDiskMemtableFlushWriter_0:52" daemon > [_thread_blocked, id=117879, stack(0x2b7e4ea94000,0x2b7e4ead5000)] > 0x2b7d39d0f520 JavaThread "MemtablePostFlush:53" daemon > [_thread_blocked, id=117878, stack(0x2b7e407dd000,0x2b7e4081e000)] > 0x2b7df31a9150 JavaThread "MemtableFlushWriter:52" daemon > [_thread_blocked, id=117877, stack(0x2b7e406d9000,0x2b7e4071a000)] > 0x2b7e53e60110 JavaThread "RMI TCP Connection(1795)-127.0.0.1" daemon > : > : > lot of threads in BLOCKED status > Other Threads: > 0x2b7d38de5ea0 VMThread [stack: 0x2b7d8208d000,0x2b7d8218d000] > [id=32098] > 0x2b7d38fa9de0 WatcherThread [stack: > 0x2b7d88ee9000,0x2b7d88fe9000] [id=32108] > VM state:not at safepoint (normal execution) > VM Mutex/Monitor currently owned by a thread: None > Heap: > garbage-first heap total 8388608K, used 6791168K [0x0003c000, > 0x0003c0404000, 0x0007c000) > region size 4096K, 785 young (3215360K), 55 survivors (225280K) > Metaspace used 40915K, capacity 42044K, committed 42368K, reserved 1087488K > class space used 4429K, capacity 4646K, committed 4736K, reserved 1048576K > Heap Regions: (Y=young(eden), SU=young(survivor), HS=humongous(starts), > HC=humongous(continues), CS=collection set, F=free, TS=gc time stamp, > PTAMS=previous top-at-mark-start, NTAMS=next top-at-mark-start) > AC 0 O TS 0 PTAMS 0x0003c040 NTAMS 0x0003c040 space 4096K, > 100% used [0x0003c000, 0x0003c040) > AC 0 O TS 0 PTAMS 0x0003c080 NTAMS 0x0003c080 space 4096K, > 100% used [0x0003c040, 0x0003c080) > AC 0 O TS 9 PTAMS 0x0003c080 NTAMS 0x0003c080 space 4096K, > 100% used [0x0003c080, 0x0003c0c0) > AC 0 O TS 11 PTAMS 0x0003c0c0 NTAMS 0x0003c0c0 space 4096K, > 100% used [0x0003c0c0, 0x0003c100) > AC 0 O TS 11 PTAMS 0x0003c100 NTAMS 0x0003c100 space 4096K, > 100% used [0x0003c100, 0x0003c140) > AC 0 O TS 11 PTAMS 0x0003c140 NTAMS 0x0003c140 space 4096K, > 100% used [0x0003c140, 0x0003c180) > : > : > lot
[jira] [Updated] (CASSANDRA-14711) Apache Cassandra 3.2 crashing with exception org.apache.cassandra.db.marshal.TimestampType.compareCustom
[ https://issues.apache.org/jira/browse/CASSANDRA-14711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saurabh updated CASSANDRA-14711: Priority: Major (was: Minor) > Apache Cassandra 3.2 crashing with exception > org.apache.cassandra.db.marshal.TimestampType.compareCustom > > > Key: CASSANDRA-14711 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14711 > Project: Cassandra > Issue Type: Bug >Reporter: Saurabh >Priority: Major > Attachments: hs_err_pid32069.log > > > Hi Team, > I am using Apache Cassandra 3.2 with Java 1.8.0_161-b12.. > Issue: > Cassandra is continuously crashing with generating an HEAP dump log. There > are no errors reported in system.log OR Debug.log. > Exception in hs_err_PID.log: > # Problematic frame: > # J 8283 C2 > org.apache.cassandra.db.marshal.TimestampType.compareCustom(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I > (6 bytes) @ 0x2b7d3d417fb4 [0x2b7d3d417c80+0x334] > Java Threads: ( => current thread ) > 0x2b7da57924a0 JavaThread "MemtableReclaimMemory:52" daemon > [_thread_blocked, id=117880, stack(0x2b7d917ff000,0x2b7d9184)] > 0x2b7d39f6a9e0 JavaThread "PerDiskMemtableFlushWriter_0:52" daemon > [_thread_blocked, id=117879, stack(0x2b7e4ea94000,0x2b7e4ead5000)] > 0x2b7d39d0f520 JavaThread "MemtablePostFlush:53" daemon > [_thread_blocked, id=117878, stack(0x2b7e407dd000,0x2b7e4081e000)] > 0x2b7df31a9150 JavaThread "MemtableFlushWriter:52" daemon > [_thread_blocked, id=117877, stack(0x2b7e406d9000,0x2b7e4071a000)] > 0x2b7e53e60110 JavaThread "RMI TCP Connection(1795)-127.0.0.1" daemon > : > : > lot of threads in BLOCKED status > Other Threads: > 0x2b7d38de5ea0 VMThread [stack: 0x2b7d8208d000,0x2b7d8218d000] > [id=32098] > 0x2b7d38fa9de0 WatcherThread [stack: > 0x2b7d88ee9000,0x2b7d88fe9000] [id=32108] > VM state:not at safepoint (normal execution) > VM Mutex/Monitor currently owned by a thread: None > Heap: > garbage-first heap total 8388608K, used 6791168K [0x0003c000, > 0x0003c0404000, 0x0007c000) > region size 4096K, 785 young (3215360K), 55 survivors (225280K) > Metaspace used 40915K, capacity 42044K, committed 42368K, reserved 1087488K > class space used 4429K, capacity 4646K, committed 4736K, reserved 1048576K > Heap Regions: (Y=young(eden), SU=young(survivor), HS=humongous(starts), > HC=humongous(continues), CS=collection set, F=free, TS=gc time stamp, > PTAMS=previous top-at-mark-start, NTAMS=next top-at-mark-start) > AC 0 O TS 0 PTAMS 0x0003c040 NTAMS 0x0003c040 space 4096K, > 100% used [0x0003c000, 0x0003c040) > AC 0 O TS 0 PTAMS 0x0003c080 NTAMS 0x0003c080 space 4096K, > 100% used [0x0003c040, 0x0003c080) > AC 0 O TS 9 PTAMS 0x0003c080 NTAMS 0x0003c080 space 4096K, > 100% used [0x0003c080, 0x0003c0c0) > AC 0 O TS 11 PTAMS 0x0003c0c0 NTAMS 0x0003c0c0 space 4096K, > 100% used [0x0003c0c0, 0x0003c100) > AC 0 O TS 11 PTAMS 0x0003c100 NTAMS 0x0003c100 space 4096K, > 100% used [0x0003c100, 0x0003c140) > AC 0 O TS 11 PTAMS 0x0003c140 NTAMS 0x0003c140 space 4096K, > 100% used [0x0003c140, 0x0003c180) > : > : > lot of such messages -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14711) Apache Cassandra 3.2 crashing with exception org.apache.cassandra.db.marshal.TimestampType.compareCustom
[ https://issues.apache.org/jira/browse/CASSANDRA-14711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saurabh updated CASSANDRA-14711: Description: Hi Team, I am using Apache Cassandra 3.2 with Java 1.8.0_161-b12.. Issue: Cassandra is continuously crashing with generating an HEAP dump log. There are no errors reported in system.log OR Debug.log. Exception in hs_err_PID.log: # Problematic frame: # J 8283 C2 org.apache.cassandra.db.marshal.TimestampType.compareCustom(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I (6 bytes) @ 0x2b7d3d417fb4 [0x2b7d3d417c80+0x334] #--- P R O C E S S --- Java Threads: ( => current thread ) 0x2b7da57924a0 JavaThread "MemtableReclaimMemory:52" daemon [_thread_blocked, id=117880, stack(0x2b7d917ff000,0x2b7d9184)] 0x2b7d39f6a9e0 JavaThread "PerDiskMemtableFlushWriter_0:52" daemon [_thread_blocked, id=117879, stack(0x2b7e4ea94000,0x2b7e4ead5000)] 0x2b7d39d0f520 JavaThread "MemtablePostFlush:53" daemon [_thread_blocked, id=117878, stack(0x2b7e407dd000,0x2b7e4081e000)] 0x2b7df31a9150 JavaThread "MemtableFlushWriter:52" daemon [_thread_blocked, id=117877, stack(0x2b7e406d9000,0x2b7e4071a000)] 0x2b7e53e60110 JavaThread "RMI TCP Connection(1795)-127.0.0.1" daemon : : lot of threads in BLOCKED status Other Threads: 0x2b7d38de5ea0 VMThread [stack: 0x2b7d8208d000,0x2b7d8218d000] [id=32098] 0x2b7d38fa9de0 WatcherThread [stack: 0x2b7d88ee9000,0x2b7d88fe9000] [id=32108] VM state:not at safepoint (normal execution) VM Mutex/Monitor currently owned by a thread: None Heap: garbage-first heap total 8388608K, used 6791168K [0x0003c000, 0x0003c0404000, 0x0007c000) region size 4096K, 785 young (3215360K), 55 survivors (225280K) Metaspace used 40915K, capacity 42044K, committed 42368K, reserved 1087488K class space used 4429K, capacity 4646K, committed 4736K, reserved 1048576K Heap Regions: (Y=young(eden), SU=young(survivor), HS=humongous(starts), HC=humongous(continues), CS=collection set, F=free, TS=gc time stamp, PTAMS=previous top-at-mark-start, NTAMS=next top-at-mark-start) AC 0 O TS 0 PTAMS 0x0003c040 NTAMS 0x0003c040 space 4096K, 100% used [0x0003c000, 0x0003c040) AC 0 O TS 0 PTAMS 0x0003c080 NTAMS 0x0003c080 space 4096K, 100% used [0x0003c040, 0x0003c080) AC 0 O TS 9 PTAMS 0x0003c080 NTAMS 0x0003c080 space 4096K, 100% used [0x0003c080, 0x0003c0c0) AC 0 O TS 11 PTAMS 0x0003c0c0 NTAMS 0x0003c0c0 space 4096K, 100% used [0x0003c0c0, 0x0003c100) AC 0 O TS 11 PTAMS 0x0003c100 NTAMS 0x0003c100 space 4096K, 100% used [0x0003c100, 0x0003c140) AC 0 O TS 11 PTAMS 0x0003c140 NTAMS 0x0003c140 space 4096K, 100% used [0x0003c140, 0x0003c180) : : lot of such messages was: Hi Team, I am using Apache Cassandra 3.2 with Java 1.8.0_161-b12.. Issue: Cassandra is continuously crashing with generating an HEAP dump log. There are no errors reported in system.log OR Debug.log. Exception in hs_err_PID.log: # Problematic frame: # J 8283 C2 org.apache.cassandra.db.marshal.TimestampType.compareCustom(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I (6 bytes) @ 0x2b7d3d417fb4 [0x2b7d3d417c80+0x334] # --- T H R E A D --- Current thread (0x2b7d3a1033e0): JavaThread "SharedPool-Worker-1" daemon [_thread_in_Java, id=32216, stack(0x2b7e4085f000,0x2b7e408a)] siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x14914c69 Registers: RAX=0x0001, RBX=0x, RCX=0x9f1fbef0, RDX=0x0004f8fdf798 RSP=0x2b7e4089e4b0, RBP=0x0001, RSI=0x14907800, RDI=0x R8 =0xd469, R9 =0x, R10=0x0004a41764c8, R11=0x R12=0x, R13=0x, R14=0xd469, R15=0x2b7d3a1033e0 RIP=0x2b7d3d417fb4, EFLAGS=0x00010283, CSGSFS=0x0033, ERR=0x0004 TRAPNO=0x000e [error occurred during error reporting (printing register info), id 0xb] Stack: [0x2b7e4085f000,0x2b7e408a], sp=0x2b7e4089e4b0, free space=253k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) J 8283 C2 org.apache.cassandra.db.marshal.TimestampType.compareCustom(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I (6 bytes) @ 0x2b7d3d417fb4 [0x2b7d3d417c80+0x334] J 12970 C2 org.apache.cassandra.db.Slice$Bound.compareTo(Lorg/apache/cassandra/db/ClusteringComparator;Ljava/util/List;)I (119 bytes) @ 0x2b7d3e0291c0 [0x2b7d3e028900+0x8c0] J 16245 C2
[jira] [Updated] (CASSANDRA-14711) Apache Cassandra 3.2 crashing with exception org.apache.cassandra.db.marshal.TimestampType.compareCustom
[ https://issues.apache.org/jira/browse/CASSANDRA-14711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saurabh updated CASSANDRA-14711: Description: Hi Team, I am using Apache Cassandra 3.2 with Java 1.8.0_161-b12.. Issue: Cassandra is continuously crashing with generating an HEAP dump log. There are no errors reported in system.log OR Debug.log. Exception in hs_err_PID.log: # Problematic frame: # J 8283 C2 org.apache.cassandra.db.marshal.TimestampType.compareCustom(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I (6 bytes) @ 0x2b7d3d417fb4 [0x2b7d3d417c80+0x334] Java Threads: ( => current thread ) 0x2b7da57924a0 JavaThread "MemtableReclaimMemory:52" daemon [_thread_blocked, id=117880, stack(0x2b7d917ff000,0x2b7d9184)] 0x2b7d39f6a9e0 JavaThread "PerDiskMemtableFlushWriter_0:52" daemon [_thread_blocked, id=117879, stack(0x2b7e4ea94000,0x2b7e4ead5000)] 0x2b7d39d0f520 JavaThread "MemtablePostFlush:53" daemon [_thread_blocked, id=117878, stack(0x2b7e407dd000,0x2b7e4081e000)] 0x2b7df31a9150 JavaThread "MemtableFlushWriter:52" daemon [_thread_blocked, id=117877, stack(0x2b7e406d9000,0x2b7e4071a000)] 0x2b7e53e60110 JavaThread "RMI TCP Connection(1795)-127.0.0.1" daemon : : lot of threads in BLOCKED status Other Threads: 0x2b7d38de5ea0 VMThread [stack: 0x2b7d8208d000,0x2b7d8218d000] [id=32098] 0x2b7d38fa9de0 WatcherThread [stack: 0x2b7d88ee9000,0x2b7d88fe9000] [id=32108] VM state:not at safepoint (normal execution) VM Mutex/Monitor currently owned by a thread: None Heap: garbage-first heap total 8388608K, used 6791168K [0x0003c000, 0x0003c0404000, 0x0007c000) region size 4096K, 785 young (3215360K), 55 survivors (225280K) Metaspace used 40915K, capacity 42044K, committed 42368K, reserved 1087488K class space used 4429K, capacity 4646K, committed 4736K, reserved 1048576K Heap Regions: (Y=young(eden), SU=young(survivor), HS=humongous(starts), HC=humongous(continues), CS=collection set, F=free, TS=gc time stamp, PTAMS=previous top-at-mark-start, NTAMS=next top-at-mark-start) AC 0 O TS 0 PTAMS 0x0003c040 NTAMS 0x0003c040 space 4096K, 100% used [0x0003c000, 0x0003c040) AC 0 O TS 0 PTAMS 0x0003c080 NTAMS 0x0003c080 space 4096K, 100% used [0x0003c040, 0x0003c080) AC 0 O TS 9 PTAMS 0x0003c080 NTAMS 0x0003c080 space 4096K, 100% used [0x0003c080, 0x0003c0c0) AC 0 O TS 11 PTAMS 0x0003c0c0 NTAMS 0x0003c0c0 space 4096K, 100% used [0x0003c0c0, 0x0003c100) AC 0 O TS 11 PTAMS 0x0003c100 NTAMS 0x0003c100 space 4096K, 100% used [0x0003c100, 0x0003c140) AC 0 O TS 11 PTAMS 0x0003c140 NTAMS 0x0003c140 space 4096K, 100% used [0x0003c140, 0x0003c180) : : lot of such messages was: Hi Team, I am using Apache Cassandra 3.2 with Java 1.8.0_161-b12.. Issue: Cassandra is continuously crashing with generating an HEAP dump log. There are no errors reported in system.log OR Debug.log. Exception in hs_err_PID.log: # Problematic frame: # J 8283 C2 org.apache.cassandra.db.marshal.TimestampType.compareCustom(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I (6 bytes) @ 0x2b7d3d417fb4 [0x2b7d3d417c80+0x334] #--- P R O C E S S --- Java Threads: ( => current thread ) 0x2b7da57924a0 JavaThread "MemtableReclaimMemory:52" daemon [_thread_blocked, id=117880, stack(0x2b7d917ff000,0x2b7d9184)] 0x2b7d39f6a9e0 JavaThread "PerDiskMemtableFlushWriter_0:52" daemon [_thread_blocked, id=117879, stack(0x2b7e4ea94000,0x2b7e4ead5000)] 0x2b7d39d0f520 JavaThread "MemtablePostFlush:53" daemon [_thread_blocked, id=117878, stack(0x2b7e407dd000,0x2b7e4081e000)] 0x2b7df31a9150 JavaThread "MemtableFlushWriter:52" daemon [_thread_blocked, id=117877, stack(0x2b7e406d9000,0x2b7e4071a000)] 0x2b7e53e60110 JavaThread "RMI TCP Connection(1795)-127.0.0.1" daemon : : lot of threads in BLOCKED status Other Threads: 0x2b7d38de5ea0 VMThread [stack: 0x2b7d8208d000,0x2b7d8218d000] [id=32098] 0x2b7d38fa9de0 WatcherThread [stack: 0x2b7d88ee9000,0x2b7d88fe9000] [id=32108] VM state:not at safepoint (normal execution) VM Mutex/Monitor currently owned by a thread: None Heap: garbage-first heap total 8388608K, used 6791168K [0x0003c000, 0x0003c0404000, 0x0007c000) region size 4096K, 785 young (3215360K), 55 survivors (225280K) Metaspace used 40915K, capacity 42044K, committed 42368K, reserved 1087488K class space used 4429K, capacity 4646K, committed 4736K, reserved 1048576K Heap Regions: (Y=young(eden), SU=young(survivor), HS=humongous(starts), HC=humongous(continues), CS=collection set, F=free, TS=gc time stamp,
[jira] [Created] (CASSANDRA-14711) Apache Cassandra 3.2 crashing with exception org.apache.cassandra.db.marshal.TimestampType.compareCustom
Saurabh created CASSANDRA-14711: --- Summary: Apache Cassandra 3.2 crashing with exception org.apache.cassandra.db.marshal.TimestampType.compareCustom Key: CASSANDRA-14711 URL: https://issues.apache.org/jira/browse/CASSANDRA-14711 Project: Cassandra Issue Type: Bug Reporter: Saurabh Attachments: hs_err_pid32069.log Hi Team, I am using Apache Cassandra 3.2 with Java 1.8.0_161-b12.. Issue: Cassandra is continuously crashing with generating an HEAP dump log. There are no errors reported in system.log OR Debug.log. Exception in hs_err_PID.log: # Problematic frame: # J 8283 C2 org.apache.cassandra.db.marshal.TimestampType.compareCustom(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I (6 bytes) @ 0x2b7d3d417fb4 [0x2b7d3d417c80+0x334] # --- T H R E A D --- Current thread (0x2b7d3a1033e0): JavaThread "SharedPool-Worker-1" daemon [_thread_in_Java, id=32216, stack(0x2b7e4085f000,0x2b7e408a)] siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x14914c69 Registers: RAX=0x0001, RBX=0x, RCX=0x9f1fbef0, RDX=0x0004f8fdf798 RSP=0x2b7e4089e4b0, RBP=0x0001, RSI=0x14907800, RDI=0x R8 =0xd469, R9 =0x, R10=0x0004a41764c8, R11=0x R12=0x, R13=0x, R14=0xd469, R15=0x2b7d3a1033e0 RIP=0x2b7d3d417fb4, EFLAGS=0x00010283, CSGSFS=0x0033, ERR=0x0004 TRAPNO=0x000e [error occurred during error reporting (printing register info), id 0xb] Stack: [0x2b7e4085f000,0x2b7e408a], sp=0x2b7e4089e4b0, free space=253k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) J 8283 C2 org.apache.cassandra.db.marshal.TimestampType.compareCustom(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I (6 bytes) @ 0x2b7d3d417fb4 [0x2b7d3d417c80+0x334] J 12970 C2 org.apache.cassandra.db.Slice$Bound.compareTo(Lorg/apache/cassandra/db/ClusteringComparator;Ljava/util/List;)I (119 bytes) @ 0x2b7d3e0291c0 [0x2b7d3e028900+0x8c0] J 16245 C2 org.apache.cassandra.db.Slices$ArrayBackedSlices.intersects(Ljava/util/List;Ljava/util/List;)Z (46 bytes) @ 0x2b7d3e619cfc [0x2b7d3e619b20+0x1dc] J 18878 C2 org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(Lorg/apache/cassandra/db/ColumnFamilyStore;Z)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator; (822 bytes) @ 0x2b7d3ebcabf4 [0x2b7d3ebc7be0+0x3014] J 9377 C2 org.apache.cassandra.db.ReadCommand.executeLocally(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator; (219 bytes) @ 0x2b7d3d80cde8 [0x2b7d3d80c0a0+0xd48] J 14198 C2 org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(Lorg/apache/cassandra/net/MessageIn;I)V (328 bytes) @ 0x2b7d3c8bcbd0 [0x2b7d3c8bca20+0x1b0] J 9731 C2 org.apache.cassandra.net.MessageDeliveryTask.run()V (187 bytes) @ 0x2b7d3d158d60 [0x2b7d3d158bc0+0x1a0] J 18999% C2 org.apache.cassandra.concurrent.SEPWorker.run()V (253 bytes) @ 0x2b7d3eaa10ec [0x2b7d3eaa0960+0x78c] j java.lang.Thread.run()V+11 v ~StubRoutines::call_stub V [libjvm.so+0x695ae6] JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056 V [libjvm.so+0x695ff1] JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x321 V [libjvm.so+0x696497] JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x47 V [libjvm.so+0x731cb0] thread_entry(JavaThread*, Thread*)+0xa0 V [libjvm.so+0xa7eaa3] JavaThread::thread_main_inner()+0x103 V [libjvm.so+0xa7ebec] JavaThread::run()+0x11c V [libjvm.so+0x92da28] java_start(Thread*)+0x108 C [libpthread.so.0+0x7e25] start_thread+0xc5 --- P R O C E S S --- Java Threads: ( => current thread ) 0x2b7da57924a0 JavaThread "MemtableReclaimMemory:52" daemon [_thread_blocked, id=117880, stack(0x2b7d917ff000,0x2b7d9184)] 0x2b7d39f6a9e0 JavaThread "PerDiskMemtableFlushWriter_0:52" daemon [_thread_blocked, id=117879, stack(0x2b7e4ea94000,0x2b7e4ead5000)] 0x2b7d39d0f520 JavaThread "MemtablePostFlush:53" daemon [_thread_blocked, id=117878, stack(0x2b7e407dd000,0x2b7e4081e000)] 0x2b7df31a9150 JavaThread "MemtableFlushWriter:52" daemon [_thread_blocked, id=117877, stack(0x2b7e406d9000,0x2b7e4071a000)] 0x2b7e53e60110 JavaThread "RMI TCP Connection(1795)-127.0.0.1" daemon : : lot of threads in BLOCKED status Other Threads: 0x2b7d38de5ea0 VMThread [stack: 0x2b7d8208d000,0x2b7d8218d000] [id=32098] 0x2b7d38fa9de0 WatcherThread [stack: 0x2b7d88ee9000,0x2b7d88fe9000] [id=32108] VM