>From last few days i am seeing on some of the nodes in cassandra cluster
DSE is getting shutdown due to the error below and i need to kill Java
process and restart DSE service.
I have cross checked reads and writes and compactions nothing looks
suspicious, but i am seeing full Gc pause on these server just before the
issue happening. what can be the reason for this how to debug that ?? how
to fine grain why on those particular nodes this is happening when these
nodes are serving same requests like rest of the cluster ??
Is this happening because of Full Gc is not getting performed properly, we
using G1GC and DSE 4.8.3
ERROR [SharedPool-Worker-25] 2016-12-27 10:14:26,100
JVMStabilityInspector.java:117 - JVM state determined to be unstable.
Exiting forcefully due to:java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3181) ~[na:1.8.0_74]
at
org.apache.cassandra.db.RangeTombstoneList.copy(RangeTombstoneList.java:112)
~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at
org.apache.cassandra.db.DeletionInfo.copy(DeletionInfo.java:104)
~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at
org.apache.cassandra.db.AtomicBTreeColumns.addAllWithSizeDelta(AtomicBTreeColumns.java:217)
~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at org.apache.cassandra.db.Memtable.put(Memtable.java:210)
~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at
org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1230)
~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396)
~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:359)
~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at
org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54)
~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_74]
at
org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at
org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
ERROR [SharedPool-Worker-25] 2016-12-27 10:14:28,100
SEPWorker.java:141 - Failed to execute task, unexpected exception killed
worker: {}
java.lang.IllegalStateException: Shutdown in progress
at
java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82)
~[na:1.8.0_74]
at java.lang.Runtime.removeShutdownHook(Runtime.java:239)
~[na:1.8.0_74]
at
org.apache.cassandra.service.StorageService.removeShutdownHook(StorageService.java:764)
~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at
org.apache.cassandra.utils.JVMStabilityInspector$Killer.killCurrentJVM(JVMStabilityInspector.java:119)
~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at
org.apache.cassandra.utils.JVMStabilityInspector$Killer.killCurrentJVM(JVMStabilityInspector.java:109)
~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at
org.apache.cassandra.utils.JVMStabilityInspector.inspectThrowable(JVMStabilityInspector.java:68)
~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at
org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:168)
~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at
org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at
java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
INFO [Thread-6] 2016-12-27 10:14:56,150 DseDaemon.java:420 - DSE
shutting down...
