Hi Pranay, what can be the reason for this
It can be due to a JVM / GC misconfiguration or to some abnormal activity in Cassandra. Often, GC issues are a consequences and not the root cause of an issue in Cassandra. > how to debug that ?? how to fine grain why on those particular nodes this is happening when > these nodes are serving same requests like rest of the cluster ?? You can enable GC logs on those nodes (use the cassandra-env.sh file to do so) and have a look at what's happening there. Also you can have a look at the system.log files (search for warning or errors - WARN / ERROR) and at "nodetool tpstats". I like to use this last command as follow "watch -d nodetool tpstats" to see variations. Having pending or dropped threads is likely to increase the GC activity. As well as having wide rows, many tomstones and some other cases. So to determine why this is happening, could you share your hardware specs, the way JVM / GC is configured (cassandra-env.sh) and let us know how nodes are handling threads and about any relevant infrmation that might be appearing in the logs. You can investigate the heap dump as well (I believe you can do this using Eclipse Memory Analyzer - MAT). C*heers, ----------------------- Alain Rodriguez - @arodream - al...@thelastpickle.com France The Last Pickle - Apache Cassandra Consulting http://www.thelastpickle.com 2017-01-19 14:00 GMT+01:00 Pranay akula <pranay.akula2...@gmail.com>: > From last few days i am seeing on some of the nodes in cassandra cluster > DSE is getting shutdown due to the error below and i need to kill Java > process and restart DSE service. > > I have cross checked reads and writes and compactions nothing looks > suspicious, but i am seeing full Gc pause on these server just before the > issue happening. what can be the reason for this how to debug that ?? how > to fine grain why on those particular nodes this is happening when these > nodes are serving same requests like rest of the cluster ?? > > Is this happening because of Full Gc is not getting performed properly, we > using G1GC and DSE 4.8.3 > > > ERROR [SharedPool-Worker-25] 2016-12-27 10:14:26,100 > JVMStabilityInspector.java:117 - JVM state determined to be unstable. > Exiting forcefully due to:java.lang.OutOfMemoryError: Java heap space > > at java.util.Arrays.copyOf(Arrays.java:3181) ~[na:1.8.0_74] > > at > org.apache.cassandra.db.RangeTombstoneList.copy(RangeTombstoneList.java:112) > ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] > > at org.apache.cassandra.db.Deleti > onInfo.copy(DeletionInfo.java:104) ~[cassandra-all-2.1.13.1131.ja > r:2.1.13.1131] > > at org.apache.cassandra.db.Atomic > BTreeColumns.addAllWithSizeDelta(AtomicBTreeColumns.java:217) > ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] > > at org.apache.cassandra.db.Memtable.put(Memtable.java:210) > ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] > > at org.apache.cassandra.db.Column > FamilyStore.apply(ColumnFamilyStore.java:1230) > ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] > > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396) > ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] > > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:359) > ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] > > at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) > ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] > > at org.apache.cassandra.db.Mutati > onVerbHandler.doVerb(MutationVerbHandler.java:54) > ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] > > at org.apache.cassandra.net.Messa > geDeliveryTask.run(MessageDeliveryTask.java:64) > ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] > > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_74] > > at org.apache.cassandra.concurren > t.AbstractTracingAwareExecutorService$FutureTask.run(Abstrac > tTracingAwareExecutorService.java:164) ~[cassandra-all-2.1.13.1131.ja > r:2.1.13.1131] > > at > org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [cassandra-all-2.1.13.1131.jar:2.1.13.1131] > > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74] > > > ERROR [SharedPool-Worker-25] 2016-12-27 10:14:28,100 > SEPWorker.java:141 - Failed to execute task, unexpected exception killed > worker: {} > > java.lang.IllegalStateException: Shutdown in progress > > at java.lang.ApplicationShutdownH > ooks.remove(ApplicationShutdownHooks.java:82) ~[na:1.8.0_74] > > at java.lang.Runtime.removeShutdownHook(Runtime.java:239) > ~[na:1.8.0_74] > > at org.apache.cassandra.service.S > torageService.removeShutdownHook(StorageService.java:764) > ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] > > at org.apache.cassandra.utils.JVM > StabilityInspector$Killer.killCurrentJVM(JVMStabilityInspector.java:119) > ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] > > at org.apache.cassandra.utils.JVM > StabilityInspector$Killer.killCurrentJVM(JVMStabilityInspector.java:109) > ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] > > at org.apache.cassandra.utils.JVM > StabilityInspector.inspectThrowable(JVMStabilityInspector.java:68) > ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] > > at org.apache.cassandra.concurren > t.AbstractTracingAwareExecutorService$FutureTask.run(Abstrac > tTracingAwareExecutorService.java:168) ~[cassandra-all-2.1.13.1131.ja > r:2.1.13.1131] > > at > > org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] > > at > > java.lang.Thread.run(Thread.java:745) [na:1.8.0_74] > > > INFO [Thread-6] 2016-12-27 10:14:56,150 DseDaemon.java:420 - DSE > shutting down... >
