You can verify unevenly data distribution using nodetool command *nodetool toppartitions keyspace table*
On Mon, Jan 23, 2017 at 12:52 PM, chetan kumar <txtti...@gmail.com> wrote: > Hi Pranay, > > i seems that your data is unevenly distributed across the cluster with > respect your insertion frequency.Please restructure your partition key > > Thanks > > On Fri, Jan 20, 2017 at 6:49 AM, Pranay akula <pranay.akula2...@gmail.com> > wrote: > >> what i have observed is 2-3 old gen GC's in 1-2 mins before OOM which i >> rarely see and seen hinted handoffs get accumulated on nodes which went >> down, and Mutation drops as well. >> >> i really don't know how to analyse hprof file is there any guide or blog >> that can help me how to analyse it ?? our cluster has 2 DC's each DC with >> 18 nodes each and 12 GB Heap and 4 GB new Heap. >> >> >> Thanks >> Pranay. >> >> On Thu, Jan 19, 2017 at 8:19 AM, Alain RODRIGUEZ <arodr...@gmail.com> >> wrote: >> >>> Hi Pranay, >>> >>> what can be the reason for this >>> >>> >>> It can be due to a JVM / GC misconfiguration or to some abnormal >>> activity in Cassandra. Often, GC issues are a consequences and not the root >>> cause of an issue in Cassandra. >>> >>> >>>> how to debug that ?? >>> >>> how to fine grain why on those particular nodes this is happening when >>>> these nodes are serving same requests like rest of the cluster ?? >>> >>> >>> You can enable GC logs on those nodes (use the cassandra-env.sh file to >>> do so) and have a look at what's happening there. Also you can have a look >>> at the system.log files (search for warning or errors - WARN / ERROR) and >>> at "nodetool tpstats". I like to use this last command as follow "watch -d >>> nodetool tpstats" to see variations. >>> >>> Having pending or dropped threads is likely to increase the GC activity. >>> As well as having wide rows, many tomstones and some other cases. >>> >>> So to determine why this is happening, could you share your hardware >>> specs, the way JVM / GC is configured (cassandra-env.sh) and let us know >>> how nodes are handling threads and about any relevant infrmation that might >>> be appearing in the logs. >>> >>> You can investigate the heap dump as well (I believe you can do this >>> using Eclipse Memory Analyzer - MAT). >>> >>> C*heers, >>> ----------------------- >>> Alain Rodriguez - @arodream - al...@thelastpickle.com >>> France >>> >>> The Last Pickle - Apache Cassandra Consulting >>> http://www.thelastpickle.com >>> >>> 2017-01-19 14:00 GMT+01:00 Pranay akula <pranay.akula2...@gmail.com>: >>> >>>> From last few days i am seeing on some of the nodes in cassandra >>>> cluster DSE is getting shutdown due to the error below and i need to kill >>>> Java process and restart DSE service. >>>> >>>> I have cross checked reads and writes and compactions nothing looks >>>> suspicious, but i am seeing full Gc pause on these server just before the >>>> issue happening. what can be the reason for this how to debug that ?? how >>>> to fine grain why on those particular nodes this is happening when these >>>> nodes are serving same requests like rest of the cluster ?? >>>> >>>> Is this happening because of Full Gc is not getting performed properly, >>>> we using G1GC and DSE 4.8.3 >>>> >>>> >>>> ERROR [SharedPool-Worker-25] 2016-12-27 10:14:26,100 >>>> JVMStabilityInspector.java:117 - JVM state determined to be >>>> unstable. Exiting forcefully due to:java.lang.OutOfMemoryError: Java heap >>>> space >>>> >>>> at java.util.Arrays.copyOf(Arrays.java:3181) ~[na:1.8.0_74] >>>> >>>> at >>>> org.apache.cassandra.db.RangeTombstoneList.copy(RangeTombstoneList.java:112) >>>> ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] >>>> >>>> at org.apache.cassandra.db.Deleti >>>> onInfo.copy(DeletionInfo.java:104) ~[cassandra-all-2.1.13.1131.ja >>>> r:2.1.13.1131] >>>> >>>> at org.apache.cassandra.db.Atomic >>>> BTreeColumns.addAllWithSizeDelta(AtomicBTreeColumns.java:217) >>>> ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] >>>> >>>> at org.apache.cassandra.db.Memtable.put(Memtable.java:210) >>>> ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] >>>> >>>> at org.apache.cassandra.db.Column >>>> FamilyStore.apply(ColumnFamilyStore.java:1230) >>>> ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] >>>> >>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396) >>>> ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] >>>> >>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:359) >>>> ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] >>>> >>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) >>>> ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] >>>> >>>> at org.apache.cassandra.db.Mutati >>>> onVerbHandler.doVerb(MutationVerbHandler.java:54) >>>> ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] >>>> >>>> at org.apache.cassandra.net.Messa >>>> geDeliveryTask.run(MessageDeliveryTask.java:64) >>>> ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] >>>> >>>> at java.util.concurrent.Executors >>>> $RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_74] >>>> >>>> at org.apache.cassandra.concurren >>>> t.AbstractTracingAwareExecutorService$FutureTask.run(Abstrac >>>> tTracingAwareExecutorService.java:164) ~[cassandra-all-2.1.13.1131.ja >>>> r:2.1.13.1131] >>>> >>>> at org.apache.cassandra.concurren >>>> t.SEPWorker.run(SEPWorker.java:105) [cassandra-all-2.1.13.1131.jar >>>> :2.1.13.1131] >>>> >>>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74] >>>> >>>> >>>> ERROR [SharedPool-Worker-25] 2016-12-27 10:14:28,100 >>>> SEPWorker.java:141 - Failed to execute task, unexpected exception killed >>>> worker: {} >>>> >>>> java.lang.IllegalStateException: Shutdown in progress >>>> >>>> at java.lang.ApplicationShutdownH >>>> ooks.remove(ApplicationShutdownHooks.java:82) ~[na:1.8.0_74] >>>> >>>> at java.lang.Runtime.removeShutdownHook(Runtime.java:239) >>>> ~[na:1.8.0_74] >>>> >>>> at org.apache.cassandra.service.S >>>> torageService.removeShutdownHook(StorageService.java:764) >>>> ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] >>>> >>>> at org.apache.cassandra.utils.JVM >>>> StabilityInspector$Killer.killCurrentJVM(JVMStabilityInspector.java:119) >>>> ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] >>>> >>>> at org.apache.cassandra.utils.JVM >>>> StabilityInspector$Killer.killCurrentJVM(JVMStabilityInspector.java:109) >>>> ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] >>>> >>>> at org.apache.cassandra.utils.JVM >>>> StabilityInspector.inspectThrowable(JVMStabilityInspector.java:68) >>>> ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] >>>> >>>> at org.apache.cassandra.concurren >>>> t.AbstractTracingAwareExecutorService$FutureTask.run(Abstrac >>>> tTracingAwareExecutorService.java:168) ~[cassandra-all-2.1.13.1131.ja >>>> r:2.1.13.1131] >>>> >>>> at >>>> >>>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) >>>> ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] >>>> >>>> at >>>> >>>> java.lang.Thread.run(Thread.java:745) [na:1.8.0_74] >>>> >>>> >>>> INFO [Thread-6] 2016-12-27 10:14:56,150 DseDaemon.java:420 - DSE >>>> shutting down... >>>> >>> >>> >> >
