[
https://issues.apache.org/jira/browse/CASSANDRA-2543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13023236#comment-13023236
]
Jonathan Ellis commented on CASSANDRA-2543:
-------------------------------------------
bq. It still interest me why the node didn't recover even though there were
nearly no requests at all from our application
the two big users of memory are usually memtables and cache and neither one
gets freed when requests stop. (until memtable_flush_after_mins is reached...
and flush needs memory to work so if you are GC storming hard enough it may not
make progress even then.)
> Node not responding, bringing down cluster, marked as up
> --------------------------------------------------------
>
> Key: CASSANDRA-2543
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2543
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.4
> Reporter: Thibaut
> Fix For: 0.7.6
>
> Attachments: jstack
>
>
> I have one node which constantly hangs and brings done the entire cluster
> (not giving any answers).
> If I restart the node, the node will hang after a certain number of time. I
> have no indication
> It's marked as up when executing the nodetool ring command.
> Executing the ring command on the node itself (without any traffic on the
> cluster) takes at least 2 minutes to execute. The node takes about 50%-100%
> of cpu over all cpus.
> Netstats doesn't list anything interesting:
> /software/cassandra/bin/nodetool -h localhost netstats
> Mode: Normal
> Not sending any streams.
> Not receiving any streams.
> Pool Name Active Pending Completed
> Commands n/a 0 51064
> Responses n/a 0 530479
> I attached the jstack of the node. There are no indications that the node has
> faulty hardware.
> /usr/bin/java -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42
> -Xms5254M -Xmx5254M -Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss128k
> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
> -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=8080
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dlog4j.configuration=log4j-server.properties
> -Dlog4j.defaultInitOverride=true -Dcassandra-foreground=yes -cp
> /software/cassandra/bin/../conf:/software/cassandra/bin/../build/classes:/software/cassandra/bin/../lib/antlr-3.1.3.jar:/software/cassandra/bin/../lib/apache-cassandra-0.7.4.jar:/software/cassandra/bin/../lib/avro-1.4.0-fixes.jar:/software/cassandra/bin/../lib/avro-1.4.0-sources-fixes.jar:/software/cassandra/bin/../lib/commons-cli-1.1.jar:/software/cassandra/bin/../lib/commons-codec-1.2.jar:/software/cassandra/bin/../lib/commons-collections-3.2.1.jar:/software/cassandra/bin/../lib/commons-lang-2.4.jar:/software/cassandra/bin/../lib/concurrentlinkedhashmap-lru-1.1.jar:/software/cassandra/bin/../lib/guava-r05.jar:/software/cassandra/bin/../lib/high-scale-lib.jar:/software/cassandra/bin/../lib/jackson-core-asl-1.4.0.jar:/software/cassandra/bin/../lib/jackson-mapper-asl-1.4.0.jar:/software/cassandra/bin/../lib/jetty-6.1.21.jar:/software/cassandra/bin/../lib/jetty-util-6.1.21.jar:/software/cassandra/bin/../lib/jline-0.9.94.jar:/software/cassandra/bin/../lib/json-simple-1.1.jar:/software/cassandra/bin/../lib/jug-2.0.0.jar:/software/cassandra/bin/../lib/libthrift-0.5.jar:/software/cassandra/bin/../lib/log4j-1.2.16.jar:/software/cassandra/bin/../lib/servlet-api-2.5-20081211.jar:/software/cassandra/bin/../lib/slf4j-api-1.6.1.jar:/software/cassandra/bin/../lib/slf4j-log4j12-1.6.1.jar:/software/cassandra/bin/../lib/snakeyaml-1.6.jar
> org.apache.cassandra.thrift.CassandraDaemon
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
