Re: cassandra full gc too often

2016-01-04 Thread Robert Coli
On Sun, Jan 3, 2016 at 5:54 PM, Shuo Chen wrote: > There are client operation in these days. Besides most columnfamily in the > cluster are supercolumnfamily created by cassandra-cli. Most rows have > average 30 sub-rows and each sub-row has 20 columns. > Supercolumns,

Re: cassandra full gc too often

2016-01-03 Thread Shuo Chen
The following is the snippets of nodetool compactions history. It shows many records relates to keyspace system. Compaction History: id keyspace_name columnfamily_namecompacted_at bytes_in bytes_out rows_merged

cassandra full gc too often

2015-12-31 Thread Shuo Chen
I have a cassandra 2.0.6 cluster with four nodes as backup database. The only operation is posting data into db. Recently, the full gc of the nodes increases apparently and blocks cluster operation. The load of each node is 10G. The heap is 8G each with default jvm memory settings. The cpu is 24

Re: cassandra full gc too often

2015-12-31 Thread Ipremyadav
Simplest option is to use java 8 with G1 gc. > On 31 Dec 2015, at 10:23 a.m., Shuo Chen wrote: > > I have a cassandra 2.0.6 cluster with four nodes as backup database. The only > operation is posting data into db. Recently, the full gc of the nodes > increases

Re: cassandra full gc too often

2015-12-31 Thread Graham Sanderson
If you are lucky that might mask the real issue, but I doubt it… that is an insane number of compaction tasks and indicative of another problem. I would check release notes of 2.0.6+, if I recall that was not a stable version and may have had leaks. Aside from that, just FYI, if you use

Re: full gc too often

2014-12-07 Thread Philo Yang
2014-12-05 15:40 GMT+08:00 Jonathan Haddad j...@jonhaddad.com: I recommend reading through https://issues.apache.org/jira/browse/CASSANDRA-8150 to get an idea of how the JVM GC works and what you can do to tune it. Also good is Blake Eggleston's writeup which can be found here:

Re: full gc too often

2014-12-07 Thread Jonathan Haddad
There's a lot of factors that go into tuning, and I don't know of any reliable formula that you can use to figure out what's going to work optimally for your hardware. Personally I recommend: 1) find the bottleneck 2) playing with a parameter (or two) 3) see what changed, performance wise If

Re: full gc too often

2014-12-05 Thread Robert Coli
On Thu, Dec 4, 2014 at 8:13 PM, Philo Yang ud1...@gmail.com wrote: In each time Old Gen reduce only a little, Survivor Space will be clear but the heap is still full so there will be another full gc very soon then the node will down. If I restart the node, it will be fine without gc trouble.

full gc too often

2014-12-04 Thread Philo Yang
Hi,all I have a cluster on C* 2.1.1 and jdk 1.7_u51. I have a trouble with full gc that sometime there may be one or two nodes full gc more than one time per minute and over 10 seconds each time, then the node will be unreachable and the latency of cluster will be increased. I grep the

Re: full gc too often

2014-12-04 Thread Tim Heckman
On Dec 4, 2014 8:14 PM, Philo Yang ud1...@gmail.com wrote: Hi,all I have a cluster on C* 2.1.1 and jdk 1.7_u51. I have a trouble with full gc that sometime there may be one or two nodes full gc more than one time per minute and over 10 seconds each time, then the node will be unreachable and

Re: full gc too often

2014-12-04 Thread Philo Yang
I have two kinds of machine: 16G RAM, with default heap size setting, about 4G. 64G RAM, with default heap size setting, about 8G. These two kinds of nodes have same number of vnodes, and both of them have gc issue, although the node of 16G have a higher probability of gc issue. Thanks, Philo

Re: full gc too often

2014-12-04 Thread Jonathan Haddad
I recommend reading through https://issues.apache.org/jira/browse/CASSANDRA-8150 to get an idea of how the JVM GC works and what you can do to tune it. Also good is Blake Eggleston's writeup which can be found here: http://blakeeggleston.com/cassandra-tuning-the-jvm-for-read-heavy-workloads.html