On Dec 4, 2014 8:14 PM, "Philo Yang" <ud1...@gmail.com> wrote: > > Hi,all > > I have a cluster on C* 2.1.1 and jdk 1.7_u51. I have a trouble with full gc that sometime there may be one or two nodes full gc more than one time per minute and over 10 seconds each time, then the node will be unreachable and the latency of cluster will be increased. > > I grep the GCInspector's log, I found when the node is running fine without gc trouble there are two kinds of gc: > ParNew GC in less than 300ms which clear the Par Eden Space and enlarge CMS Old Gen/ Par Survivor Space little (because it only show gc in more than 200ms, there is only a small number of ParNew GC in log) > ConcurrentMarkSweep in 4000~8000ms which reduce CMS Old Gen much and enlarge Par Eden Space little, each 1-2 hours it will be executed once. > > However, sometimes ConcurrentMarkSweep will be strange like it shows: > > INFO [Service Thread] 2014-12-05 11:28:44,629 GCInspector.java:142 - ConcurrentMarkSweep GC in 12648ms. CMS Old Gen: 3579838424 -> 3579838464; Par Eden Space: 503316480 -> 294794576; Par Survivor Space: 62914528 -> 0 > INFO [Service Thread] 2014-12-05 11:28:59,581 GCInspector.java:142 - ConcurrentMarkSweep GC in 12227ms. CMS Old Gen: 3579838464 -> 3579836512; Par Eden Space: 503316480 -> 310562032; Par Survivor Space: 62872496 -> 0 > INFO [Service Thread] 2014-12-05 11:29:14,686 GCInspector.java:142 - ConcurrentMarkSweep GC in 11538ms. CMS Old Gen: 3579836688 -> 3579805792; Par Eden Space: 503316480 -> 332391096; Par Survivor Space: 62914544 -> 0 > INFO [Service Thread] 2014-12-05 11:29:29,371 GCInspector.java:142 - ConcurrentMarkSweep GC in 12180ms. CMS Old Gen: 3579835784 -> 3579829760; Par Eden Space: 503316480 -> 351991456; Par Survivor Space: 62914552 -> 0 > INFO [Service Thread] 2014-12-05 11:29:45,028 GCInspector.java:142 - ConcurrentMarkSweep GC in 10574ms. CMS Old Gen: 3579838112 -> 3579799752; Par Eden Space: 503316480 -> 366222584; Par Survivor Space: 62914560 -> 0 > INFO [Service Thread] 2014-12-05 11:29:59,546 GCInspector.java:142 - ConcurrentMarkSweep GC in 11594ms. CMS Old Gen: 3579831424 -> 3579817392; Par Eden Space: 503316480 -> 388702928; Par Survivor Space: 62914552 -> 0 > INFO [Service Thread] 2014-12-05 11:30:14,153 GCInspector.java:142 - ConcurrentMarkSweep GC in 11463ms. CMS Old Gen: 3579817392 -> 3579838424; Par Eden Space: 503316480 -> 408992784; Par Survivor Space: 62896720 -> 0 > INFO [Service Thread] 2014-12-05 11:30:25,009 GCInspector.java:142 - ConcurrentMarkSweep GC in 9576ms. CMS Old Gen: 3579838424 -> 3579816424; Par Eden Space: 503316480 -> 438633608; Par Survivor Space: 62914544 -> 0 > INFO [Service Thread] 2014-12-05 11:30:39,929 GCInspector.java:142 - ConcurrentMarkSweep GC in 11556ms. CMS Old Gen: 3579816424 -> 3579785496; Par Eden Space: 503316480 -> 441354856; Par Survivor Space: 62889528 -> 0 > INFO [Service Thread] 2014-12-05 11:30:54,085 GCInspector.java:142 - ConcurrentMarkSweep GC in 12082ms. CMS Old Gen: 3579786592 -> 3579814464; Par Eden Space: 503316480 -> 448782440; Par Survivor Space: 62914560 -> 0 > > In each time Old Gen reduce only a little, Survivor Space will be clear but the heap is still full so there will be another full gc very soon then the node will down. If I restart the node, it will be fine without gc trouble. > > Can anyone help me to find out where is the problem that full gc can't reduce CMS Old Gen? Is it because there are too many objects in heap can't be recycled? I think review the table scheme designing and add new nodes into cluster is a good idea, but I still want to know if there is any other reason causing this trouble.
How much total system memory do you have? How much is allocated for heap usage? How big is your working data set? The reason I ask is that I've seen problems with lots of GC with no room gained, and it was memory pressure. Not enough for the heap. We decided that just increasing the heap size was a bad idea, as we did rely on free RAM being used for filesystem caching. So some vertical and horizontal scaling allowed us to give Cass more heap space, as well as distribute the workload to try and avoid further problems. > Thanks, > Philo Yang Cheers! -Tim
