R: Re: Hints handoff is memory intensive

[Romain Hardouin]

+1 for G1 with such a heap size. And don't set heap new size, let g1 decide.
Inviato da Yahoo Mail su Android 
 
  Il mar, 20 set, 2016 alle 9:01, Jeff Jirsa ha scritto:   
Major changes in 3.0, but the big ones involve storage - parent ticket is 
https://issues.apache.org/jira/browse/CASSANDRA-9427

64G cms heap with 4G new gen is somewhat atypical - would expect g1 at that 
size, or at the very least a larger new gen to try to avoid promotion which 
will bring you that parnew pause. 



-- 
Jeff Jirsa


> On Sep 19, 2016, at 11:12 PM, Dikang Gu  wrote:
> 
> In our 2.1 cluster, I find that hints handoff is using a lot of memory on
> our proxy nodes, when delivering hints to a data node that was dead for 3+
> hours (our hints window is 3 hours). It makes the young gen GC time as long
> as 2 secs.
> 
> I'm using 64G max heap size, and 4G young gen size. I'm considering
> increasing the young gen size, I'm wondering any improvements to hints
> handoff in newer version?
> 
> some logs:
> 2016-09-20_05:04:30.67928 INFO  05:04:30 [Service Thread]: ParNew GC in
> 2200ms.  CMS Old Gen: 49112623080 -> 50052787792; Par Eden Space:
> 2863398912 -> 0;
> 
> mem usage:
> 2016-09-20T01:25:23.522+ Process summary
>  process cpu=0.00%
>  application cpu=140.36% (user=132.28% sys=8.08%)
>  other: cpu=-140.36%
>  heap allocation rate 1500mb/s
> [001343] user=90.42% sys= 0.01% alloc= 1439mb/s - HintedHandoff:3
> [290234] user=18.85% sys= 5.14% alloc=  40mb/s - CompactionExecutor:176226
> [064445] user=17.57% sys= 1.09% alloc=  10mb/s - RMI TCP
> Connection(148376)-127.0.0.1
> [290439] user= 3.20% sys= 1.10% alloc= 4607kb/s - RMI TCP
> Connection(148380)-127.0.0.1
> [16] user= 0.75% sys=-0.03% alloc= 2321kb/s - ScheduledTasks:1
> [000149] user= 0.32% sys=-0.04% alloc= 1516kb/s - GossipStage:1
> [001370] user= 0.11% sys= 0.04% alloc=  420kb/s - GossipTasks:1
> 
> Thanks
> Dikang.  

Re: Hints handoff is memory intensive

[Jeff Jirsa]

Major changes in 3.0, but the big ones involve storage - parent ticket is 
https://issues.apache.org/jira/browse/CASSANDRA-9427

64G cms heap with 4G new gen is somewhat atypical - would expect g1 at that 
size, or at the very least a larger new gen to try to avoid promotion which 
will bring you that parnew pause. 



-- 
Jeff Jirsa


> On Sep 19, 2016, at 11:12 PM, Dikang Gu  wrote:
> 
> In our 2.1 cluster, I find that hints handoff is using a lot of memory on
> our proxy nodes, when delivering hints to a data node that was dead for 3+
> hours (our hints window is 3 hours). It makes the young gen GC time as long
> as 2 secs.
> 
> I'm using 64G max heap size, and 4G young gen size. I'm considering
> increasing the young gen size, I'm wondering any improvements to hints
> handoff in newer version?
> 
> some logs:
> 2016-09-20_05:04:30.67928 INFO  05:04:30 [Service Thread]: ParNew GC in
> 2200ms.  CMS Old Gen: 49112623080 -> 50052787792; Par Eden Space:
> 2863398912 -> 0;
> 
> mem usage:
> 2016-09-20T01:25:23.522+ Process summary
>  process cpu=0.00%
>  application cpu=140.36% (user=132.28% sys=8.08%)
>  other: cpu=-140.36%
>  heap allocation rate 1500mb/s
> [001343] user=90.42% sys= 0.01% alloc= 1439mb/s - HintedHandoff:3
> [290234] user=18.85% sys= 5.14% alloc=   40mb/s - CompactionExecutor:176226
> [064445] user=17.57% sys= 1.09% alloc=   10mb/s - RMI TCP
> Connection(148376)-127.0.0.1
> [290439] user= 3.20% sys= 1.10% alloc= 4607kb/s - RMI TCP
> Connection(148380)-127.0.0.1
> [16] user= 0.75% sys=-0.03% alloc= 2321kb/s - ScheduledTasks:1
> [000149] user= 0.32% sys=-0.04% alloc= 1516kb/s - GossipStage:1
> [001370] user= 0.11% sys= 0.04% alloc=  420kb/s - GossipTasks:1
> 
> Thanks
> Dikang.

Re: Hints handoff is memory intensive

[Eduard Tudenhoefner]

The way how hints work under the hood was improved in C* 3.0.

Please take a look at
http://www.datastax.com/dev/blog/whats-coming-to-cassandra-in-3-0-improved-hint-storage-and-delivery
for further details

On Tue, Sep 20, 2016 at 8:12 AM, Dikang Gu  wrote:

> In our 2.1 cluster, I find that hints handoff is using a lot of memory on
> our proxy nodes, when delivering hints to a data node that was dead for 3+
> hours (our hints window is 3 hours). It makes the young gen GC time as long
> as 2 secs.
>
> I'm using 64G max heap size, and 4G young gen size. I'm considering
> increasing the young gen size, I'm wondering any improvements to hints
> handoff in newer version?
>
> some logs:
> 2016-09-20_05:04:30.67928 INFO  05:04:30 [Service Thread]: ParNew GC in
> 2200ms.  CMS Old Gen: 49112623080 -> 50052787792; Par Eden Space:
> 2863398912 -> 0;
>
> mem usage:
> 2016-09-20T01:25:23.522+ Process summary
>   process cpu=0.00%
>   application cpu=140.36% (user=132.28% sys=8.08%)
>   other: cpu=-140.36%
>   heap allocation rate 1500mb/s
> [001343] user=90.42% sys= 0.01% alloc= 1439mb/s - HintedHandoff:3
> [290234] user=18.85% sys= 5.14% alloc=   40mb/s - CompactionExecutor:176226
> [064445] user=17.57% sys= 1.09% alloc=   10mb/s - RMI TCP
> Connection(148376)-127.0.0.1
> [290439] user= 3.20% sys= 1.10% alloc= 4607kb/s - RMI TCP
> Connection(148380)-127.0.0.1
> [16] user= 0.75% sys=-0.03% alloc= 2321kb/s - ScheduledTasks:1
> [000149] user= 0.32% sys=-0.04% alloc= 1516kb/s - GossipStage:1
> [001370] user= 0.11% sys= 0.04% alloc=  420kb/s - GossipTasks:1
>
> Thanks
> Dikang.
>



-- 

[image: DataStaxLogo copy3.png] 

Eduard Tudenhoefner

Software Engineer | +49 151 206 111 97 | eduard.tudenhoef...@datastax.com



 

 

Hints handoff is memory intensive

[Dikang Gu]

In our 2.1 cluster, I find that hints handoff is using a lot of memory on
our proxy nodes, when delivering hints to a data node that was dead for 3+
hours (our hints window is 3 hours). It makes the young gen GC time as long
as 2 secs.

I'm using 64G max heap size, and 4G young gen size. I'm considering
increasing the young gen size, I'm wondering any improvements to hints
handoff in newer version?

some logs:
2016-09-20_05:04:30.67928 INFO  05:04:30 [Service Thread]: ParNew GC in
2200ms.  CMS Old Gen: 49112623080 -> 50052787792; Par Eden Space:
2863398912 -> 0;

mem usage:
2016-09-20T01:25:23.522+ Process summary
  process cpu=0.00%
  application cpu=140.36% (user=132.28% sys=8.08%)
  other: cpu=-140.36%
  heap allocation rate 1500mb/s
[001343] user=90.42% sys= 0.01% alloc= 1439mb/s - HintedHandoff:3
[290234] user=18.85% sys= 5.14% alloc=   40mb/s - CompactionExecutor:176226
[064445] user=17.57% sys= 1.09% alloc=   10mb/s - RMI TCP
Connection(148376)-127.0.0.1
[290439] user= 3.20% sys= 1.10% alloc= 4607kb/s - RMI TCP
Connection(148380)-127.0.0.1
[16] user= 0.75% sys=-0.03% alloc= 2321kb/s - ScheduledTasks:1
[000149] user= 0.32% sys=-0.04% alloc= 1516kb/s - GossipStage:1
[001370] user= 0.11% sys= 0.04% alloc=  420kb/s - GossipTasks:1

Thanks
Dikang.