Hints handoff is memory intensive

[Dikang Gu]

In our 2.1 cluster, I find that hints handoff is using a lot of memory on
our proxy nodes, when delivering hints to a data node that was dead for 3+
hours (our hints window is 3 hours). It makes the young gen GC time as long
as 2 secs.

I'm using 64G max heap size, and 4G young gen size. I'm considering
increasing the young gen size, I'm wondering any improvements to hints
handoff in newer version?

some logs:
2016-09-20_05:04:30.67928 INFO  05:04:30 [Service Thread]: ParNew GC in
2200ms.  CMS Old Gen: 49112623080 -> 50052787792; Par Eden Space:
2863398912 -> 0;

mem usage:
2016-09-20T01:25:23.522+ Process summary
  process cpu=0.00%
  application cpu=140.36% (user=132.28% sys=8.08%)
  other: cpu=-140.36%
  heap allocation rate 1500mb/s
[001343] user=90.42% sys= 0.01% alloc= 1439mb/s - HintedHandoff:3
[290234] user=18.85% sys= 5.14% alloc=   40mb/s - CompactionExecutor:176226
[064445] user=17.57% sys= 1.09% alloc=   10mb/s - RMI TCP
Connection(148376)-127.0.0.1
[290439] user= 3.20% sys= 1.10% alloc= 4607kb/s - RMI TCP
Connection(148380)-127.0.0.1
[16] user= 0.75% sys=-0.03% alloc= 2321kb/s - ScheduledTasks:1
[000149] user= 0.32% sys=-0.04% alloc= 1516kb/s - GossipStage:1
[001370] user= 0.11% sys= 0.04% alloc=  420kb/s - GossipTasks:1

Thanks
Dikang.

Re: Hints handoff is memory intensive

[Jeff Jirsa]

Major changes in 3.0, but the big ones involve storage - parent ticket is 
https://issues.apache.org/jira/browse/CASSANDRA-9427

64G cms heap with 4G new gen is somewhat atypical - would expect g1 at that 
size, or at the very least a larger new gen to try to avoid promotion which 
will bring you that parnew pause. 



-- 
Jeff Jirsa


> On Sep 19, 2016, at 11:12 PM, Dikang Gu  wrote:
> 
> In our 2.1 cluster, I find that hints handoff is using a lot of memory on
> our proxy nodes, when delivering hints to a data node that was dead for 3+
> hours (our hints window is 3 hours). It makes the young gen GC time as long
> as 2 secs.
> 
> I'm using 64G max heap size, and 4G young gen size. I'm considering
> increasing the young gen size, I'm wondering any improvements to hints
> handoff in newer version?
> 
> some logs:
> 2016-09-20_05:04:30.67928 INFO  05:04:30 [Service Thread]: ParNew GC in
> 2200ms.  CMS Old Gen: 49112623080 -> 50052787792; Par Eden Space:
> 2863398912 -> 0;
> 
> mem usage:
> 2016-09-20T01:25:23.522+ Process summary
>  process cpu=0.00%
>  application cpu=140.36% (user=132.28% sys=8.08%)
>  other: cpu=-140.36%
>  heap allocation rate 1500mb/s
> [001343] user=90.42% sys= 0.01% alloc= 1439mb/s - HintedHandoff:3
> [290234] user=18.85% sys= 5.14% alloc=   40mb/s - CompactionExecutor:176226
> [064445] user=17.57% sys= 1.09% alloc=   10mb/s - RMI TCP
> Connection(148376)-127.0.0.1
> [290439] user= 3.20% sys= 1.10% alloc= 4607kb/s - RMI TCP
> Connection(148380)-127.0.0.1
> [16] user= 0.75% sys=-0.03% alloc= 2321kb/s - ScheduledTasks:1
> [000149] user= 0.32% sys=-0.04% alloc= 1516kb/s - GossipStage:1
> [001370] user= 0.11% sys= 0.04% alloc=  420kb/s - GossipTasks:1
> 
> Thanks
> Dikang.

Re: Hints handoff is memory intensive

[Eduard Tudenhoefner]

The way how hints work under the hood was improved in C* 3.0.

Please take a look at
http://www.datastax.com/dev/blog/whats-coming-to-cassandra-in-3-0-improved-hint-storage-and-delivery
for further details

On Tue, Sep 20, 2016 at 8:12 AM, Dikang Gu  wrote:

> In our 2.1 cluster, I find that hints handoff is using a lot of memory on
> our proxy nodes, when delivering hints to a data node that was dead for 3+
> hours (our hints window is 3 hours). It makes the young gen GC time as long
> as 2 secs.
>
> I'm using 64G max heap size, and 4G young gen size. I'm considering
> increasing the young gen size, I'm wondering any improvements to hints
> handoff in newer version?
>
> some logs:
> 2016-09-20_05:04:30.67928 INFO  05:04:30 [Service Thread]: ParNew GC in
> 2200ms.  CMS Old Gen: 49112623080 -> 50052787792; Par Eden Space:
> 2863398912 -> 0;
>
> mem usage:
> 2016-09-20T01:25:23.522+ Process summary
>   process cpu=0.00%
>   application cpu=140.36% (user=132.28% sys=8.08%)
>   other: cpu=-140.36%
>   heap allocation rate 1500mb/s
> [001343] user=90.42% sys= 0.01% alloc= 1439mb/s - HintedHandoff:3
> [290234] user=18.85% sys= 5.14% alloc=   40mb/s - CompactionExecutor:176226
> [064445] user=17.57% sys= 1.09% alloc=   10mb/s - RMI TCP
> Connection(148376)-127.0.0.1
> [290439] user= 3.20% sys= 1.10% alloc= 4607kb/s - RMI TCP
> Connection(148380)-127.0.0.1
> [16] user= 0.75% sys=-0.03% alloc= 2321kb/s - ScheduledTasks:1
> [000149] user= 0.32% sys=-0.04% alloc= 1516kb/s - GossipStage:1
> [001370] user= 0.11% sys= 0.04% alloc=  420kb/s - GossipTasks:1
>
> Thanks
> Dikang.
>



-- 

[image: DataStaxLogo copy3.png] 

Eduard Tudenhoefner

Software Engineer | +49 151 206 111 97 | eduard.tudenhoef...@datastax.com



 

 

R: Re: Hints handoff is memory intensive

[Romain Hardouin]

+1 for G1 with such a heap size. And don't set heap new size, let g1 decide.
Inviato da Yahoo Mail su Android 
 
  Il mar, 20 set, 2016 alle 9:01, Jeff Jirsa ha scritto:   
Major changes in 3.0, but the big ones involve storage - parent ticket is 
https://issues.apache.org/jira/browse/CASSANDRA-9427

64G cms heap with 4G new gen is somewhat atypical - would expect g1 at that 
size, or at the very least a larger new gen to try to avoid promotion which 
will bring you that parnew pause. 



-- 
Jeff Jirsa


> On Sep 19, 2016, at 11:12 PM, Dikang Gu  wrote:
> 
> In our 2.1 cluster, I find that hints handoff is using a lot of memory on
> our proxy nodes, when delivering hints to a data node that was dead for 3+
> hours (our hints window is 3 hours). It makes the young gen GC time as long
> as 2 secs.
> 
> I'm using 64G max heap size, and 4G young gen size. I'm considering
> increasing the young gen size, I'm wondering any improvements to hints
> handoff in newer version?
> 
> some logs:
> 2016-09-20_05:04:30.67928 INFO  05:04:30 [Service Thread]: ParNew GC in
> 2200ms.  CMS Old Gen: 49112623080 -> 50052787792; Par Eden Space:
> 2863398912 -> 0;
> 
> mem usage:
> 2016-09-20T01:25:23.522+ Process summary
>  process cpu=0.00%
>  application cpu=140.36% (user=132.28% sys=8.08%)
>  other: cpu=-140.36%
>  heap allocation rate 1500mb/s
> [001343] user=90.42% sys= 0.01% alloc= 1439mb/s - HintedHandoff:3
> [290234] user=18.85% sys= 5.14% alloc=  40mb/s - CompactionExecutor:176226
> [064445] user=17.57% sys= 1.09% alloc=  10mb/s - RMI TCP
> Connection(148376)-127.0.0.1
> [290439] user= 3.20% sys= 1.10% alloc= 4607kb/s - RMI TCP
> Connection(148380)-127.0.0.1
> [16] user= 0.75% sys=-0.03% alloc= 2321kb/s - ScheduledTasks:1
> [000149] user= 0.32% sys=-0.04% alloc= 1516kb/s - GossipStage:1
> [001370] user= 0.11% sys= 0.04% alloc=  420kb/s - GossipTasks:1
> 
> Thanks
> Dikang.  

[RELEASE] Apache Cassandra 3.0.9 released

[Jake Luciani]

The Cassandra team is pleased to announce the release of Apache Cassandra
version 3.0.9.

Apache Cassandra is a fully distributed database. It is the right choice
when you need scalability and high availability without compromising
performance.

 http://cassandra.apache.org/

Downloads of source and binary distributions are listed in our download
section:

 http://cassandra.apache.org/download/

This version is a bug fix release[1] on the 3.0 series. As always, please
pay
attention to the release notes[2] and Let us know[3] if you were to
encounter
any problem.

Enjoy!

[1]: https://goo.gl/YfvFn8 (CHANGES.txt)
[2]: https://goo.gl/k9leqx (NEWS.txt)
[3]: https://issues.apache.org/jira/browse/CASSANDRA

Re: Proposal - 3.5.1

[Jonathan Haddad]

@Sylvain - I see what you're saying now on the branches.  I suppose a
branching strategy like that does give some flexibility to have multiple
things in the pipeline so it does give some additional flexibility there.


On Mon, Sep 19, 2016 at 9:06 AM Eric Evans 
wrote:

> On Fri, Sep 16, 2016 at 5:05 AM, Sylvain Lebresne 
> wrote:
> > In light of all this, my suggesting for a release cycle woud be:
> > - To have 3 branches: 'features', 'testing' and 'stable', with an X month
> >   rotation: 'features' becomes 'testing' after X months and then 'stable'
> > after
> >   X more, before getting EOL X months later.
> > - The feature branch gets everything. The testing branch only gets bug
> > fixes.
> >   The stable branch only gets critical bug fixes. And imo, we should be
> very
> >   strict on this (I acknowledge that there is sometimes a bit of
> > subjectivity on
> >   whether something is a bug or an improvement, and if it's critical or
> > not, but
> >   I think it's not that hard to get consensus if we're all reasonable
> > (though it
> >   might worth agreeing on some rough but written guideline upfront)).
> > - We release on a short and fixed cadence of Y month(s) for both the
> > feature and
> >   testing branch. For the stable branch, given that it already had X
> months
> > of
> >   only bug fixes during the testing phase, one can hope critical fixes
> will
> > be
> >   fairly rare, less than 1 per Y period on average). Further, it's
> supposed
> > to
> >   be stable and fixes are supposed to be critical, so doing hot-fix
> releases
> >   probably makes the most sense (though it probably only work if we're
> > indeed
> >   strict on what is considered critical).
>
> This seems pretty close to what Mck suggested; I think this could work.
>
>
> --
> Eric Evans
> john.eric.ev...@gmail.com
>