Re: Java GC pauses, reality check

[Kant Kodali]

@Harikrishnan Pillai

Thanks for that. What about the following?


"We are using g1GC in most clusters with *26GB heap* and extra threads
given to parallel and old gen collection. Those clusters 99% is also under
5 ms and doing good".

*So with G1GC you are able get under 5ms not the C4 (Zing's Garbage
Collector?)*

*What timeouts are you referring to ?*

On Mon, Nov 28, 2016 at 7:39 AM, Harikrishnan Pillai <
hpil...@walmartlabs.com> wrote:

> Hi @Kant Kodali,
>
> 11 /11 , 11 nodes in DC1 and 11 nodes in DC2.
>
>
> --
> *From:* Kant Kodali <k...@peernova.com>
> *Sent:* Monday, November 28, 2016 6:56 AM
>
> *To:* user@cassandra.apache.org
> *Subject:* Re: Java GC pauses, reality check
>
> Hi Hari,
>
> I am a little bit confused.
>
> What you mean 11/11 ?
>
> "We are using g1GC in most clusters with *26GB heap* and extra threads
> given to parallel and old gen collection. Those clusters 99% is also under
> 5 ms and doing good". So with G1GC you are able get under 5ms not the C4
> (Zing's Garbage Collector?)
>
> What timeouts are you referring to here?
>
> Thanks,
> kant
>
> On Sun, Nov 27, 2016 at 9:57 PM, Harikrishnan Pillai <
> hpil...@walmartlabs.com> wrote:
>
>> Hi @Kant Kodali,
>>
>> We have multiple clusters running zing .
>>
>> One cluster has 11/11 and another one also has 11/11.(190 GB mem,6TB hard
>> disk and 16 Physical core machines)
>>
>> The average read size is around 200KB and it can go upto 6 MB.
>>
>> We are using g1GC in most clusters with *26GB heap* and extra threads
>> given to parallel and old gen collection. Those clusters 99% is also under
>> 5 ms and doing good. We used Zing to remove all timeouts . If application
>> is not having that requirement G1GC is good.
>>
>> with g1gGC i have seen average 200-300 ms min pauses every 4 minutes and
>> 600 ms pauses every 6 hours and 99% latency is under 5-10 ms for most of
>> the clusters having 10- 100 KB of read data.
>>
>> Regards
>>
>> Hari
>> --
>> *From:* Kant Kodali <k...@peernova.com>
>> *Sent:* Saturday, November 26, 2016 8:39:01 PM
>> *To:* user@cassandra.apache.org
>> *Subject:* Re: Java GC pauses, reality check
>>
>> @Harikrishnan Pillai: How many nodes you guys are running? and what is an
>> approximate read size and an approximate write size?
>>
>> On Fri, Nov 25, 2016 at 7:32 PM, Harikrishnan Pillai <
>> hpil...@walmartlabs.com> wrote:
>>
>>> We are running azul zing in prod with 1 million reads/s and 100 K
>>> writes/s with azul .we never had a major gc above 10 ms .
>>>
>>> Sent from my iPhone
>>>
>>> > On Nov 25, 2016, at 3:49 PM, Martin Schröder <mar...@oneiros.de>
>>> wrote:
>>> >
>>> > 2016-11-25 23:38 GMT+01:00 Kant Kodali <k...@peernova.com>:
>>> >> I would also restate the following sentence "java GC pauses are
>>> pretty much
>>> >> a fact of life" to "Any GC based system pauses are pretty much a fact
>>> of
>>> >> life".
>>> >>
>>> >> I would be more than happy to see if someone can counter prove.
>>> >
>>> > Azul disagrees.
>>> > https://www.azul.com/products/zing/pgc/
>>> >
>>> > Best
>>> >   Martin
>>>
>>
>>
>

Re: Java GC pauses, reality check

[Harikrishnan Pillai]

Hi @Kant Kodali,

11 /11 , 11 nodes in DC1 and 11 nodes in DC2.



From: Kant Kodali <k...@peernova.com>
Sent: Monday, November 28, 2016 6:56 AM
To: user@cassandra.apache.org
Subject: Re: Java GC pauses, reality check

Hi Hari,

I am a little bit confused.

What you mean 11/11 ?

"We are using g1GC in most clusters with 26GB heap and extra threads given to 
parallel and old gen collection. Those clusters 99% is also under 5 ms and 
doing good". So with G1GC you are able get under 5ms not the C4 (Zing's Garbage 
Collector?)

What timeouts are you referring to here?

Thanks,
kant

On Sun, Nov 27, 2016 at 9:57 PM, Harikrishnan Pillai 
<hpil...@walmartlabs.com<mailto:hpil...@walmartlabs.com>> wrote:

Hi @Kant Kodali,

We have multiple clusters running zing .

One cluster has 11/11 and another one also has 11/11.(190 GB mem,6TB hard disk 
and 16 Physical core machines)

The average read size is around 200KB and it can go upto 6 MB.

We are using g1GC in most clusters with 26GB heap and extra threads given to 
parallel and old gen collection. Those clusters 99% is also under 5 ms and 
doing good. We used Zing to remove all timeouts . If application is not having 
that requirement G1GC is good.

with g1gGC i have seen average 200-300 ms min pauses every 4 minutes and 600 ms 
pauses every 6 hours and 99% latency is under 5-10 ms for most of the clusters 
having 10- 100 KB of read data.

Regards

Hari


From: Kant Kodali <k...@peernova.com<mailto:k...@peernova.com>>
Sent: Saturday, November 26, 2016 8:39:01 PM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Re: Java GC pauses, reality check

@Harikrishnan Pillai: How many nodes you guys are running? and what is an 
approximate read size and an approximate write size?

On Fri, Nov 25, 2016 at 7:32 PM, Harikrishnan Pillai 
<hpil...@walmartlabs.com<mailto:hpil...@walmartlabs.com>> wrote:
We are running azul zing in prod with 1 million reads/s and 100 K writes/s with 
azul .we never had a major gc above 10 ms .

Sent from my iPhone

> On Nov 25, 2016, at 3:49 PM, Martin Schröder 
> <mar...@oneiros.de<mailto:mar...@oneiros.de>> wrote:
>
> 2016-11-25 23:38 GMT+01:00 Kant Kodali 
> <k...@peernova.com<mailto:k...@peernova.com>>:
>> I would also restate the following sentence "java GC pauses are pretty much
>> a fact of life" to "Any GC based system pauses are pretty much a fact of
>> life".
>>
>> I would be more than happy to see if someone can counter prove.
>
> Azul disagrees.
> https://www.azul.com/products/zing/pgc/
>
> Best
>   Martin

Re: Java GC pauses, reality check

[Kant Kodali]

Hi Hari,

I am a little bit confused.

What you mean 11/11 ?

"We are using g1GC in most clusters with *26GB heap* and extra threads
given to parallel and old gen collection. Those clusters 99% is also under
5 ms and doing good". So with G1GC you are able get under 5ms not the C4
(Zing's Garbage Collector?)

What timeouts are you referring to here?

Thanks,
kant

On Sun, Nov 27, 2016 at 9:57 PM, Harikrishnan Pillai <
hpil...@walmartlabs.com> wrote:

> Hi @Kant Kodali,
>
> We have multiple clusters running zing .
>
> One cluster has 11/11 and another one also has 11/11.(190 GB mem,6TB hard
> disk and 16 Physical core machines)
>
> The average read size is around 200KB and it can go upto 6 MB.
>
> We are using g1GC in most clusters with *26GB heap* and extra threads
> given to parallel and old gen collection. Those clusters 99% is also under
> 5 ms and doing good. We used Zing to remove all timeouts . If application
> is not having that requirement G1GC is good.
>
> with g1gGC i have seen average 200-300 ms min pauses every 4 minutes and
> 600 ms pauses every 6 hours and 99% latency is under 5-10 ms for most of
> the clusters having 10- 100 KB of read data.
>
> Regards
>
> Hari
> --
> *From:* Kant Kodali <k...@peernova.com>
> *Sent:* Saturday, November 26, 2016 8:39:01 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Java GC pauses, reality check
>
> @Harikrishnan Pillai: How many nodes you guys are running? and what is an
> approximate read size and an approximate write size?
>
> On Fri, Nov 25, 2016 at 7:32 PM, Harikrishnan Pillai <
> hpil...@walmartlabs.com> wrote:
>
>> We are running azul zing in prod with 1 million reads/s and 100 K
>> writes/s with azul .we never had a major gc above 10 ms .
>>
>> Sent from my iPhone
>>
>> > On Nov 25, 2016, at 3:49 PM, Martin Schröder <mar...@oneiros.de> wrote:
>> >
>> > 2016-11-25 23:38 GMT+01:00 Kant Kodali <k...@peernova.com>:
>> >> I would also restate the following sentence "java GC pauses are pretty
>> much
>> >> a fact of life" to "Any GC based system pauses are pretty much a fact
>> of
>> >> life".
>> >>
>> >> I would be more than happy to see if someone can counter prove.
>> >
>> > Azul disagrees.
>> > https://www.azul.com/products/zing/pgc/
>> >
>> > Best
>> >   Martin
>>
>
>

Re: Java GC pauses, reality check

[Bill Hastings]

Hi Hari

Could you share your G1GC settings please?

On Sun, Nov 27, 2016 at 9:57 PM, Harikrishnan Pillai <
hpil...@walmartlabs.com> wrote:

> Hi @Kant Kodali,
>
> We have multiple clusters running zing .
>
> One cluster has 11/11 and another one also has 11/11.(190 GB mem,6TB hard
> disk and 16 Physical core machines)
>
> The average read size is around 200KB and it can go upto 6 MB.
>
> We are using g1GC in most clusters with *26GB heap* and extra threads
> given to parallel and old gen collection. Those clusters 99% is also under
> 5 ms and doing good. We used Zing to remove all timeouts . If application
> is not having that requirement G1GC is good.
>
> with g1gGC i have seen average 200-300 ms min pauses every 4 minutes and
> 600 ms pauses every 6 hours and 99% latency is under 5-10 ms for most of
> the clusters having 10- 100 KB of read data.
>
> Regards
>
> Hari
> --
> *From:* Kant Kodali <k...@peernova.com>
> *Sent:* Saturday, November 26, 2016 8:39:01 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Java GC pauses, reality check
>
> @Harikrishnan Pillai: How many nodes you guys are running? and what is an
> approximate read size and an approximate write size?
>
> On Fri, Nov 25, 2016 at 7:32 PM, Harikrishnan Pillai <
> hpil...@walmartlabs.com> wrote:
>
>> We are running azul zing in prod with 1 million reads/s and 100 K
>> writes/s with azul .we never had a major gc above 10 ms .
>>
>> Sent from my iPhone
>>
>> > On Nov 25, 2016, at 3:49 PM, Martin Schröder <mar...@oneiros.de> wrote:
>> >
>> > 2016-11-25 23:38 GMT+01:00 Kant Kodali <k...@peernova.com>:
>> >> I would also restate the following sentence "java GC pauses are pretty
>> much
>> >> a fact of life" to "Any GC based system pauses are pretty much a fact
>> of
>> >> life".
>> >>
>> >> I would be more than happy to see if someone can counter prove.
>> >
>> > Azul disagrees.
>> > https://www.azul.com/products/zing/pgc/
>> >
>> > Best
>> >   Martin
>>
>
>


-- 
Cheers
Bill

Re: Java GC pauses, reality check

[Harikrishnan Pillai]

Hi @Kant Kodali,

We have multiple clusters running zing .

One cluster has 11/11 and another one also has 11/11.(190 GB mem,6TB hard disk 
and 16 Physical core machines)

The average read size is around 200KB and it can go upto 6 MB.

We are using g1GC in most clusters with 26GB heap and extra threads given to 
parallel and old gen collection. Those clusters 99% is also under 5 ms and 
doing good. We used Zing to remove all timeouts . If application is not having 
that requirement G1GC is good.

with g1gGC i have seen average 200-300 ms min pauses every 4 minutes and 600 ms 
pauses every 6 hours and 99% latency is under 5-10 ms for most of the clusters 
having 10- 100 KB of read data.

Regards

Hari


From: Kant Kodali <k...@peernova.com>
Sent: Saturday, November 26, 2016 8:39:01 PM
To: user@cassandra.apache.org
Subject: Re: Java GC pauses, reality check

@Harikrishnan Pillai: How many nodes you guys are running? and what is an 
approximate read size and an approximate write size?

On Fri, Nov 25, 2016 at 7:32 PM, Harikrishnan Pillai 
<hpil...@walmartlabs.com<mailto:hpil...@walmartlabs.com>> wrote:
We are running azul zing in prod with 1 million reads/s and 100 K writes/s with 
azul .we never had a major gc above 10 ms .

Sent from my iPhone

> On Nov 25, 2016, at 3:49 PM, Martin Schr?der 
> <mar...@oneiros.de<mailto:mar...@oneiros.de>> wrote:
>
> 2016-11-25 23:38 GMT+01:00 Kant Kodali 
> <k...@peernova.com<mailto:k...@peernova.com>>:
>> I would also restate the following sentence "java GC pauses are pretty much
>> a fact of life" to "Any GC based system pauses are pretty much a fact of
>> life".
>>
>> I would be more than happy to see if someone can counter prove.
>
> Azul disagrees.
> https://www.azul.com/products/zing/pgc/
>
> Best
>   Martin

Re: Java GC pauses, reality check

[Benjamin Roth]

Maybe I was not totally clear. Reference counting is of course done at
runtime but the compiler automates where + when to do the counting.
Before, the developer had to retain + release objects manually. Since ARC,
this is done by the compiler at file level.
Nothing is "free" in this world. There are also drawbacks on it. But there
is indeed no GC like in Java (at least not in Clang). Cycles have to be
avoided by the developer.
See here https://en.wikipedia.org/wiki/Automatic_Reference_Counting

2016-11-27 15:28 GMT+01:00 Jonathan Haddad :

> Reference counting happens at run time, not compile time. It's not free
> either. Every time a reference is added, there's overhead in tracking it.
> It also doesn't catch cycles. You still need garbage collection to avoid
> memory leaks.
>
> On Sun, Nov 27, 2016 at 12:31 AM Benjamin Roth 
> wrote:
>
>> Arc means Automatic Reference counting which is done at compilen time. Eg
>> Objektive c and Swift use this technique. There are absolutely No gc's. Its
>> a completely different memory Management technique.
>>
>> Why i dont like Java on Server side? Because gc is a pain in the ass. I
>> am doing this Business since over 15 years and running/maintaining Apps
>> that are build in c or c++ has never been such a pain.
>>
>> On the other Hand Java is easier to handle for Developers. And coding
>> plain c is also a pain.
>>
>> Thats why i Said its a philosophic discussion.
>> Anyway Cassandra rund on Java so We have to Deal with it.
>>
>> Am 27.11.2016 05:28 schrieb "Kant Kodali" :
>>
>> Benjamin Roth: How do you know Arc eliminates GC pauses completely? By
>> completely I mean no GC pauses whatsoever.
>>
>> When you say Java is NOT the First choice for Server Applications you
>> are generalizing it too much I would say since many of them fall under that
>> category. Either way the statement you made is purely subjective.
>>
>> On Fri, Nov 25, 2016 at 2:41 PM, Benjamin Roth 
>> wrote:
>>
>> Lol. The counter proof is to use another memory Model like Arc. Thats why
>> i personally think Java is NOT the First choice for Server Applications.
>> But thats a philosophic discussion.
>>
>> Am 25.11.2016 23:38 schrieb "Kant Kodali" :
>>
>> +1 Chris Lohfink response
>>
>> I would also restate the following sentence "java GC pauses are pretty
>> much a fact of life" to "Any GC based system pauses are pretty much a
>> fact of life".
>>
>> I would be more than happy to see if someone can counter prove.
>>
>>
>>
>> On Fri, Nov 25, 2016 at 1:41 PM, Chris Lohfink 
>> wrote:
>>
>> No tuning will eliminate gcs.
>>
>> 20-30 seconds is horrific and out of the ordinary. Most likely
>> implementing antipatterns and/or poorly configured. Sub 1s is realistic but
>> with some workloads still may require some tuning to maintain. Some
>> workloads are very unfriendly to GCs though (ie heavy tombstones, very wide
>> partitions).
>>
>> Chris
>>
>> On Fri, Nov 25, 2016 at 3:25 PM, S Ahmed  wrote:
>>
>> Hello!
>>
>> From what I understand java GC pauses are pretty much a fact of life, but
>> you can tune the jvm to reduce the likelihood of the frequency and length
>> of GC pauses.
>>
>> When using Cassandra, how frequent or long have these pauses known to
>> be?  Even with tuning, is it safe to assume they cannot be eliminated?
>>
>> Would a 20-30 second pause be something out of the ordinary?
>>
>> Thanks.
>>
>>
>>
>>
>>


-- 
Benjamin Roth
Prokurist

Jaumo GmbH · www.jaumo.com
Wehrstraße 46 · 73035 Göppingen · Germany
Phone +49 7161 304880-6 · Fax +49 7161 304880-1
AG Ulm · HRB 731058 · Managing Director: Jens Kammerer

Re: Java GC pauses, reality check

[Jonathan Haddad]

Reference counting happens at run time, not compile time. It's not free
either. Every time a reference is added, there's overhead in tracking it.
It also doesn't catch cycles. You still need garbage collection to avoid
memory leaks.

On Sun, Nov 27, 2016 at 12:31 AM Benjamin Roth 
wrote:

> Arc means Automatic Reference counting which is done at compilen time. Eg
> Objektive c and Swift use this technique. There are absolutely No gc's. Its
> a completely different memory Management technique.
>
> Why i dont like Java on Server side? Because gc is a pain in the ass. I am
> doing this Business since over 15 years and running/maintaining Apps that
> are build in c or c++ has never been such a pain.
>
> On the other Hand Java is easier to handle for Developers. And coding
> plain c is also a pain.
>
> Thats why i Said its a philosophic discussion.
> Anyway Cassandra rund on Java so We have to Deal with it.
>
> Am 27.11.2016 05:28 schrieb "Kant Kodali" :
>
> Benjamin Roth: How do you know Arc eliminates GC pauses completely? By
> completely I mean no GC pauses whatsoever.
>
> When you say Java is NOT the First choice for Server Applications you are
> generalizing it too much I would say since many of them fall under that
> category. Either way the statement you made is purely subjective.
>
> On Fri, Nov 25, 2016 at 2:41 PM, Benjamin Roth 
> wrote:
>
> Lol. The counter proof is to use another memory Model like Arc. Thats why
> i personally think Java is NOT the First choice for Server Applications.
> But thats a philosophic discussion.
>
> Am 25.11.2016 23:38 schrieb "Kant Kodali" :
>
> +1 Chris Lohfink response
>
> I would also restate the following sentence "java GC pauses are pretty
> much a fact of life" to "Any GC based system pauses are pretty much a
> fact of life".
>
> I would be more than happy to see if someone can counter prove.
>
>
>
> On Fri, Nov 25, 2016 at 1:41 PM, Chris Lohfink 
> wrote:
>
> No tuning will eliminate gcs.
>
> 20-30 seconds is horrific and out of the ordinary. Most likely
> implementing antipatterns and/or poorly configured. Sub 1s is realistic but
> with some workloads still may require some tuning to maintain. Some
> workloads are very unfriendly to GCs though (ie heavy tombstones, very wide
> partitions).
>
> Chris
>
> On Fri, Nov 25, 2016 at 3:25 PM, S Ahmed  wrote:
>
> Hello!
>
> From what I understand java GC pauses are pretty much a fact of life, but
> you can tune the jvm to reduce the likelihood of the frequency and length
> of GC pauses.
>
> When using Cassandra, how frequent or long have these pauses known to be?
> Even with tuning, is it safe to assume they cannot be eliminated?
>
> Would a 20-30 second pause be something out of the ordinary?
>
> Thanks.
>
>
>
>
>

Re: Java GC pauses, reality check

[Benjamin Roth]

I didn't even know there are plans to move to TPC in Cs. Thanks for that
update. After all I will follow the development of both Scylla and Cs and
am excited about the future of both!

Am 27.11.2016 10:02 schrieb "Kant Kodali" :

> Yes I am well aware of Scyalldb. It might be well written in C++ but the
> performance gain they are claiming has very little to do with moving from
> Java to C++. They had major design changes such as moving away from SEDA to
> TPC and so on. Moreover I would say it still needs to mature. Lot of users
> had complained that they cannot get the benchmarks similar to the ones that
> are posted online and I keep seeing comments stating that you need to use a
> specific hardware and specific tuning mechanisms and so on (I don't mean to
> say what scylladb is claiming is wrong I certainly haven't verified it but
> I do know for the fact lot of people are having trouble to reach those
> benchmarks).
>
> SEDA to TPC is a very big change. Let's see how long it would take for
> Apache C*
>
> https://issues.apache.org/jira/browse/CASSANDRA-10989
>
>
>
>
> On Sat, Nov 26, 2016 at 11:45 PM, Benjamin Roth 
> wrote:
>
>> You are of course right. There is no solution and no language that is a
>> perfect match for every situation and every solution and language has it's
>> own pros, cons, pitfalls and drawbacks.
>> Actually that article you posted points at some aspect of ARC, I wasn't
>> aware of, yet.
>> Nevertheless, GC is an issue for Cassandra, otherwise this thread would
>> not exist, right? But we have to deal with it and get the best out of it.
>>
>> Another option, besides optimizing your GC: You could check if
>> http://www.scylladb.com/ is an option for you.
>> They rewrote CS from the scratch. The goal is to be completely compatible
>> with CS but to be much, much faster. Check their benchmarks and their
>> architecture.
>> I really do not want do depreciate the work of all the Cassandra
>> Developers - they did a great job - but what I have seen there looked very
>> interesting and promising! By the way it's written in C++.
>>
>>
>> 2016-11-27 7:06 GMT+01:00 Kant Kodali :
>>
>>> Automatic Reference counting sounds like college level idea that we all
>>> have been hearing for since GC is born! There seem to be bunch of cons of
>>> ARC as explained here
>>>
>>> https://www.quora.com/Why-doesnt-Apple-Swift-adopt-the-memor
>>> y-management-method-of-garbage-collection-like-in-Java
>>>
>>> Maintaining C and C++ APPS are never a pain? How about versioning and
>>> static time libraries? There is work there too. so its all pros and cons
>>>
>>> "gc is a pain in the ass". How about seg faults? they aren't any lesser
>>> pain :)
>>>
>>> Not only Cassandra that runs on JVM. Majority of Apache projects do run
>>> on JVM for a reason.
>>>
>>> Bottom line. My point here is there are pros and cons of every language.
>>> It doesn't make much sense to target one language.
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Sat, Nov 26, 2016 at 9:31 PM, Benjamin Roth 
>>> wrote:
>>>
 Arc means Automatic Reference counting which is done at compilen time.
 Eg Objektive c and Swift use this technique. There are absolutely No gc's.
 Its a completely different memory Management technique.

 Why i dont like Java on Server side? Because gc is a pain in the ass. I
 am doing this Business since over 15 years and running/maintaining Apps
 that are build in c or c++ has never been such a pain.

 On the other Hand Java is easier to handle for Developers. And coding
 plain c is also a pain.

 Thats why i Said its a philosophic discussion.
 Anyway Cassandra rund on Java so We have to Deal with it.

 Am 27.11.2016 05:28 schrieb "Kant Kodali" :

> Benjamin Roth: How do you know Arc eliminates GC pauses completely? By
> completely I mean no GC pauses whatsoever.
>
> When you say Java is NOT the First choice for Server Applications you
> are generalizing it too much I would say since many of them fall under 
> that
> category. Either way the statement you made is purely subjective.
>
> On Fri, Nov 25, 2016 at 2:41 PM, Benjamin Roth <
> benjamin.r...@jaumo.com> wrote:
>
>> Lol. The counter proof is to use another memory Model like Arc. Thats
>> why i personally think Java is NOT the First choice for Server
>> Applications. But thats a philosophic discussion.
>>
>> Am 25.11.2016 23:38 schrieb "Kant Kodali" :
>>
>>> +1 Chris Lohfink response
>>>
>>> I would also restate the following sentence "java GC pauses are
>>> pretty much a fact of life" to "Any GC based system pauses are
>>> pretty much a fact of life".
>>>
>>> I would be more than happy to see if someone can counter prove.
>>>
>>>
>>>
>>> On Fri, Nov 25, 2016 at 1:41 PM, Chris Lohfink 

Re: Java GC pauses, reality check

[Kant Kodali]

Yes I am well aware of Scyalldb. It might be well written in C++ but the
performance gain they are claiming has very little to do with moving from
Java to C++. They had major design changes such as moving away from SEDA to
TPC and so on. Moreover I would say it still needs to mature. Lot of users
had complained that they cannot get the benchmarks similar to the ones that
are posted online and I keep seeing comments stating that you need to use a
specific hardware and specific tuning mechanisms and so on (I don't mean to
say what scylladb is claiming is wrong I certainly haven't verified it but
I do know for the fact lot of people are having trouble to reach those
benchmarks).

SEDA to TPC is a very big change. Let's see how long it would take for
Apache C*

https://issues.apache.org/jira/browse/CASSANDRA-10989




On Sat, Nov 26, 2016 at 11:45 PM, Benjamin Roth 
wrote:

> You are of course right. There is no solution and no language that is a
> perfect match for every situation and every solution and language has it's
> own pros, cons, pitfalls and drawbacks.
> Actually that article you posted points at some aspect of ARC, I wasn't
> aware of, yet.
> Nevertheless, GC is an issue for Cassandra, otherwise this thread would
> not exist, right? But we have to deal with it and get the best out of it.
>
> Another option, besides optimizing your GC: You could check if
> http://www.scylladb.com/ is an option for you.
> They rewrote CS from the scratch. The goal is to be completely compatible
> with CS but to be much, much faster. Check their benchmarks and their
> architecture.
> I really do not want do depreciate the work of all the Cassandra
> Developers - they did a great job - but what I have seen there looked very
> interesting and promising! By the way it's written in C++.
>
>
> 2016-11-27 7:06 GMT+01:00 Kant Kodali :
>
>> Automatic Reference counting sounds like college level idea that we all
>> have been hearing for since GC is born! There seem to be bunch of cons of
>> ARC as explained here
>>
>> https://www.quora.com/Why-doesnt-Apple-Swift-adopt-the-memor
>> y-management-method-of-garbage-collection-like-in-Java
>>
>> Maintaining C and C++ APPS are never a pain? How about versioning and
>> static time libraries? There is work there too. so its all pros and cons
>>
>> "gc is a pain in the ass". How about seg faults? they aren't any lesser
>> pain :)
>>
>> Not only Cassandra that runs on JVM. Majority of Apache projects do run
>> on JVM for a reason.
>>
>> Bottom line. My point here is there are pros and cons of every language.
>> It doesn't make much sense to target one language.
>>
>>
>>
>>
>>
>>
>> On Sat, Nov 26, 2016 at 9:31 PM, Benjamin Roth 
>> wrote:
>>
>>> Arc means Automatic Reference counting which is done at compilen time.
>>> Eg Objektive c and Swift use this technique. There are absolutely No gc's.
>>> Its a completely different memory Management technique.
>>>
>>> Why i dont like Java on Server side? Because gc is a pain in the ass. I
>>> am doing this Business since over 15 years and running/maintaining Apps
>>> that are build in c or c++ has never been such a pain.
>>>
>>> On the other Hand Java is easier to handle for Developers. And coding
>>> plain c is also a pain.
>>>
>>> Thats why i Said its a philosophic discussion.
>>> Anyway Cassandra rund on Java so We have to Deal with it.
>>>
>>> Am 27.11.2016 05:28 schrieb "Kant Kodali" :
>>>
 Benjamin Roth: How do you know Arc eliminates GC pauses completely? By
 completely I mean no GC pauses whatsoever.

 When you say Java is NOT the First choice for Server Applications you
 are generalizing it too much I would say since many of them fall under that
 category. Either way the statement you made is purely subjective.

 On Fri, Nov 25, 2016 at 2:41 PM, Benjamin Roth  wrote:

> Lol. The counter proof is to use another memory Model like Arc. Thats
> why i personally think Java is NOT the First choice for Server
> Applications. But thats a philosophic discussion.
>
> Am 25.11.2016 23:38 schrieb "Kant Kodali" :
>
>> +1 Chris Lohfink response
>>
>> I would also restate the following sentence "java GC pauses are
>> pretty much a fact of life" to "Any GC based system pauses are
>> pretty much a fact of life".
>>
>> I would be more than happy to see if someone can counter prove.
>>
>>
>>
>> On Fri, Nov 25, 2016 at 1:41 PM, Chris Lohfink 
>> wrote:
>>
>>> No tuning will eliminate gcs.
>>>
>>> 20-30 seconds is horrific and out of the ordinary. Most likely
>>> implementing antipatterns and/or poorly configured. Sub 1s is realistic 
>>> but
>>> with some workloads still may require some tuning to maintain. Some
>>> workloads are very unfriendly to GCs though 

Re: Java GC pauses, reality check

[Benjamin Roth]

You are of course right. There is no solution and no language that is a
perfect match for every situation and every solution and language has it's
own pros, cons, pitfalls and drawbacks.
Actually that article you posted points at some aspect of ARC, I wasn't
aware of, yet.
Nevertheless, GC is an issue for Cassandra, otherwise this thread would not
exist, right? But we have to deal with it and get the best out of it.

Another option, besides optimizing your GC: You could check if
http://www.scylladb.com/ is an option for you.
They rewrote CS from the scratch. The goal is to be completely compatible
with CS but to be much, much faster. Check their benchmarks and their
architecture.
I really do not want do depreciate the work of all the Cassandra Developers
- they did a great job - but what I have seen there looked very interesting
and promising! By the way it's written in C++.


2016-11-27 7:06 GMT+01:00 Kant Kodali :

> Automatic Reference counting sounds like college level idea that we all
> have been hearing for since GC is born! There seem to be bunch of cons of
> ARC as explained here
>
> https://www.quora.com/Why-doesnt-Apple-Swift-adopt-the-
> memory-management-method-of-garbage-collection-like-in-Java
>
> Maintaining C and C++ APPS are never a pain? How about versioning and
> static time libraries? There is work there too. so its all pros and cons
>
> "gc is a pain in the ass". How about seg faults? they aren't any lesser
> pain :)
>
> Not only Cassandra that runs on JVM. Majority of Apache projects do run on
> JVM for a reason.
>
> Bottom line. My point here is there are pros and cons of every language.
> It doesn't make much sense to target one language.
>
>
>
>
>
>
> On Sat, Nov 26, 2016 at 9:31 PM, Benjamin Roth 
> wrote:
>
>> Arc means Automatic Reference counting which is done at compilen time. Eg
>> Objektive c and Swift use this technique. There are absolutely No gc's. Its
>> a completely different memory Management technique.
>>
>> Why i dont like Java on Server side? Because gc is a pain in the ass. I
>> am doing this Business since over 15 years and running/maintaining Apps
>> that are build in c or c++ has never been such a pain.
>>
>> On the other Hand Java is easier to handle for Developers. And coding
>> plain c is also a pain.
>>
>> Thats why i Said its a philosophic discussion.
>> Anyway Cassandra rund on Java so We have to Deal with it.
>>
>> Am 27.11.2016 05:28 schrieb "Kant Kodali" :
>>
>>> Benjamin Roth: How do you know Arc eliminates GC pauses completely? By
>>> completely I mean no GC pauses whatsoever.
>>>
>>> When you say Java is NOT the First choice for Server Applications you
>>> are generalizing it too much I would say since many of them fall under that
>>> category. Either way the statement you made is purely subjective.
>>>
>>> On Fri, Nov 25, 2016 at 2:41 PM, Benjamin Roth 
>>> wrote:
>>>
 Lol. The counter proof is to use another memory Model like Arc. Thats
 why i personally think Java is NOT the First choice for Server
 Applications. But thats a philosophic discussion.

 Am 25.11.2016 23:38 schrieb "Kant Kodali" :

> +1 Chris Lohfink response
>
> I would also restate the following sentence "java GC pauses are
> pretty much a fact of life" to "Any GC based system pauses are pretty
> much a fact of life".
>
> I would be more than happy to see if someone can counter prove.
>
>
>
> On Fri, Nov 25, 2016 at 1:41 PM, Chris Lohfink 
> wrote:
>
>> No tuning will eliminate gcs.
>>
>> 20-30 seconds is horrific and out of the ordinary. Most likely
>> implementing antipatterns and/or poorly configured. Sub 1s is realistic 
>> but
>> with some workloads still may require some tuning to maintain. Some
>> workloads are very unfriendly to GCs though (ie heavy tombstones, very 
>> wide
>> partitions).
>>
>> Chris
>>
>> On Fri, Nov 25, 2016 at 3:25 PM, S Ahmed 
>> wrote:
>>
>>> Hello!
>>>
>>> From what I understand java GC pauses are pretty much a fact of
>>> life, but you can tune the jvm to reduce the likelihood of the frequency
>>> and length of GC pauses.
>>>
>>> When using Cassandra, how frequent or long have these pauses known
>>> to be?  Even with tuning, is it safe to assume they cannot be 
>>> eliminated?
>>>
>>> Would a 20-30 second pause be something out of the ordinary?
>>>
>>> Thanks.
>>>
>>
>>
>
>>>
>


-- 
Benjamin Roth
Prokurist

Jaumo GmbH · www.jaumo.com
Wehrstraße 46 · 73035 Göppingen · Germany
Phone +49 7161 304880-6 · Fax +49 7161 304880-1
AG Ulm · HRB 731058 · Managing Director: Jens Kammerer

Re: Java GC pauses, reality check

[Kant Kodali]

Automatic Reference counting sounds like college level idea that we all
have been hearing for since GC is born! There seem to be bunch of cons of
ARC as explained here

https://www.quora.com/Why-doesnt-Apple-Swift-adopt-the-memory-management-method-of-garbage-collection-like-in-Java

Maintaining C and C++ APPS are never a pain? How about versioning and
static time libraries? There is work there too. so its all pros and cons

"gc is a pain in the ass". How about seg faults? they aren't any lesser
pain :)

Not only Cassandra that runs on JVM. Majority of Apache projects do run on
JVM for a reason.

Bottom line. My point here is there are pros and cons of every language. It
doesn't make much sense to target one language.






On Sat, Nov 26, 2016 at 9:31 PM, Benjamin Roth 
wrote:

> Arc means Automatic Reference counting which is done at compilen time. Eg
> Objektive c and Swift use this technique. There are absolutely No gc's. Its
> a completely different memory Management technique.
>
> Why i dont like Java on Server side? Because gc is a pain in the ass. I am
> doing this Business since over 15 years and running/maintaining Apps that
> are build in c or c++ has never been such a pain.
>
> On the other Hand Java is easier to handle for Developers. And coding
> plain c is also a pain.
>
> Thats why i Said its a philosophic discussion.
> Anyway Cassandra rund on Java so We have to Deal with it.
>
> Am 27.11.2016 05:28 schrieb "Kant Kodali" :
>
>> Benjamin Roth: How do you know Arc eliminates GC pauses completely? By
>> completely I mean no GC pauses whatsoever.
>>
>> When you say Java is NOT the First choice for Server Applications you
>> are generalizing it too much I would say since many of them fall under that
>> category. Either way the statement you made is purely subjective.
>>
>> On Fri, Nov 25, 2016 at 2:41 PM, Benjamin Roth 
>> wrote:
>>
>>> Lol. The counter proof is to use another memory Model like Arc. Thats
>>> why i personally think Java is NOT the First choice for Server
>>> Applications. But thats a philosophic discussion.
>>>
>>> Am 25.11.2016 23:38 schrieb "Kant Kodali" :
>>>
 +1 Chris Lohfink response

 I would also restate the following sentence "java GC pauses are pretty
 much a fact of life" to "Any GC based system pauses are pretty much a
 fact of life".

 I would be more than happy to see if someone can counter prove.



 On Fri, Nov 25, 2016 at 1:41 PM, Chris Lohfink 
 wrote:

> No tuning will eliminate gcs.
>
> 20-30 seconds is horrific and out of the ordinary. Most likely
> implementing antipatterns and/or poorly configured. Sub 1s is realistic 
> but
> with some workloads still may require some tuning to maintain. Some
> workloads are very unfriendly to GCs though (ie heavy tombstones, very 
> wide
> partitions).
>
> Chris
>
> On Fri, Nov 25, 2016 at 3:25 PM, S Ahmed  wrote:
>
>> Hello!
>>
>> From what I understand java GC pauses are pretty much a fact of life,
>> but you can tune the jvm to reduce the likelihood of the frequency and
>> length of GC pauses.
>>
>> When using Cassandra, how frequent or long have these pauses known to
>> be?  Even with tuning, is it safe to assume they cannot be eliminated?
>>
>> Would a 20-30 second pause be something out of the ordinary?
>>
>> Thanks.
>>
>
>

>>

Re: Java GC pauses, reality check

[Benjamin Roth]

Arc means Automatic Reference counting which is done at compilen time. Eg
Objektive c and Swift use this technique. There are absolutely No gc's. Its
a completely different memory Management technique.

Why i dont like Java on Server side? Because gc is a pain in the ass. I am
doing this Business since over 15 years and running/maintaining Apps that
are build in c or c++ has never been such a pain.

On the other Hand Java is easier to handle for Developers. And coding plain
c is also a pain.

Thats why i Said its a philosophic discussion.
Anyway Cassandra rund on Java so We have to Deal with it.

Am 27.11.2016 05:28 schrieb "Kant Kodali" :

> Benjamin Roth: How do you know Arc eliminates GC pauses completely? By
> completely I mean no GC pauses whatsoever.
>
> When you say Java is NOT the First choice for Server Applications you are
> generalizing it too much I would say since many of them fall under that
> category. Either way the statement you made is purely subjective.
>
> On Fri, Nov 25, 2016 at 2:41 PM, Benjamin Roth 
> wrote:
>
>> Lol. The counter proof is to use another memory Model like Arc. Thats why
>> i personally think Java is NOT the First choice for Server Applications.
>> But thats a philosophic discussion.
>>
>> Am 25.11.2016 23:38 schrieb "Kant Kodali" :
>>
>>> +1 Chris Lohfink response
>>>
>>> I would also restate the following sentence "java GC pauses are pretty
>>> much a fact of life" to "Any GC based system pauses are pretty much a
>>> fact of life".
>>>
>>> I would be more than happy to see if someone can counter prove.
>>>
>>>
>>>
>>> On Fri, Nov 25, 2016 at 1:41 PM, Chris Lohfink 
>>> wrote:
>>>
 No tuning will eliminate gcs.

 20-30 seconds is horrific and out of the ordinary. Most likely
 implementing antipatterns and/or poorly configured. Sub 1s is realistic but
 with some workloads still may require some tuning to maintain. Some
 workloads are very unfriendly to GCs though (ie heavy tombstones, very wide
 partitions).

 Chris

 On Fri, Nov 25, 2016 at 3:25 PM, S Ahmed  wrote:

> Hello!
>
> From what I understand java GC pauses are pretty much a fact of life,
> but you can tune the jvm to reduce the likelihood of the frequency and
> length of GC pauses.
>
> When using Cassandra, how frequent or long have these pauses known to
> be?  Even with tuning, is it safe to assume they cannot be eliminated?
>
> Would a 20-30 second pause be something out of the ordinary?
>
> Thanks.
>


>>>
>

Re: Java GC pauses, reality check

[Kant Kodali]

@Harikrishnan Pillai: How many nodes you guys are running? and what is an
approximate read size and an approximate write size?

On Fri, Nov 25, 2016 at 7:32 PM, Harikrishnan Pillai <
hpil...@walmartlabs.com> wrote:

> We are running azul zing in prod with 1 million reads/s and 100 K writes/s
> with azul .we never had a major gc above 10 ms .
>
> Sent from my iPhone
>
> > On Nov 25, 2016, at 3:49 PM, Martin Schröder  wrote:
> >
> > 2016-11-25 23:38 GMT+01:00 Kant Kodali :
> >> I would also restate the following sentence "java GC pauses are pretty
> much
> >> a fact of life" to "Any GC based system pauses are pretty much a fact of
> >> life".
> >>
> >> I would be more than happy to see if someone can counter prove.
> >
> > Azul disagrees.
> > https://www.azul.com/products/zing/pgc/
> >
> > Best
> >   Martin
>

Re: Java GC pauses, reality check

[Kant Kodali]

Good to know about Zing! I will have to take a look.

On Sat, Nov 26, 2016 at 8:27 PM, Kant Kodali  wrote:

> Benjamin Roth: How do you know Arc eliminates GC pauses completely? By
> completely I mean no GC pauses whatsoever.
>
> When you say Java is NOT the First choice for Server Applications you are
> generalizing it too much I would say since many of them fall under that
> category. Either way the statement you made is purely subjective.
>
> On Fri, Nov 25, 2016 at 2:41 PM, Benjamin Roth 
> wrote:
>
>> Lol. The counter proof is to use another memory Model like Arc. Thats why
>> i personally think Java is NOT the First choice for Server Applications.
>> But thats a philosophic discussion.
>>
>> Am 25.11.2016 23:38 schrieb "Kant Kodali" :
>>
>>> +1 Chris Lohfink response
>>>
>>> I would also restate the following sentence "java GC pauses are pretty
>>> much a fact of life" to "Any GC based system pauses are pretty much a
>>> fact of life".
>>>
>>> I would be more than happy to see if someone can counter prove.
>>>
>>>
>>>
>>> On Fri, Nov 25, 2016 at 1:41 PM, Chris Lohfink 
>>> wrote:
>>>
 No tuning will eliminate gcs.

 20-30 seconds is horrific and out of the ordinary. Most likely
 implementing antipatterns and/or poorly configured. Sub 1s is realistic but
 with some workloads still may require some tuning to maintain. Some
 workloads are very unfriendly to GCs though (ie heavy tombstones, very wide
 partitions).

 Chris

 On Fri, Nov 25, 2016 at 3:25 PM, S Ahmed  wrote:

> Hello!
>
> From what I understand java GC pauses are pretty much a fact of life,
> but you can tune the jvm to reduce the likelihood of the frequency and
> length of GC pauses.
>
> When using Cassandra, how frequent or long have these pauses known to
> be?  Even with tuning, is it safe to assume they cannot be eliminated?
>
> Would a 20-30 second pause be something out of the ordinary?
>
> Thanks.
>


>>>
>

Re: Java GC pauses, reality check

[Kant Kodali]

Benjamin Roth: How do you know Arc eliminates GC pauses completely? By
completely I mean no GC pauses whatsoever.

When you say Java is NOT the First choice for Server Applications you are
generalizing it too much I would say since many of them fall under that
category. Either way the statement you made is purely subjective.

On Fri, Nov 25, 2016 at 2:41 PM, Benjamin Roth 
wrote:

> Lol. The counter proof is to use another memory Model like Arc. Thats why
> i personally think Java is NOT the First choice for Server Applications.
> But thats a philosophic discussion.
>
> Am 25.11.2016 23:38 schrieb "Kant Kodali" :
>
>> +1 Chris Lohfink response
>>
>> I would also restate the following sentence "java GC pauses are pretty
>> much a fact of life" to "Any GC based system pauses are pretty much a
>> fact of life".
>>
>> I would be more than happy to see if someone can counter prove.
>>
>>
>>
>> On Fri, Nov 25, 2016 at 1:41 PM, Chris Lohfink 
>> wrote:
>>
>>> No tuning will eliminate gcs.
>>>
>>> 20-30 seconds is horrific and out of the ordinary. Most likely
>>> implementing antipatterns and/or poorly configured. Sub 1s is realistic but
>>> with some workloads still may require some tuning to maintain. Some
>>> workloads are very unfriendly to GCs though (ie heavy tombstones, very wide
>>> partitions).
>>>
>>> Chris
>>>
>>> On Fri, Nov 25, 2016 at 3:25 PM, S Ahmed  wrote:
>>>
 Hello!

 From what I understand java GC pauses are pretty much a fact of life,
 but you can tune the jvm to reduce the likelihood of the frequency and
 length of GC pauses.

 When using Cassandra, how frequent or long have these pauses known to
 be?  Even with tuning, is it safe to assume they cannot be eliminated?

 Would a 20-30 second pause be something out of the ordinary?

 Thanks.

>>>
>>>
>>

Re: Java GC pauses, reality check

[Oleksandr Shulgin]

On Nov 26, 2016 20:52, "Graham Sanderson"  wrote:

It was removed in the 3.0.x line, but not in the 3.x line (post 9472) as
far as I can tell. It looks to be available in 3.11 and in 3.X branches


Thanks, you are correct. I'm confused.

On Nov 26, 2016, at 1:17 PM, Oleksandr Shulgin 
wrote:

On Nov 26, 2016 20:04, "Graham Sanderson"  wrote:

Not AFAIK; https://issues.apache.org/jira/browse/CASSANDRA-9472 is marked
as resolved in 3.4, though we are not running it so I can’t say much about
it.


But I was referring to https://issues.apache.org/jira/browse/CASSANDRA-11039
which removed it again in 3.10 and 3.0.10.

--
Alex

It looks like Zing is no longer tied price wise per core which was a show
stopper for us, but it is now priced per server which may affect others
differently.

Note in fact ironically, running 2.1.x with off heap memtables, we had some
of our JVMs running for over a year which made us hit
https://issues.apache.org/jira/browse/CASSANDRA-10969 when we restarted
some nodes for other reasons.

On Nov 26, 2016, at 12:07 AM, Oleksandr Shulgin <
oleksandr.shul...@zalando.de> wrote:

On Nov 25, 2016 23:47, "Graham Sanderson"  wrote:

If you are seeing 25-30 second GC pauses then (unless you are so badly
configured) seeing full GC under CMS (though G1 may have similar problems).

With CMS eventual fragmentation causing promotion failure is inevitable
(unless you cycle your nodes before it happens). Either your heap has way
too big an old gen, or too small a young gen (but then you need pretty
hefty boxes to be able to run with a large young gen - of the say 4-8G
range) without young collections taking too long.

Depending on your C* version I would highly recommend off heap men-tables.
With those we were able to considerably reduce our heap sizes, despite
having large throughput on a smallish number of nodes.


Aren't offheap memtables discontinued in the most recent releases of 3.0
and 3.x for a good reason? I thought using them could lead to segfaults?

--
Alex

I recommend reading this if you use CMS http://blog.ragozin.info/2
011/10/java-cg-hotspots-cms-and-heap.html, and also not that if you see a
lot of objects of size 131074 in promotion failures then memtables are the
problem - you can try and flush them sooner, but moving them off heap works
better I think.

On Nov 25, 2016, at 4:38 PM, Kant Kodali  wrote:

+1 Chris Lohfink response

I would also restate the following sentence "java GC pauses are pretty much
a fact of life" to "Any GC based system pauses are pretty much a fact of
life".

I would be more than happy to see if someone can counter prove.



On Fri, Nov 25, 2016 at 1:41 PM, Chris Lohfink  wrote:

> No tuning will eliminate gcs.
>
> 20-30 seconds is horrific and out of the ordinary. Most likely
> implementing antipatterns and/or poorly configured. Sub 1s is realistic but
> with some workloads still may require some tuning to maintain. Some
> workloads are very unfriendly to GCs though (ie heavy tombstones, very wide
> partitions).
>
> Chris
>
> On Fri, Nov 25, 2016 at 3:25 PM, S Ahmed  wrote:
>
>> Hello!
>>
>> From what I understand java GC pauses are pretty much a fact of life, but
>> you can tune the jvm to reduce the likelihood of the frequency and length
>> of GC pauses.
>>
>> When using Cassandra, how frequent or long have these pauses known to
>> be?  Even with tuning, is it safe to assume they cannot be eliminated?
>>
>> Would a 20-30 second pause be something out of the ordinary?
>>
>> Thanks.
>>
>
>

Re: Java GC pauses, reality check

[Graham Sanderson]

It was removed in the 3.0.x line, but not in the 3.x line (post 9472) as far as 
I can tell. It looks to be available in 3.11 and in 3.X branches

> On Nov 26, 2016, at 1:17 PM, Oleksandr Shulgin  
> wrote:
> 
> On Nov 26, 2016 20:04, "Graham Sanderson"  > wrote:
> Not AFAIK; https://issues.apache.org/jira/browse/CASSANDRA-9472 
>  is marked as resolved 
> in 3.4, though we are not running it so I can’t say much about it.
> 
> But I was referring to https://issues.apache.org/jira/browse/CASSANDRA-11039 
>  which removed it 
> again in 3.10 and 3.0.10.
> 
> --
> Alex
> 
> It looks like Zing is no longer tied price wise per core which was a show 
> stopper for us, but it is now priced per server which may affect others 
> differently.
> 
> Note in fact ironically, running 2.1.x with off heap memtables, we had some 
> of our JVMs running for over a year which made us hit 
> https://issues.apache.org/jira/browse/CASSANDRA-10969 
>  when we restarted 
> some nodes for other reasons.
> 
>> On Nov 26, 2016, at 12:07 AM, Oleksandr Shulgin 
>> > wrote:
>> 
>> On Nov 25, 2016 23:47, "Graham Sanderson" > > wrote:
>> If you are seeing 25-30 second GC pauses then (unless you are so badly 
>> configured) seeing full GC under CMS (though G1 may have similar problems).
>> 
>> With CMS eventual fragmentation causing promotion failure is inevitable 
>> (unless you cycle your nodes before it happens). Either your heap has way 
>> too big an old gen, or too small a young gen (but then you need pretty hefty 
>> boxes to be able to run with a large young gen - of the say 4-8G range) 
>> without young collections taking too long.
>> 
>> Depending on your C* version I would highly recommend off heap men-tables. 
>> With those we were able to considerably reduce our heap sizes, despite 
>> having large throughput on a smallish number of nodes.
>> 
>> Aren't offheap memtables discontinued in the most recent releases of 3.0 and 
>> 3.x for a good reason? I thought using them could lead to segfaults?
>> 
>> --
>> Alex
>> 
>> I recommend reading this if you use CMS 
>> http://blog.ragozin.info/2011/10/java-cg-hotspots-cms-and-heap.html 
>> , and 
>> also not that if you see a lot of objects of size 131074 in promotion 
>> failures then memtables are the problem - you can try and flush them sooner, 
>> but moving them off heap works better I think.
>> 
>>> On Nov 25, 2016, at 4:38 PM, Kant Kodali >> > wrote:
>>> 
>>> +1 Chris Lohfink response
>>> 
>>> I would also restate the following sentence "java GC pauses are pretty much 
>>> a fact of life" to "Any GC based system pauses are pretty much a fact of 
>>> life".
>>> 
>>> I would be more than happy to see if someone can counter prove.
>>> 
>>> 
>>> 
>>> On Fri, Nov 25, 2016 at 1:41 PM, Chris Lohfink >> > wrote:
>>> No tuning will eliminate gcs.
>>> 
>>> 20-30 seconds is horrific and out of the ordinary. Most likely implementing 
>>> antipatterns and/or poorly configured. Sub 1s is realistic but with some 
>>> workloads still may require some tuning to maintain. Some workloads are 
>>> very unfriendly to GCs though (ie heavy tombstones, very wide partitions).
>>> 
>>> Chris
>>> 
>>> On Fri, Nov 25, 2016 at 3:25 PM, S Ahmed >> > wrote:
>>> Hello!
>>> 
>>> From what I understand java GC pauses are pretty much a fact of life, but 
>>> you can tune the jvm to reduce the likelihood of the frequency and length 
>>> of GC pauses.
>>> 
>>> When using Cassandra, how frequent or long have these pauses known to be?  
>>> Even with tuning, is it safe to assume they cannot be eliminated?
>>> 
>>> Would a 20-30 second pause be something out of the ordinary?
>>> 
>>> Thanks.
>>> 
>>> 
>> 
>> 
> 
> 



smime.p7s
Description: S/MIME cryptographic signature

Re: Java GC pauses, reality check

[Oleksandr Shulgin]

On Nov 26, 2016 20:04, "Graham Sanderson"  wrote:

Not AFAIK; https://issues.apache.org/jira/browse/CASSANDRA-9472 is marked
as resolved in 3.4, though we are not running it so I can’t say much about
it.


But I was referring to https://issues.apache.org/jira/browse/CASSANDRA-11039
which removed it again in 3.10 and 3.0.10.

--
Alex

It looks like Zing is no longer tied price wise per core which was a show
stopper for us, but it is now priced per server which may affect others
differently.

Note in fact ironically, running 2.1.x with off heap memtables, we had some
of our JVMs running for over a year which made us hit
https://issues.apache.org/jira/browse/CASSANDRA-10969 when we restarted
some nodes for other reasons.

On Nov 26, 2016, at 12:07 AM, Oleksandr Shulgin <
oleksandr.shul...@zalando.de> wrote:

On Nov 25, 2016 23:47, "Graham Sanderson"  wrote:

If you are seeing 25-30 second GC pauses then (unless you are so badly
configured) seeing full GC under CMS (though G1 may have similar problems).

With CMS eventual fragmentation causing promotion failure is inevitable
(unless you cycle your nodes before it happens). Either your heap has way
too big an old gen, or too small a young gen (but then you need pretty
hefty boxes to be able to run with a large young gen - of the say 4-8G
range) without young collections taking too long.

Depending on your C* version I would highly recommend off heap men-tables.
With those we were able to considerably reduce our heap sizes, despite
having large throughput on a smallish number of nodes.


Aren't offheap memtables discontinued in the most recent releases of 3.0
and 3.x for a good reason? I thought using them could lead to segfaults?

--
Alex

I recommend reading this if you use CMS http://blog.ragozin.info/2
011/10/java-cg-hotspots-cms-and-heap.html, and also not that if you see a
lot of objects of size 131074 in promotion failures then memtables are the
problem - you can try and flush them sooner, but moving them off heap works
better I think.

On Nov 25, 2016, at 4:38 PM, Kant Kodali  wrote:

+1 Chris Lohfink response

I would also restate the following sentence "java GC pauses are pretty much
a fact of life" to "Any GC based system pauses are pretty much a fact of
life".

I would be more than happy to see if someone can counter prove.



On Fri, Nov 25, 2016 at 1:41 PM, Chris Lohfink  wrote:

> No tuning will eliminate gcs.
>
> 20-30 seconds is horrific and out of the ordinary. Most likely
> implementing antipatterns and/or poorly configured. Sub 1s is realistic but
> with some workloads still may require some tuning to maintain. Some
> workloads are very unfriendly to GCs though (ie heavy tombstones, very wide
> partitions).
>
> Chris
>
> On Fri, Nov 25, 2016 at 3:25 PM, S Ahmed  wrote:
>
>> Hello!
>>
>> From what I understand java GC pauses are pretty much a fact of life, but
>> you can tune the jvm to reduce the likelihood of the frequency and length
>> of GC pauses.
>>
>> When using Cassandra, how frequent or long have these pauses known to
>> be?  Even with tuning, is it safe to assume they cannot be eliminated?
>>
>> Would a 20-30 second pause be something out of the ordinary?
>>
>> Thanks.
>>
>
>

Re: Java GC pauses, reality check

[Graham Sanderson]

Not AFAIK; https://issues.apache.org/jira/browse/CASSANDRA-9472 
 is marked as resolved in 
3.4, though we are not running it so I can’t say much about it.

It looks like Zing is no longer tied price wise per core which was a show 
stopper for us, but it is now priced per server which may affect others 
differently.

Note in fact ironically, running 2.1.x with off heap memtables, we had some of 
our JVMs running for over a year which made us hit 
https://issues.apache.org/jira/browse/CASSANDRA-10969 
 when we restarted some 
nodes for other reasons.

> On Nov 26, 2016, at 12:07 AM, Oleksandr Shulgin 
>  wrote:
> 
> On Nov 25, 2016 23:47, "Graham Sanderson"  > wrote:
> If you are seeing 25-30 second GC pauses then (unless you are so badly 
> configured) seeing full GC under CMS (though G1 may have similar problems).
> 
> With CMS eventual fragmentation causing promotion failure is inevitable 
> (unless you cycle your nodes before it happens). Either your heap has way too 
> big an old gen, or too small a young gen (but then you need pretty hefty 
> boxes to be able to run with a large young gen - of the say 4-8G range) 
> without young collections taking too long.
> 
> Depending on your C* version I would highly recommend off heap men-tables. 
> With those we were able to considerably reduce our heap sizes, despite having 
> large throughput on a smallish number of nodes.
> 
> Aren't offheap memtables discontinued in the most recent releases of 3.0 and 
> 3.x for a good reason? I thought using them could lead to segfaults?
> 
> --
> Alex
> 
> I recommend reading this if you use CMS 
> http://blog.ragozin.info/2011/10/java-cg-hotspots-cms-and-heap.html 
> , and 
> also not that if you see a lot of objects of size 131074 in promotion 
> failures then memtables are the problem - you can try and flush them sooner, 
> but moving them off heap works better I think.
> 
>> On Nov 25, 2016, at 4:38 PM, Kant Kodali > > wrote:
>> 
>> +1 Chris Lohfink response
>> 
>> I would also restate the following sentence "java GC pauses are pretty much 
>> a fact of life" to "Any GC based system pauses are pretty much a fact of 
>> life".
>> 
>> I would be more than happy to see if someone can counter prove.
>> 
>> 
>> 
>> On Fri, Nov 25, 2016 at 1:41 PM, Chris Lohfink > > wrote:
>> No tuning will eliminate gcs.
>> 
>> 20-30 seconds is horrific and out of the ordinary. Most likely implementing 
>> antipatterns and/or poorly configured. Sub 1s is realistic but with some 
>> workloads still may require some tuning to maintain. Some workloads are very 
>> unfriendly to GCs though (ie heavy tombstones, very wide partitions).
>> 
>> Chris
>> 
>> On Fri, Nov 25, 2016 at 3:25 PM, S Ahmed > > wrote:
>> Hello!
>> 
>> From what I understand java GC pauses are pretty much a fact of life, but 
>> you can tune the jvm to reduce the likelihood of the frequency and length of 
>> GC pauses.
>> 
>> When using Cassandra, how frequent or long have these pauses known to be?  
>> Even with tuning, is it safe to assume they cannot be eliminated?
>> 
>> Would a 20-30 second pause be something out of the ordinary?
>> 
>> Thanks.
>> 
>> 
> 
> 



smime.p7s
Description: S/MIME cryptographic signature

Re: Java GC pauses, reality check

[Benjamin Roth]

Thanks!

But getting back to the original issue:
I think the GC itself is not the root cause for such a long pause. I
remember having had issues with 1 minute GCs in the beginning. I also tried
around with larger and smaller heap sizes and different GCs (G1, CMS),
different settings but what helped in the end (as far as I remember -
please nail me down on that) was to increase memtable flush writers.
I could explain it like that:
If available mem is getting fuller and fuller, the GC has to ran more often
and longer to reclaim the last available bit that is currently required.
Memtables use a considerable amount of it and if they can't be flushed in
time, they grow and use more and more memory putting more and more pressure
on the GC - also known as the GC death spiral. In my case I never ran in an
OOM crash but the node became totally unresponsive.

I don't tell this must be the case here but it is one possible case.

P.S.: In my case memtable flush writers default was 2 AFAIR as I had only
one SSD but the node could easily handle many more with 8 real cores and an
SSD.

2016-11-26 7:52 GMT+01:00 Work :

> I'm not affiliated with them, I've just been impressed by them. They have
> done amazing work in performance measurement. They discovered a major flaw
> in most performance testing ... I've never seen their pricing. But,
> recently, they made their product available for testing by developers. And
> the assured me that pricing is on a sliding scale depending upon
> utilization, and not ridiculous.
>
> - James
>
> Sent from my iPhone
>
> On Nov 25, 2016, at 10:40 PM, Benjamin Roth 
> wrote:
>
> This sounds amazing but also expensive - I don't see pricing on their
> page. Are you able and allowed to tell a rough pricing range?
>
> Am 26.11.2016 04:33 schrieb "Harikrishnan Pillai"  >:
>
>> We are running azul zing in prod with 1 million reads/s and 100 K
>> writes/s with azul .we never had a major gc above 10 ms .
>>
>> Sent from my iPhone
>>
>> > On Nov 25, 2016, at 3:49 PM, Martin Schröder  wrote:
>> >
>> > 2016-11-25 23:38 GMT+01:00 Kant Kodali :
>> >> I would also restate the following sentence "java GC pauses are pretty
>> much
>> >> a fact of life" to "Any GC based system pauses are pretty much a fact
>> of
>> >> life".
>> >>
>> >> I would be more than happy to see if someone can counter prove.
>> >
>> > Azul disagrees.
>> > https://www.azul.com/products/zing/pgc/
>> >
>> > Best
>> >   Martin
>>
>


-- 
Benjamin Roth
Prokurist

Jaumo GmbH · www.jaumo.com
Wehrstraße 46 · 73035 Göppingen · Germany
Phone +49 7161 304880-6 · Fax +49 7161 304880-1
AG Ulm · HRB 731058 · Managing Director: Jens Kammerer

Re: Java GC pauses, reality check

[Work]

I'm not affiliated with them, I've just been impressed by them. They have done 
amazing work in performance measurement. They discovered a major flaw in most 
performance testing ... I've never seen their pricing. But, recently, they made 
their product available for testing by developers. And the assured me that 
pricing is on a sliding scale depending upon utilization, and not ridiculous. 

- James 

Sent from my iPhone

> On Nov 25, 2016, at 10:40 PM, Benjamin Roth  wrote:
> 
> This sounds amazing but also expensive - I don't see pricing on their page. 
> Are you able and allowed to tell a rough pricing range?
> 
> 
> Am 26.11.2016 04:33 schrieb "Harikrishnan Pillai" :
>> We are running azul zing in prod with 1 million reads/s and 100 K writes/s 
>> with azul .we never had a major gc above 10 ms .
>> 
>> Sent from my iPhone
>> 
>> > On Nov 25, 2016, at 3:49 PM, Martin Schröder  wrote:
>> >
>> > 2016-11-25 23:38 GMT+01:00 Kant Kodali :
>> >> I would also restate the following sentence "java GC pauses are pretty 
>> >> much
>> >> a fact of life" to "Any GC based system pauses are pretty much a fact of
>> >> life".
>> >>
>> >> I would be more than happy to see if someone can counter prove.
>> >
>> > Azul disagrees.
>> > https://www.azul.com/products/zing/pgc/
>> >
>> > Best
>> >   Martin

Re: Java GC pauses, reality check

[Benjamin Roth]

This sounds amazing but also expensive - I don't see pricing on their page.
Are you able and allowed to tell a rough pricing range?

Am 26.11.2016 04:33 schrieb "Harikrishnan Pillai" :

> We are running azul zing in prod with 1 million reads/s and 100 K writes/s
> with azul .we never had a major gc above 10 ms .
>
> Sent from my iPhone
>
> > On Nov 25, 2016, at 3:49 PM, Martin Schröder  wrote:
> >
> > 2016-11-25 23:38 GMT+01:00 Kant Kodali :
> >> I would also restate the following sentence "java GC pauses are pretty
> much
> >> a fact of life" to "Any GC based system pauses are pretty much a fact of
> >> life".
> >>
> >> I would be more than happy to see if someone can counter prove.
> >
> > Azul disagrees.
> > https://www.azul.com/products/zing/pgc/
> >
> > Best
> >   Martin
>

Re: Java GC pauses, reality check

[Vladimir Yudovin]

Hi Ahmed,



obviously, 20-30 sec. pause is unacceptable.

I suppose check the following:



- disable swapping completely

- check Java version, v8. is desirable (depending on Cassandra version)

- use multiprocessor machine (it allows concurrent GC)



Best regards, Vladimir Yudovin, 

Winguzone - Cloud Cassandra Hosting






 On Fri, 25 Nov 2016 16:25:15 -0500 S Ahmed sahmed1...@gmail.com 
wrote 




Hello!



>From what I understand java GC pauses are pretty much a fact of life, but you 
>can tune the jvm to reduce the likelihood of the frequency and length of GC 
>pauses.



When using Cassandra, how frequent or long have these pauses known to be?  Even 
with tuning, is it safe to assume they cannot be eliminated?



Would a 20-30 second pause be something out of the ordinary?



Thanks.

Re: Java GC pauses, reality check

[Oleksandr Shulgin]

On Nov 25, 2016 23:47, "Graham Sanderson"  wrote:

If you are seeing 25-30 second GC pauses then (unless you are so badly
configured) seeing full GC under CMS (though G1 may have similar problems).

With CMS eventual fragmentation causing promotion failure is inevitable
(unless you cycle your nodes before it happens). Either your heap has way
too big an old gen, or too small a young gen (but then you need pretty
hefty boxes to be able to run with a large young gen - of the say 4-8G
range) without young collections taking too long.

Depending on your C* version I would highly recommend off heap men-tables.
With those we were able to considerably reduce our heap sizes, despite
having large throughput on a smallish number of nodes.


Aren't offheap memtables discontinued in the most recent releases of 3.0
and 3.x for a good reason? I thought using them could lead to segfaults?

--
Alex

I recommend reading this if you use CMS http://blog.ragozin.info/
2011/10/java-cg-hotspots-cms-and-heap.html, and also not that if you see a
lot of objects of size 131074 in promotion failures then memtables are the
problem - you can try and flush them sooner, but moving them off heap works
better I think.

On Nov 25, 2016, at 4:38 PM, Kant Kodali  wrote:

+1 Chris Lohfink response

I would also restate the following sentence "java GC pauses are pretty much
a fact of life" to "Any GC based system pauses are pretty much a fact of
life".

I would be more than happy to see if someone can counter prove.



On Fri, Nov 25, 2016 at 1:41 PM, Chris Lohfink  wrote:

> No tuning will eliminate gcs.
>
> 20-30 seconds is horrific and out of the ordinary. Most likely
> implementing antipatterns and/or poorly configured. Sub 1s is realistic but
> with some workloads still may require some tuning to maintain. Some
> workloads are very unfriendly to GCs though (ie heavy tombstones, very wide
> partitions).
>
> Chris
>
> On Fri, Nov 25, 2016 at 3:25 PM, S Ahmed  wrote:
>
>> Hello!
>>
>> From what I understand java GC pauses are pretty much a fact of life, but
>> you can tune the jvm to reduce the likelihood of the frequency and length
>> of GC pauses.
>>
>> When using Cassandra, how frequent or long have these pauses known to
>> be?  Even with tuning, is it safe to assume they cannot be eliminated?
>>
>> Would a 20-30 second pause be something out of the ordinary?
>>
>> Thanks.
>>
>
>

Re: Java GC pauses, reality check

[Harikrishnan Pillai]

We are running azul zing in prod with 1 million reads/s and 100 K writes/s with 
azul .we never had a major gc above 10 ms .

Sent from my iPhone

> On Nov 25, 2016, at 3:49 PM, Martin Schröder  wrote:
> 
> 2016-11-25 23:38 GMT+01:00 Kant Kodali :
>> I would also restate the following sentence "java GC pauses are pretty much
>> a fact of life" to "Any GC based system pauses are pretty much a fact of
>> life".
>> 
>> I would be more than happy to see if someone can counter prove.
> 
> Azul disagrees.
> https://www.azul.com/products/zing/pgc/
> 
> Best
>   Martin

Re: Java GC pauses, reality check

[Harikrishnan Pillai]

Zing jvm reduces the pause under 10ms for most use cases.

Sent from my iPhone

On Nov 25, 2016, at 2:44 PM, Kant Kodali 
> wrote:

+1 Chris Lohfink response

I would also restate the following sentence "java GC pauses are pretty much a 
fact of life" to "Any GC based system pauses are pretty much a fact of life".

I would be more than happy to see if someone can counter prove.



On Fri, Nov 25, 2016 at 1:41 PM, Chris Lohfink 
> wrote:
No tuning will eliminate gcs.

20-30 seconds is horrific and out of the ordinary. Most likely implementing 
antipatterns and/or poorly configured. Sub 1s is realistic but with some 
workloads still may require some tuning to maintain. Some workloads are very 
unfriendly to GCs though (ie heavy tombstones, very wide partitions).

Chris

On Fri, Nov 25, 2016 at 3:25 PM, S Ahmed 
> wrote:
Hello!

>From what I understand java GC pauses are pretty much a fact of life, but you 
>can tune the jvm to reduce the likelihood of the frequency and length of GC 
>pauses.

When using Cassandra, how frequent or long have these pauses known to be?  Even 
with tuning, is it safe to assume they cannot be eliminated?

Would a 20-30 second pause be something out of the ordinary?

Thanks.

Re: Java GC pauses, reality check

[Martin Schröder]

2016-11-25 23:38 GMT+01:00 Kant Kodali :
> I would also restate the following sentence "java GC pauses are pretty much
> a fact of life" to "Any GC based system pauses are pretty much a fact of
> life".
>
> I would be more than happy to see if someone can counter prove.

Azul disagrees.
https://www.azul.com/products/zing/pgc/

Best
   Martin

Re: Java GC pauses, reality check

[Graham Sanderson]

If you are seeing 25-30 second GC pauses then (unless you are so badly 
configured) seeing full GC under CMS (though G1 may have similar problems).

With CMS eventual fragmentation causing promotion failure is inevitable (unless 
you cycle your nodes before it happens). Either your heap has way too big an 
old gen, or too small a young gen (but then you need pretty hefty boxes to be 
able to run with a large young gen - of the say 4-8G range) without young 
collections taking too long.

Depending on your C* version I would highly recommend off heap men-tables. With 
those we were able to considerably reduce our heap sizes, despite having large 
throughput on a smallish number of nodes.

I recommend reading this if you use CMS 
http://blog.ragozin.info/2011/10/java-cg-hotspots-cms-and-heap.html 
, and also 
not that if you see a lot of objects of size 131074 in promotion failures then 
memtables are the problem - you can try and flush them sooner, but moving them 
off heap works better I think.

> On Nov 25, 2016, at 4:38 PM, Kant Kodali  wrote:
> 
> +1 Chris Lohfink response
> 
> I would also restate the following sentence "java GC pauses are pretty much a 
> fact of life" to "Any GC based system pauses are pretty much a fact of life".
> 
> I would be more than happy to see if someone can counter prove.
> 
> 
> 
> On Fri, Nov 25, 2016 at 1:41 PM, Chris Lohfink  > wrote:
> No tuning will eliminate gcs.
> 
> 20-30 seconds is horrific and out of the ordinary. Most likely implementing 
> antipatterns and/or poorly configured. Sub 1s is realistic but with some 
> workloads still may require some tuning to maintain. Some workloads are very 
> unfriendly to GCs though (ie heavy tombstones, very wide partitions).
> 
> Chris
> 
> On Fri, Nov 25, 2016 at 3:25 PM, S Ahmed  > wrote:
> Hello!
> 
> From what I understand java GC pauses are pretty much a fact of life, but you 
> can tune the jvm to reduce the likelihood of the frequency and length of GC 
> pauses.
> 
> When using Cassandra, how frequent or long have these pauses known to be?  
> Even with tuning, is it safe to assume they cannot be eliminated?
> 
> Would a 20-30 second pause be something out of the ordinary?
> 
> Thanks.
> 
> 



smime.p7s
Description: S/MIME cryptographic signature

Re: Java GC pauses, reality check

[Benjamin Roth]

Lol. The counter proof is to use another memory Model like Arc. Thats why i
personally think Java is NOT the First choice for Server Applications. But
thats a philosophic discussion.

Am 25.11.2016 23:38 schrieb "Kant Kodali" :

> +1 Chris Lohfink response
>
> I would also restate the following sentence "java GC pauses are pretty
> much a fact of life" to "Any GC based system pauses are pretty much a
> fact of life".
>
> I would be more than happy to see if someone can counter prove.
>
>
>
> On Fri, Nov 25, 2016 at 1:41 PM, Chris Lohfink 
> wrote:
>
>> No tuning will eliminate gcs.
>>
>> 20-30 seconds is horrific and out of the ordinary. Most likely
>> implementing antipatterns and/or poorly configured. Sub 1s is realistic but
>> with some workloads still may require some tuning to maintain. Some
>> workloads are very unfriendly to GCs though (ie heavy tombstones, very wide
>> partitions).
>>
>> Chris
>>
>> On Fri, Nov 25, 2016 at 3:25 PM, S Ahmed  wrote:
>>
>>> Hello!
>>>
>>> From what I understand java GC pauses are pretty much a fact of life,
>>> but you can tune the jvm to reduce the likelihood of the frequency and
>>> length of GC pauses.
>>>
>>> When using Cassandra, how frequent or long have these pauses known to
>>> be?  Even with tuning, is it safe to assume they cannot be eliminated?
>>>
>>> Would a 20-30 second pause be something out of the ordinary?
>>>
>>> Thanks.
>>>
>>
>>
>

Re: Java GC pauses, reality check

[Kant Kodali]

+1 Chris Lohfink response

I would also restate the following sentence "java GC pauses are pretty much
a fact of life" to "Any GC based system pauses are pretty much a fact of
life".

I would be more than happy to see if someone can counter prove.



On Fri, Nov 25, 2016 at 1:41 PM, Chris Lohfink  wrote:

> No tuning will eliminate gcs.
>
> 20-30 seconds is horrific and out of the ordinary. Most likely
> implementing antipatterns and/or poorly configured. Sub 1s is realistic but
> with some workloads still may require some tuning to maintain. Some
> workloads are very unfriendly to GCs though (ie heavy tombstones, very wide
> partitions).
>
> Chris
>
> On Fri, Nov 25, 2016 at 3:25 PM, S Ahmed  wrote:
>
>> Hello!
>>
>> From what I understand java GC pauses are pretty much a fact of life, but
>> you can tune the jvm to reduce the likelihood of the frequency and length
>> of GC pauses.
>>
>> When using Cassandra, how frequent or long have these pauses known to
>> be?  Even with tuning, is it safe to assume they cannot be eliminated?
>>
>> Would a 20-30 second pause be something out of the ordinary?
>>
>> Thanks.
>>
>
>

Re: Java GC pauses, reality check

[Chris Lohfink]

No tuning will eliminate gcs.

20-30 seconds is horrific and out of the ordinary. Most likely implementing
antipatterns and/or poorly configured. Sub 1s is realistic but with some
workloads still may require some tuning to maintain. Some workloads are
very unfriendly to GCs though (ie heavy tombstones, very wide partitions).

Chris

On Fri, Nov 25, 2016 at 3:25 PM, S Ahmed  wrote:

> Hello!
>
> From what I understand java GC pauses are pretty much a fact of life, but
> you can tune the jvm to reduce the likelihood of the frequency and length
> of GC pauses.
>
> When using Cassandra, how frequent or long have these pauses known to be?
> Even with tuning, is it safe to assume they cannot be eliminated?
>
> Would a 20-30 second pause be something out of the ordinary?
>
> Thanks.
>