Re: read request is slow

[Jon Haddad]

I'm guessing you're getting 100MB from the comments in the config, which
suggest 100MB per core.  This advice is pretty outdated and should be
updated.

I'd use 8GB total heap and 4GB new gen as a starting point.  I really
suggest reading up on how GC works, I linked to a post in an earlier email.

These are the flags you'd need to set in your jvm.options, or
jvm-server.options depending on the version you're using:

-Xmx8G
-Xms8G
-Xmn4G

1 core is probably going to be a problem, Cassandra creates a lot of
threads and relies on doing work concurrently.  I wouldn't use less than 8
cores in a production environment.

On Sun, Mar 17, 2019 at 3:12 AM Dieudonné Madishon NGAYA 
wrote:

> Starting point for me: max_heap_size to 8gb and heap_newsize to 100mb.
> Then restart node by node then watch system.log to see if you are seeing G.C
>
> On Sat, Mar 16, 2019 at 9:56 AM Sundaramoorthy, Natarajan <
> natarajan_sundaramoor...@optum.com> wrote:
>
>> So you guys are suggesting
>>
>>
>>
>> MAX_HEAP_SIZE  by 8/12/16GB
>>
>>
>>
>> And
>>
>>
>>
>> HEAP_NEWSIZE to 100 MB
>>
>>
>>
>> And
>>
>>
>>
>> heap with 50% of that as a starting point? Hw do I do this?
>>
>>
>>
>> Thanks
>>
>>
>>
>>
>>
>> *From:* Dieudonné Madishon NGAYA [mailto:dmng...@gmail.com]
>> *Sent:* Saturday, March 16, 2019 12:15 AM
>> *To:* user@cassandra.apache.org
>> *Subject:* Re: read request is slow
>>
>>
>>
>> I agreed with jon haddad , your MAX_HEAP_SIZE is very small. you have lot
>> of RAM (256 GB), you can start your  MAX_HEAP_SIZE  by 8GB and increase if
>> necessary.
>>
>> Since you have only 1 physical core if i understood , you can set your 
>> HEAP_NEWSIZE
>> to 100 MB
>>
>>
>>
>> Best regards
>>
>> _
>>
>>
>> [image:
>> https://www.facebook.com/DMN-BigData-371074727032197/?modal=admin_todo_tour]
>> 
>>   
>> 
>>
>> *Dieudonne Madishon NGAYA*
>> Datastax, Cassandra Architect
>> *P: *7048580065
>> *w: *www.dmnbigdata.com
>> *E: *dmng...@dmnbigdata.com
>> *Private E: *dmng...@gmail.com
>> *A: *Charlotte,NC,28273, USA
>>
>>
>>
>>
>>
>>
>>
>> On Sat, Mar 16, 2019 at 1:07 AM Jon Haddad  wrote:
>>
>> I can't say I've ever used 100MB new gen with Cassandra, but in my
>> experience I've found small new gen to be incredibly harmful for
>> performance.  It doesn't surprise me at all that you'd hit some serious GC
>> issues.  My guess is you're filling up the new gen very quickly and
>> promoting everything in very quick cycles, leading to memory fragmentation
>> and soon after full GCs.  2GB is a tiny heap and I would never, under any
>> circumstances, run a 2GB heap in a production environment.  I'd only use
>> under 8 GB in a circle CI free tier for integration tests.
>>
>>
>>
>> I suggest you use a minimum of 8, preferably 12-16GB of total heap with
>> 50% of that as a starting point.  There's a bunch of posts floating around
>> on the topic, here's one I wrote:
>> http://thelastpickle.com/blog/2018/04/11/gc-tuning.html
>>
>>
>>
>> Jon
>>
>>
>>
>> On Sat, Mar 16, 2019 at 5:49 PM Sundaramoorthy, Natarajan <
>> natarajan_sundaramoor...@optum.com> wrote:
>>
>> Here you go. Thanks
>>
>> - name: MAX_HEAP_SIZE
>>
>>   value: 2048M
>>
>> - name: MY_POD_NAMESPACE
>>
>>   valueFrom:
>>
>> fieldRef:
>>
>>   apiVersion: v1
>>
>>   fieldPath: metadata.namespace
>>
>> - name: HEAP_NEWSIZE
>>
>>   value: 100M
>>
>>
>>
>>
>>
>>
>>
>> *From:* Dieudonné Madishon NGAYA [mailto:dmng...@gmail.com]
>> *Sent:* Friday, March 15, 2019 11:18 PM
>> *To:* user@cassandra.apache.org
>> *Subject:* Re: read request is slow
>>
>>
>>
>> Is it possible to have these parameters from cassandra-env.sh if they are
>> set:
>>
>> MAX_HEAP_SIZE and HEAP_NEWSIZE
>>
>>
>>
>> Best regards
>>
>> _
>>
>>
>> [image:
>> https://www.facebook.com/DMN-BigData-371074727032197/?modal=admin_todo_tour]
>> 
>>   
>> 
>>
>> *Dieudonne Madishon NGAYA*
>> Datastax, Cassandra Architect
>> *P: *7048580065
>> *w: *www.dmnbigdata.com
>> *E: *dmng...@dmnbigdata.com
>> *Private E: *dmng...@gmail.com
>> *A: *Charlotte,NC,28273, USA
>>
>>
>>
>>
>>
>>
>>
>> On Sat, Mar 16, 2019 at 12:10 AM Sundaramoorthy, Natarajan <
>> natarajan_sundaramoor...@optum.com> wrote:
>>
>> Thanks for the quick response.
>>
>>
>>
>> Here is the cassandra.yaml attached.
>>
>>
>>
>> 1.  What was the read request?  Are you fetching a single row, a
>> million, something else?
>>
>>
>>
>> *Trying to get the details*
>>
>>
>>
>> 2. What are your GC 

Re: read request is slow

[Dieudonné Madishon NGAYA]

Starting point for me: max_heap_size to 8gb and heap_newsize to 100mb. Then
restart node by node then watch system.log to see if you are seeing G.C

On Sat, Mar 16, 2019 at 9:56 AM Sundaramoorthy, Natarajan <
natarajan_sundaramoor...@optum.com> wrote:

> So you guys are suggesting
>
>
>
> MAX_HEAP_SIZE  by 8/12/16GB
>
>
>
> And
>
>
>
> HEAP_NEWSIZE to 100 MB
>
>
>
> And
>
>
>
> heap with 50% of that as a starting point? Hw do I do this?
>
>
>
> Thanks
>
>
>
>
>
> *From:* Dieudonné Madishon NGAYA [mailto:dmng...@gmail.com]
> *Sent:* Saturday, March 16, 2019 12:15 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: read request is slow
>
>
>
> I agreed with jon haddad , your MAX_HEAP_SIZE is very small. you have lot
> of RAM (256 GB), you can start your  MAX_HEAP_SIZE  by 8GB and increase if
> necessary.
>
> Since you have only 1 physical core if i understood , you can set your 
> HEAP_NEWSIZE
> to 100 MB
>
>
>
> Best regards
>
> _
>
>
> [image:
> https://www.facebook.com/DMN-BigData-371074727032197/?modal=admin_todo_tour]
> 
>   
> 
>
> *Dieudonne Madishon NGAYA*
> Datastax, Cassandra Architect
> *P: *7048580065
> *w: *www.dmnbigdata.com
> *E: *dmng...@dmnbigdata.com
> *Private E: *dmng...@gmail.com
> *A: *Charlotte,NC,28273, USA
>
>
>
>
>
>
>
> On Sat, Mar 16, 2019 at 1:07 AM Jon Haddad  wrote:
>
> I can't say I've ever used 100MB new gen with Cassandra, but in my
> experience I've found small new gen to be incredibly harmful for
> performance.  It doesn't surprise me at all that you'd hit some serious GC
> issues.  My guess is you're filling up the new gen very quickly and
> promoting everything in very quick cycles, leading to memory fragmentation
> and soon after full GCs.  2GB is a tiny heap and I would never, under any
> circumstances, run a 2GB heap in a production environment.  I'd only use
> under 8 GB in a circle CI free tier for integration tests.
>
>
>
> I suggest you use a minimum of 8, preferably 12-16GB of total heap with
> 50% of that as a starting point.  There's a bunch of posts floating around
> on the topic, here's one I wrote:
> http://thelastpickle.com/blog/2018/04/11/gc-tuning.html
>
>
>
> Jon
>
>
>
> On Sat, Mar 16, 2019 at 5:49 PM Sundaramoorthy, Natarajan <
> natarajan_sundaramoor...@optum.com> wrote:
>
> Here you go. Thanks
>
> - name: MAX_HEAP_SIZE
>
>   value: 2048M
>
> - name: MY_POD_NAMESPACE
>
>   valueFrom:
>
> fieldRef:
>
>   apiVersion: v1
>
>   fieldPath: metadata.namespace
>
> - name: HEAP_NEWSIZE
>
>   value: 100M
>
>
>
>
>
>
>
> *From:* Dieudonné Madishon NGAYA [mailto:dmng...@gmail.com]
> *Sent:* Friday, March 15, 2019 11:18 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: read request is slow
>
>
>
> Is it possible to have these parameters from cassandra-env.sh if they are
> set:
>
> MAX_HEAP_SIZE and HEAP_NEWSIZE
>
>
>
> Best regards
>
> _
>
>
> [image:
> https://www.facebook.com/DMN-BigData-371074727032197/?modal=admin_todo_tour]
> 
>   
> 
>
> *Dieudonne Madishon NGAYA*
> Datastax, Cassandra Architect
> *P: *7048580065
> *w: *www.dmnbigdata.com
> *E: *dmng...@dmnbigdata.com
> *Private E: *dmng...@gmail.com
> *A: *Charlotte,NC,28273, USA
>
>
>
>
>
>
>
> On Sat, Mar 16, 2019 at 12:10 AM Sundaramoorthy, Natarajan <
> natarajan_sundaramoor...@optum.com> wrote:
>
> Thanks for the quick response.
>
>
>
> Here is the cassandra.yaml attached.
>
>
>
> 1.  What was the read request?  Are you fetching a single row, a
> million, something else?
>
>
>
> *Trying to get the details*
>
>
>
> 2. What are your GC settings?
>
>
>
> *I have no name!@cassandra-0:/etc/cassandra$ nodetool gcstats*
>
> *   Interval (ms) Max GC Elapsed (ms)Total GC Elapsed (ms)Stdev GC
> Elapsed (ms)   GC Reclaimed (MB) Collections  Direct Memory
> Bytes*
>
> *   54292 157
> 157   0   317432560
>1   -1*
>
> *I have no name!@cassandra-0:/etc/cassandra$*
>
> #
>
>
>
> 3. What's the hardware in use?  What resources have been allocated to each
> instance?
>
>
>
> *CPU: 1 core to 1 core*
>
> *Memory: 4 GiB to 4 GiB*
>
>
>
> *have no name!@cassandra-0:/etc/cassandra$ free -h*
>
> *  totalusedfree  shared  buff/cache
> available*
>
> *Mem:   251G 79G 39G122M
> 132G169G*
>
> *Swap:0B  0B  0B*
>
> *I have no