Re: Fw: read request is slow

[Jon Haddad]

Sjk plus is bundled with dse, not oss.

Grab it here if you need it:
https://github.com/aragozin/jvm-tools

On Mon, Mar 18, 2019 at 6:38 AM Dieudonné Madishon NGAYA 
wrote:

> Never mind : nodetool sjk help
>
> On Mon, Mar 18, 2019 at 9:36 AM Dieudonné Madishon NGAYA <
> dmng...@gmail.com> wrote:
>
>> Hi, please send us result of: nodetool help sjk
>>
>> On Mon, Mar 18, 2019 at 9:28 AM ishib...@gmail.com 
>> wrote:
>>
>>> Hi!
>>>
>>> This is interesting note for me.
>>>
>>> We have C* 3.11.1, but ‘sjk’ still  unexpected parameters for nodetool
>>> utility.
>>>
>>> Am I  missed something or ‘sjk’ available as separate rpm-package?
>>>
>>>
>>>
>>> Best regards, Ilya
>>>
>>>
>>> 
>>> Тема: Re: read request is slow
>>> От: Dieudonné Madishon NGAYA
>>> Кому: user@cassandra.apache.org
>>> Копия:
>>>
>>>
>>>
>>> For your information,since cassandra 3.0, it includes ttop and other
>>> options inside sjk
>>> nodetool sjk
>>>
>>> https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsSjk.html
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *From:* Jon Haddad [mailto:j...@jonhaddad.com]
>>> *Sent:* Saturday, March 16, 2019 5:25 PM
>>> *To:* user@cassandra.apache.org
>>> *Subject:* Re: read request is slow
>>>
>>>
>>>
>>> I'm guessing you're getting 100MB from the comments in the config, which
>>> suggest 100MB per core.  This advice is pretty outdated and should be
>>> updated.
>>>
>>>
>>>
>>> I'd use 8GB total heap and 4GB new gen as a starting point.  I really
>>> suggest reading up on how GC works, I linked to a post in an earlier email.
>>>
>>>
>>>
>>> These are the flags you'd need to set in your jvm.options, or
>>> jvm-server.options depending on the version you're using:
>>>
>>>
>>>
>>> -Xmx8G
>>>
>>> -Xms8G
>>>
>>> -Xmn4G
>>>
>>>
>>>
>>> 1 core is probably going to be a problem, Cassandra creates a lot of
>>> threads and relies on doing work concurrently.  I wouldn't use less than 8
>>> cores in a production environment.
>>>
>>>
>>> ---
>>> This message and any attachment are confidential and may be privileged
>>> or otherwise protected from disclosure. If you are not the intended
>>> recipient any use, distribution, copying or disclosure is strictly
>>> prohibited. If you have received this message in error, please notify the
>>> sender immediately either by telephone or by e-mail and delete this message
>>> and any attachment from your system. Correspondence via e-mail is for
>>> information purposes only. AO Raiffeisenbank neither makes nor accepts
>>> legally binding statements by e-mail unless otherwise agreed.
>>> ---
>>>
>>> - To
>>> unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For
>>> additional commands, e-mail: user-h...@cassandra.apache.org
>>
>> --
>>
>> Best regards
>> _
>>
>> [image:
>> https://www.facebook.com/DMN-BigData-371074727032197/?modal=admin_todo_tour]
>> 
>>   
>> 
>>
>> *Dieudonne Madishon NGAYA*
>> Datastax, Cassandra Architect
>> *P: *7048580065
>> *w: *www.dmnbigdata.com
>> *E: *dmng...@dmnbigdata.com
>> *Private E: *dmng...@gmail.com
>> *A: *Charlotte,NC,28273, USA
>>
> --
>
> Best regards
> _
>
> [image:
> https://www.facebook.com/DMN-BigData-371074727032197/?modal=admin_todo_tour]
> 
>   
> 
>
> *Dieudonne Madishon NGAYA*
> Datastax, Cassandra Architect
> *P: *7048580065
> *w: *www.dmnbigdata.com
> *E: *dmng...@dmnbigdata.com
> *Private E: *dmng...@gmail.com
> *A: *Charlotte,NC,28273, USA
>

Re: Fw: read request is slow

[Dieudonné Madishon NGAYA]

Never mind : nodetool sjk help

On Mon, Mar 18, 2019 at 9:36 AM Dieudonné Madishon NGAYA 
wrote:

> Hi, please send us result of: nodetool help sjk
>
> On Mon, Mar 18, 2019 at 9:28 AM ishib...@gmail.com 
> wrote:
>
>> Hi!
>>
>> This is interesting note for me.
>>
>> We have C* 3.11.1, but ‘sjk’ still  unexpected parameters for nodetool
>> utility.
>>
>> Am I  missed something or ‘sjk’ available as separate rpm-package?
>>
>>
>>
>> Best regards, Ilya
>>
>>
>> 
>> Тема: Re: read request is slow
>> От: Dieudonné Madishon NGAYA
>> Кому: user@cassandra.apache.org
>> Копия:
>>
>>
>>
>> For your information,since cassandra 3.0, it includes ttop and other
>> options inside sjk
>> nodetool sjk
>>
>> https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsSjk.html
>>
>>
>>
>>
>>
>>
>>
>> *From:* Jon Haddad [mailto:j...@jonhaddad.com]
>> *Sent:* Saturday, March 16, 2019 5:25 PM
>> *To:* user@cassandra.apache.org
>> *Subject:* Re: read request is slow
>>
>>
>>
>> I'm guessing you're getting 100MB from the comments in the config, which
>> suggest 100MB per core.  This advice is pretty outdated and should be
>> updated.
>>
>>
>>
>> I'd use 8GB total heap and 4GB new gen as a starting point.  I really
>> suggest reading up on how GC works, I linked to a post in an earlier email.
>>
>>
>>
>> These are the flags you'd need to set in your jvm.options, or
>> jvm-server.options depending on the version you're using:
>>
>>
>>
>> -Xmx8G
>>
>> -Xms8G
>>
>> -Xmn4G
>>
>>
>>
>> 1 core is probably going to be a problem, Cassandra creates a lot of
>> threads and relies on doing work concurrently.  I wouldn't use less than 8
>> cores in a production environment.
>>
>>
>> ---
>> This message and any attachment are confidential and may be privileged or
>> otherwise protected from disclosure. If you are not the intended recipient
>> any use, distribution, copying or disclosure is strictly prohibited. If you
>> have received this message in error, please notify the sender immediately
>> either by telephone or by e-mail and delete this message and any attachment
>> from your system. Correspondence via e-mail is for information purposes
>> only. AO Raiffeisenbank neither makes nor accepts legally binding
>> statements by e-mail unless otherwise agreed.
>> ---
>>
>> - To
>> unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For
>> additional commands, e-mail: user-h...@cassandra.apache.org
>
> --
>
> Best regards
> _
>
> [image:
> https://www.facebook.com/DMN-BigData-371074727032197/?modal=admin_todo_tour]
> 
>   
> 
>
> *Dieudonne Madishon NGAYA*
> Datastax, Cassandra Architect
> *P: *7048580065
> *w: *www.dmnbigdata.com
> *E: *dmng...@dmnbigdata.com
> *Private E: *dmng...@gmail.com
> *A: *Charlotte,NC,28273, USA
>
-- 

Best regards
_

[image:
https://www.facebook.com/DMN-BigData-371074727032197/?modal=admin_todo_tour]

      


*Dieudonne Madishon NGAYA*
Datastax, Cassandra Architect
*P: *7048580065
*w: *www.dmnbigdata.com
*E: *dmng...@dmnbigdata.com
*Private E: *dmng...@gmail.com
*A: *Charlotte,NC,28273, USA

Re: Fw: read request is slow

[Dieudonné Madishon NGAYA]

Hi, please send us result of: nodetool help sjk

On Mon, Mar 18, 2019 at 9:28 AM ishib...@gmail.com 
wrote:

> Hi!
>
> This is interesting note for me.
>
> We have C* 3.11.1, but ‘sjk’ still  unexpected parameters for nodetool
> utility.
>
> Am I  missed something or ‘sjk’ available as separate rpm-package?
>
>
>
> Best regards, Ilya
>
>
> 
> Тема: Re: read request is slow
> От: Dieudonné Madishon NGAYA
> Кому: user@cassandra.apache.org
> Копия:
>
>
>
> For your information,since cassandra 3.0, it includes ttop and other
> options inside sjk
> nodetool sjk
>
> https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsSjk.html
>
>
>
>
>
>
>
> *From:* Jon Haddad [mailto:j...@jonhaddad.com]
> *Sent:* Saturday, March 16, 2019 5:25 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: read request is slow
>
>
>
> I'm guessing you're getting 100MB from the comments in the config, which
> suggest 100MB per core.  This advice is pretty outdated and should be
> updated.
>
>
>
> I'd use 8GB total heap and 4GB new gen as a starting point.  I really
> suggest reading up on how GC works, I linked to a post in an earlier email.
>
>
>
> These are the flags you'd need to set in your jvm.options, or
> jvm-server.options depending on the version you're using:
>
>
>
> -Xmx8G
>
> -Xms8G
>
> -Xmn4G
>
>
>
> 1 core is probably going to be a problem, Cassandra creates a lot of
> threads and relies on doing work concurrently.  I wouldn't use less than 8
> cores in a production environment.
>
>
> ---
> This message and any attachment are confidential and may be privileged or
> otherwise protected from disclosure. If you are not the intended recipient
> any use, distribution, copying or disclosure is strictly prohibited. If you
> have received this message in error, please notify the sender immediately
> either by telephone or by e-mail and delete this message and any attachment
> from your system. Correspondence via e-mail is for information purposes
> only. AO Raiffeisenbank neither makes nor accepts legally binding
> statements by e-mail unless otherwise agreed.
> ---
>
> - To
> unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional
> commands, e-mail: user-h...@cassandra.apache.org

-- 

Best regards
_

[image:
https://www.facebook.com/DMN-BigData-371074727032197/?modal=admin_todo_tour]

      


*Dieudonne Madishon NGAYA*
Datastax, Cassandra Architect
*P: *7048580065
*w: *www.dmnbigdata.com
*E: *dmng...@dmnbigdata.com
*Private E: *dmng...@gmail.com
*A: *Charlotte,NC,28273, USA

Fw: read request is slow

[ishib...@gmail.com]

Hi!This is interesting note for me.We have C* 3.11.1, but ‘sjk’ still  unexpected parameters for nodetool utility.Am I  missed something or ‘sjk’ available as separate rpm-package?Best regards, Ilya

Тема: Re: read request is slow
От: Dieudonné Madishon NGAYA 
Кому: user@cassandra.apache.org
Копия: 


























For your information,since cassandra 3.0, it includes ttop and other options inside sjk



nodetool sjk


https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsSjk.html

































 


 








 
From: Jon Haddad
 [mailto:j...@jonhaddad.com]

Sent: Saturday, March 16, 2019 5:25 PM
To: user@cassandra.apache.org
Subject: Re: read request is slow
 

I'm guessing you're getting 100MB from the comments in the config, which suggest 100MB per core.  This advice is pretty outdated and should be updated.

 


I'd use 8GB total heap and 4GB new gen as a starting point.  I really suggest reading up on how GC works, I linked to a post in an earlier email.


 


These are the flags you'd need to set in your jvm.options, or jvm-server.options depending on the version you're using:


 


-Xmx8G


-Xms8G


-Xmn4G


 


1 core is probably going to be a problem, Cassandra creates a lot of threads and relies on doing work concurrently.  I wouldn't use less than 8 cores in a production
 environment.













---
This message and any attachment are confidential and may be privileged or otherwise protected from disclosure. If you are not the intended recipient any use, distribution, copying or disclosure is strictly prohibited. If you have received this message in error, please notify the sender immediately either by telephone or by e-mail and delete this message and any attachment from your system. Correspondence via e-mail is for information purposes only. AO Raiffeisenbank neither makes nor accepts legally binding statements by e-mail unless otherwise agreed. 
---


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: TWCS and tombstone purging

[Alexander Dejanovski]

Hi Nick,

the strategy will depend on your compaction strategy and how tombstones are
generated (DELETE statements or TTLs), and also your version of Cassandra.

If you're working with TTLs, your best option is definitely TWCS with the
unsafe_aggressive_sstable_expiration flag that was introduced by
CASSANDRA-13418 .
It'll delete all fully expired SSTables even when there are timestamp
overlaps with other SSTables. If you have different TTLs, you can also
enable *unchecked_tombstone_compaction* to trigger single sstables
compactions more often (and adjust the tombstone_threshold to your
particular workload). You can lower gc_grace_seconds to 3 hours (no less
otherwise you'll reduce the hint window) in order to avoid keeping
tombstones on disk.
That's the easy case.

Then if you're generating tombstones from DELETE statements, it can be
trickier as you'll need the tombstones to be compacted with the data they
shadow in order to get a chance to evict it eventually. You also cannot
reduce gc_grace_seconds below your repair cycle as it will create a
possibility of reviving deleted data (zombie data).
LCS doesn't get along very well with tombstones, as they can get "stuck" in
higher level with the data they shadow being stored in the lower levels.
LCS major compactions are also fairly long to run (and single threaded).
TWCS doesn't apply to data that isn't TTLed (your tombstones will possibly
be stored in a different time window than the data they shadow).
That leaves us with STCS. If you want to be as aggressive as possible there
and purge your deletes ASAP, you'll need to :

   - run repair very often to secure your deletions
   - reduce gc_grace_seconds to a value that's slighty higher than your
   repair cycle
   - run major compactions with the -s flag, in order to avoid creating a
   single big file, and create one file per size tier instead. The best idea
   that I can think of here is to trigger a major compaction right after a
   successful repair.

We have a few posts on our blog  that cover
the tombstones and compaction strategies topic (search for "tombstone" on
that page), notably this one:
http://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html

Cheers,



On Sat, Mar 16, 2019 at 1:04 AM Nick Hatfield 
wrote:

> Hey guys,
>
>
>
> Can someone give me some idea or link some good material for determining a
> good / aggressive tombstone strategy? I want to make sure my tombstones are
> getting purged as soon as possible to reclaim disk.
>
>
>
> Thanks
>
-- 
-
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com