Re: [EXTERNAL] How to reduce vnodes without downtime

[Erick Ramirez]

>
> If you are after more details about the trade-offs between different sized
> token values, please see the discussion on the dev mailing list: "[Discuss]
> num_tokens default in Cassandra 4.0
> 
> ".
>
> Regards,
> Anthony
>
> On Sat, 1 Feb 2020 at 10:07, Sergio  wrote:
>
>>
>> https://thelastpickle.com/blog/2019/02/21/set-up-a-cluster-with-even-token-distribution.html
>>  This
>> is the article with 4 token recommendations.
>> @Erick Ramirez. which is the dev thread for the default 32 tokens
>> recommendation?
>>
>> Thanks,
>> Sergio
>>
>
Sergio, my apologies for not replying. For some reason, your reply went to
my spam folder and I didn't see it.

Thanks, Anthony, for responding. I was indeed referring to that dev thread.
Cheers!

Re: [EXTERNAL] How to reduce vnodes without downtime

[Sergio]

Thanks Anthony!

I will read more about it

Best,

Sergio



Il giorno dom 2 feb 2020 alle ore 18:36 Anthony Grasso <
anthony.gra...@gmail.com> ha scritto:

> Hi Sergio,
>
> There is a misunderstanding here. My post makes no recommendation for the
> value of num_tokens. Rather, it focuses on how to use
> the allocate_tokens_for_keyspace setting when creating a new cluster.
>
> Whilst a value of 4 is used for num_tokens in the post, it was chosen for
> demonstration purposes. Specifically it makes:
>
>- the uneven token distribution in a small cluster very obvious,
>- identifying the endpoints displayed in nodetool ring easy, and
>- the initial_token setup less verbose and easier to follow.
>
> I will add an editorial note to the post with the above information
> so there is no confusion about why 4 tokens were used.
>
> I would only consider moving a cluster to 4 tokens if it is larger than
> 100 nodes. If you read through the paper that Erick mentioned, written
> by Joe Lynch & Josh Snyder, they show that the num_tokens impacts the
> availability of large scale clusters.
>
> If you are after more details about the trade-offs between different sized
> token values, please see the discussion on the dev mailing list: "[Discuss]
> num_tokens default in Cassandra 4.0
> 
> ".
>
> Regards,
> Anthony
>
> On Sat, 1 Feb 2020 at 10:07, Sergio  wrote:
>
>>
>> https://thelastpickle.com/blog/2019/02/21/set-up-a-cluster-with-even-token-distribution.html
>>  This
>> is the article with 4 token recommendations.
>> @Erick Ramirez. which is the dev thread for the default 32 tokens
>> recommendation?
>>
>> Thanks,
>> Sergio
>>
>> Il giorno ven 31 gen 2020 alle ore 14:49 Erick Ramirez <
>> flightc...@gmail.com> ha scritto:
>>
>>> There's an active discussion going on right now in a separate dev
>>> thread. The current "default recommendation" is 32 tokens. But there's a
>>> push for 4 in combination with allocate_tokens_for_keyspace from Jon
>>> Haddad & co (based on a paper from Joe Lynch & Josh Snyder).
>>>
>>> If you're satisfied with the results from your own testing, go with 4
>>> tokens. And that's the key -- you must test, test, TEST! Cheers!
>>>
>>> On Sat, Feb 1, 2020 at 5:17 AM Arvinder Dhillon 
>>> wrote:
>>>
 What is recommended vnodes now? I read 8 in later cassandra 3.x
 Is the new recommendation 4 now even in version 3.x (asking for 3.11)?
 Thanks

 On Fri, Jan 31, 2020 at 9:49 AM Durity, Sean R <
 sean_r_dur...@homedepot.com> wrote:

> These are good clarifications and expansions.
>
>
>
> Sean Durity
>
>
>
> *From:* Anthony Grasso 
> *Sent:* Thursday, January 30, 2020 7:25 PM
> *To:* user 
> *Subject:* Re: [EXTERNAL] How to reduce vnodes without downtime
>
>
>
> Hi Maxim,
>
>
>
> Basically what Sean suggested is the way to do this without downtime.
>
>
>
> To clarify the, the *three* steps following the "Decommission each
> node in the DC you are working on" step should be applied to *only*
> the decommissioned nodes. So where it say "*all nodes*" or "*every
> node*" it applies to only the decommissioned nodes.
>
>
>
> In addition, the step that says "Wipe data on all the nodes", I would
> delete all files in the following directories on the decommissioned nodes.
>
>- data (usually located in /var/lib/cassandra/data)
>- commitlogs (usually located in /var/lib/cassandra/commitlogs)
>- hints (usually located in /var/lib/casandra/hints)
>- saved_caches (usually located in /var/lib/cassandra/saved_caches)
>
>
>
> Cheers,
>
> Anthony
>
>
>
> On Fri, 31 Jan 2020 at 03:05, Durity, Sean R <
> sean_r_dur...@homedepot.com> wrote:
>
> Your procedure won’t work very well. On the first node, if you
> switched to 4, you would end up with only a tiny fraction of the data
> (because the other nodes would still be at 256). I updated a large cluster
> (over 150 nodes – 2 DCs) to smaller number of vnodes. The basic outline 
> was
> this:
>
>
>
>- Stop all repairs
>- Make sure the app is running against one DC only
>- Change the replication settings on keyspaces to use only 1 DC
>(basically cutting off the other DC)
>- Decommission each node in the DC you are working on. Because the
>replication setting are changed, no streaming occurs. But it releases 
> the
>token assignments
>- Wipe data on all the nodes
>- Update configuration on every node to your new settings,
>including auto_bootstrap = false
>- Start all nodes. They will choose tokens, but not stream any data
>- Update replication factor for all 

Re: [EXTERNAL] How to reduce vnodes without downtime

[Anthony Grasso]

Hi Sergio,

There is a misunderstanding here. My post makes no recommendation for the
value of num_tokens. Rather, it focuses on how to use
the allocate_tokens_for_keyspace setting when creating a new cluster.

Whilst a value of 4 is used for num_tokens in the post, it was chosen for
demonstration purposes. Specifically it makes:

   - the uneven token distribution in a small cluster very obvious,
   - identifying the endpoints displayed in nodetool ring easy, and
   - the initial_token setup less verbose and easier to follow.

I will add an editorial note to the post with the above information
so there is no confusion about why 4 tokens were used.

I would only consider moving a cluster to 4 tokens if it is larger than 100
nodes. If you read through the paper that Erick mentioned, written by Joe
Lynch & Josh Snyder, they show that the num_tokens impacts the availability
of large scale clusters.

If you are after more details about the trade-offs between different sized
token values, please see the discussion on the dev mailing list: "[Discuss]
num_tokens default in Cassandra 4.0

".

Regards,
Anthony

On Sat, 1 Feb 2020 at 10:07, Sergio  wrote:

>
> https://thelastpickle.com/blog/2019/02/21/set-up-a-cluster-with-even-token-distribution.html
>  This
> is the article with 4 token recommendations.
> @Erick Ramirez. which is the dev thread for the default 32 tokens
> recommendation?
>
> Thanks,
> Sergio
>
> Il giorno ven 31 gen 2020 alle ore 14:49 Erick Ramirez <
> flightc...@gmail.com> ha scritto:
>
>> There's an active discussion going on right now in a separate dev thread.
>> The current "default recommendation" is 32 tokens. But there's a push for 4
>> in combination with allocate_tokens_for_keyspace from Jon Haddad & co
>> (based on a paper from Joe Lynch & Josh Snyder).
>>
>> If you're satisfied with the results from your own testing, go with 4
>> tokens. And that's the key -- you must test, test, TEST! Cheers!
>>
>> On Sat, Feb 1, 2020 at 5:17 AM Arvinder Dhillon 
>> wrote:
>>
>>> What is recommended vnodes now? I read 8 in later cassandra 3.x
>>> Is the new recommendation 4 now even in version 3.x (asking for 3.11)?
>>> Thanks
>>>
>>> On Fri, Jan 31, 2020 at 9:49 AM Durity, Sean R <
>>> sean_r_dur...@homedepot.com> wrote:
>>>
 These are good clarifications and expansions.



 Sean Durity



 *From:* Anthony Grasso 
 *Sent:* Thursday, January 30, 2020 7:25 PM
 *To:* user 
 *Subject:* Re: [EXTERNAL] How to reduce vnodes without downtime



 Hi Maxim,



 Basically what Sean suggested is the way to do this without downtime.



 To clarify the, the *three* steps following the "Decommission each
 node in the DC you are working on" step should be applied to *only*
 the decommissioned nodes. So where it say "*all nodes*" or "*every
 node*" it applies to only the decommissioned nodes.



 In addition, the step that says "Wipe data on all the nodes", I would
 delete all files in the following directories on the decommissioned nodes.

- data (usually located in /var/lib/cassandra/data)
- commitlogs (usually located in /var/lib/cassandra/commitlogs)
- hints (usually located in /var/lib/casandra/hints)
- saved_caches (usually located in /var/lib/cassandra/saved_caches)



 Cheers,

 Anthony



 On Fri, 31 Jan 2020 at 03:05, Durity, Sean R <
 sean_r_dur...@homedepot.com> wrote:

 Your procedure won’t work very well. On the first node, if you switched
 to 4, you would end up with only a tiny fraction of the data (because the
 other nodes would still be at 256). I updated a large cluster (over 150
 nodes – 2 DCs) to smaller number of vnodes. The basic outline was this:



- Stop all repairs
- Make sure the app is running against one DC only
- Change the replication settings on keyspaces to use only 1 DC
(basically cutting off the other DC)
- Decommission each node in the DC you are working on. Because the
replication setting are changed, no streaming occurs. But it releases 
 the
token assignments
- Wipe data on all the nodes
- Update configuration on every node to your new settings,
including auto_bootstrap = false
- Start all nodes. They will choose tokens, but not stream any data
- Update replication factor for all keyspaces to include the new DC
- I disabled binary on those nodes to prevent app connections
- Run nodetool reduild with -dc (other DC) on as many nodes as your
system can safely handle until they are all rebuilt.
- Re-enable binary (and app connections to the rebuilt DC)
- Turn on