There's an active discussion going on right now in a separate dev thread. The current "default recommendation" is 32 tokens. But there's a push for 4 in combination with allocate_tokens_for_keyspace from Jon Haddad & co (based on a paper from Joe Lynch & Josh Snyder).
If you're satisfied with the results from your own testing, go with 4 tokens. And that's the key -- you must test, test, TEST! Cheers! On Sat, Feb 1, 2020 at 5:17 AM Arvinder Dhillon <dhillona...@gmail.com> wrote: > What is recommended vnodes now? I read 8 in later cassandra 3.x > Is the new recommendation 4 now even in version 3.x (asking for 3.11)? > Thanks > > On Fri, Jan 31, 2020 at 9:49 AM Durity, Sean R < > sean_r_dur...@homedepot.com> wrote: > >> These are good clarifications and expansions. >> >> >> >> Sean Durity >> >> >> >> *From:* Anthony Grasso <anthony.gra...@gmail.com> >> *Sent:* Thursday, January 30, 2020 7:25 PM >> *To:* user <user@cassandra.apache.org> >> *Subject:* Re: [EXTERNAL] How to reduce vnodes without downtime >> >> >> >> Hi Maxim, >> >> >> >> Basically what Sean suggested is the way to do this without downtime. >> >> >> >> To clarify the, the *three* steps following the "Decommission each node >> in the DC you are working on" step should be applied to *only* the >> decommissioned nodes. So where it say "*all nodes*" or "*every node*" it >> applies to only the decommissioned nodes. >> >> >> >> In addition, the step that says "Wipe data on all the nodes", I would >> delete all files in the following directories on the decommissioned nodes. >> >> - data (usually located in /var/lib/cassandra/data) >> - commitlogs (usually located in /var/lib/cassandra/commitlogs) >> - hints (usually located in /var/lib/casandra/hints) >> - saved_caches (usually located in /var/lib/cassandra/saved_caches) >> >> >> >> Cheers, >> >> Anthony >> >> >> >> On Fri, 31 Jan 2020 at 03:05, Durity, Sean R <sean_r_dur...@homedepot.com> >> wrote: >> >> Your procedure won’t work very well. On the first node, if you switched >> to 4, you would end up with only a tiny fraction of the data (because the >> other nodes would still be at 256). I updated a large cluster (over 150 >> nodes – 2 DCs) to smaller number of vnodes. The basic outline was this: >> >> >> >> - Stop all repairs >> - Make sure the app is running against one DC only >> - Change the replication settings on keyspaces to use only 1 DC >> (basically cutting off the other DC) >> - Decommission each node in the DC you are working on. Because the >> replication setting are changed, no streaming occurs. But it releases the >> token assignments >> - Wipe data on all the nodes >> - Update configuration on every node to your new settings, including >> auto_bootstrap = false >> - Start all nodes. They will choose tokens, but not stream any data >> - Update replication factor for all keyspaces to include the new DC >> - I disabled binary on those nodes to prevent app connections >> - Run nodetool reduild with -dc (other DC) on as many nodes as your >> system can safely handle until they are all rebuilt. >> - Re-enable binary (and app connections to the rebuilt DC) >> - Turn on repairs >> - Rest for a bit, then reverse the process for the remaining DCs >> >> >> >> >> >> >> >> Sean Durity – Staff Systems Engineer, Cassandra >> >> >> >> *From:* Maxim Parkachov <lazy.gop...@gmail.com> >> *Sent:* Thursday, January 30, 2020 10:05 AM >> *To:* user@cassandra.apache.org >> *Subject:* [EXTERNAL] How to reduce vnodes without downtime >> >> >> >> Hi everyone, >> >> >> >> with discussion about reducing default vnodes in version 4.0 I would like >> to ask, what would be optimal procedure to perform reduction of vnodes in >> existing 3.11.x cluster which was set up with default value 256. Cluster >> has 2 DC with 5 nodes each and RF=3. There is one more restriction, I could >> not add more servers, nor to create additional DC, everything is physical. >> This should be done without downtime. >> >> >> >> My idea for such procedure would be >> >> >> >> for each node: >> >> - decommission node >> >> - set auto_bootstrap to true and vnodes to 4 >> >> - start and wait till node joins cluster >> >> - run cleanup on rest of nodes in cluster >> >> - run repair on whole cluster (not sure if needed after cleanup) >> >> - set auto_bootstrap to false >> >> repeat for each node >> >> >> >> rolling restart of cluster >> >> cluster repair >> >> >> >> Is this sounds right ? My concern is that after decommission, node will >> start on the same IP which could create some confusion. >> >> >> >> Regards, >> >> Maxim. >> >> >> ------------------------------ >> >> >> The information in this Internet Email is confidential and may be legally >> privileged. It is intended solely for the addressee. Access to this Email >> by anyone else is unauthorized. If you are not the intended recipient, any >> disclosure, copying, distribution or any action taken or omitted to be >> taken in reliance on it, is prohibited and may be unlawful. When addressed >> to our clients any opinions or advice contained in this Email are subject >> to the terms and conditions expressed in any applicable governing The Home >> Depot terms of business or client engagement letter. The Home Depot >> disclaims all responsibility and liability for the accuracy and content of >> this attachment and for any damages or losses arising from any >> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other >> items of a destructive nature, which may be contained in this attachment >> and shall not be liable for direct, indirect, consequential or special >> damages in connection with this e-mail message or its attachment. >> >> >> ------------------------------ >> >> The information in this Internet Email is confidential and may be legally >> privileged. It is intended solely for the addressee. Access to this Email >> by anyone else is unauthorized. If you are not the intended recipient, any >> disclosure, copying, distribution or any action taken or omitted to be >> taken in reliance on it, is prohibited and may be unlawful. When addressed >> to our clients any opinions or advice contained in this Email are subject >> to the terms and conditions expressed in any applicable governing The Home >> Depot terms of business or client engagement letter. The Home Depot >> disclaims all responsibility and liability for the accuracy and content of >> this attachment and for any damages or losses arising from any >> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other >> items of a destructive nature, which may be contained in this attachment >> and shall not be liable for direct, indirect, consequential or special >> damages in connection with this e-mail message or its attachment. >> >
