These are good clarifications and expansions. Sean Durity
From: Anthony Grasso <anthony.gra...@gmail.com> Sent: Thursday, January 30, 2020 7:25 PM To: user <user@cassandra.apache.org> Subject: Re: [EXTERNAL] How to reduce vnodes without downtime Hi Maxim, Basically what Sean suggested is the way to do this without downtime. To clarify the, the three steps following the "Decommission each node in the DC you are working on" step should be applied to only the decommissioned nodes. So where it say "all nodes" or "every node" it applies to only the decommissioned nodes. In addition, the step that says "Wipe data on all the nodes", I would delete all files in the following directories on the decommissioned nodes. * data (usually located in /var/lib/cassandra/data) * commitlogs (usually located in /var/lib/cassandra/commitlogs) * hints (usually located in /var/lib/casandra/hints) * saved_caches (usually located in /var/lib/cassandra/saved_caches) Cheers, Anthony On Fri, 31 Jan 2020 at 03:05, Durity, Sean R <sean_r_dur...@homedepot.com<mailto:sean_r_dur...@homedepot.com>> wrote: Your procedure won’t work very well. On the first node, if you switched to 4, you would end up with only a tiny fraction of the data (because the other nodes would still be at 256). I updated a large cluster (over 150 nodes – 2 DCs) to smaller number of vnodes. The basic outline was this: * Stop all repairs * Make sure the app is running against one DC only * Change the replication settings on keyspaces to use only 1 DC (basically cutting off the other DC) * Decommission each node in the DC you are working on. Because the replication setting are changed, no streaming occurs. But it releases the token assignments * Wipe data on all the nodes * Update configuration on every node to your new settings, including auto_bootstrap = false * Start all nodes. They will choose tokens, but not stream any data * Update replication factor for all keyspaces to include the new DC * I disabled binary on those nodes to prevent app connections * Run nodetool reduild with -dc (other DC) on as many nodes as your system can safely handle until they are all rebuilt. * Re-enable binary (and app connections to the rebuilt DC) * Turn on repairs * Rest for a bit, then reverse the process for the remaining DCs Sean Durity – Staff Systems Engineer, Cassandra From: Maxim Parkachov <lazy.gop...@gmail.com<mailto:lazy.gop...@gmail.com>> Sent: Thursday, January 30, 2020 10:05 AM To: user@cassandra.apache.org<mailto:user@cassandra.apache.org> Subject: [EXTERNAL] How to reduce vnodes without downtime Hi everyone, with discussion about reducing default vnodes in version 4.0 I would like to ask, what would be optimal procedure to perform reduction of vnodes in existing 3.11.x cluster which was set up with default value 256. Cluster has 2 DC with 5 nodes each and RF=3. There is one more restriction, I could not add more servers, nor to create additional DC, everything is physical. This should be done without downtime. My idea for such procedure would be for each node: - decommission node - set auto_bootstrap to true and vnodes to 4 - start and wait till node joins cluster - run cleanup on rest of nodes in cluster - run repair on whole cluster (not sure if needed after cleanup) - set auto_bootstrap to false repeat for each node rolling restart of cluster cluster repair Is this sounds right ? My concern is that after decommission, node will start on the same IP which could create some confusion. Regards, Maxim. ________________________________ The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment. ________________________________ The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.
