RE: Changing num_tokens and migrating to 4.0
I have a cluster (almost 200 nodes) with a variety of disk sizes and use different numbers of tokens so that the machines can use the disk they have. It is a very handy feature! While I agree that a node with larger disk may handle more requests, that may not be enough to impact CPU or memory. I rarely see Cassandra CPU-bound for my use cases. These are primarily write use cases with a low number of clients with far fewer reads. There is just a lot of data to keep. Sean Durity From: Alex Ott Sent: Saturday, March 20, 2021 1:01 PM To: user Subject: [EXTERNAL] Re: Changing num_tokens and migrating to 4.0 if the nodes are almost the same, except the disk space, then giving them more may make siltation worse - they will get more requests than other nodes, and won't have resources to process them. In Cassandra the disk size isn't the main "success" factor - it's a memory, CPU, disk type (SSD), etc. On Sat, Mar 20, 2021 at 5:26 PM Lapo Luchini mailto:l...@lapo.it>> wrote: Hi, thanks for suggestions! I'll definitely migrate to 4.0 after all this is done, then. Old prod DC I fear can't suffer losing a node right now (a few nodes have the disk 70% full), but I can maybe find a third node for the new DC right away. BTW the new nodes have got 3× the disk space, but are not so much different regarding CPU and RAM: does it make any sense giving them a bit more num_tokens (maybe 20-30 instead of 16) than the rest of the old DC hosts or "asymmetrical" clusters lead to problems? No real need to do that anyways, moving from 6 nodes to (eventually) 8 should be enough lessen the load on the disks, and before more space is needed I will probably have more nodes. Lapo On 2021-03-20 16:23, Alex Ott wrote: > I personally maybe would go following way (need to calculate how many > joins/decommissions will be at the end): > > * Decommission one node from prod DC > * Form new DC from two new machines and decommissioned one. > * Rebuild DC from existing one, make sure that repair finished, etc. > * Switch traffic > * Remove old DC > * Add nodes from old DC one by one into new DC > - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org<mailto:user-unsubscr...@cassandra.apache.org> For additional commands, e-mail: user-h...@cassandra.apache.org<mailto:user-h...@cassandra.apache.org> -- With best wishes,Alex Ott http://alexott.net/ [alexott.net]<https://urldefense.com/v3/__http:/alexott.net/__;!!M-nmYVHPHQ!Z88elRlG30NKZR9PseTGVv5t4zPzVdEmbaqJdUVKjfSbuqPg3fU1FnbH1o6KTwF8XWTzS8A$> Twitter: alexott_en (English), alexott (Russian) The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.
Re: Changing num_tokens and migrating to 4.0
On 2021-03-22 01:27, Kane Wilson wrote: You should be able to get repairs working fine if you use a tool such as cassandra-reaper to manage it for you for such a small cluster. I would look into that before doing major cluster topology changes, as these can be complex and risky. I was looking into a migration as I'm already using Cassanrea Reaper and on the biggest keyspace is often taking more than 7 days to complete (I set the segment timeout up to 2h, and most of 138 segments take more than 1h, sometimes even failing the 2h limits due to, AFAICT, lengthy compactions). -- Lapo Luchini l...@lapo.it - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: Changing num_tokens and migrating to 4.0
You should be able to get repairs working fine if you use a tool such as cassandra-reaper to manage it for you for such a small cluster. I would look into that before doing major cluster topology changes, as these can be complex and risky. I definitely wouldn't go about it in the way you've described above, as you're likely to run into load issues. I also would advise against upgrading to the 4.0 beta releases in production. Leave running betas in production to the experts. There has not been enough real world testing to really have any kind of guarantee about the stability of 4.0 beta2. If cassandra-reaper doesn't work for you you'll want to migrate in one of two ways, both of which have their own caveats: 1. A complete DC migration, ideally with the same number of nodes to handle load. All new nodes would have the lower token count and should use the token allocation algorithm. Downside here is expensive but this is the least risk option as it can be managed independently of the existing DC. 2. Decommission and re-add one node at a time, utilising the token allocation algorithm when you re-add nodes. This is more risky as both decommission and addition will incur considerable load on the existing nodes, and you also may not have the spare storage capacity for it to work. You could solve this however by adding a new node first. Note I've left a lot of details out here, as these processes can be quite involved. You should test each method before you do it. raft.so - Cassandra consulting, support, and managed services On Sun, Mar 21, 2021 at 2:09 AM Lapo Luchini wrote: > I have a 6 nodes production cluster running 3.11.9 with the default > num_tokens=256… which is fine but I later discovered is a bit of a > hassle to do repairs and is probably better to lower that to 16. > > I'm adding two new nodes with much higher space storage and I was > wondering which migration strategy is better. > > If I got it correct I was thinking about this: > 1. add the 2 new nodes as a new "temporary DC", with num_token=16 RF=3 > 2. repair it all, then test it a bit > 3. switch production applications to "DC-temp" > 4. drop the old 6-node DC > 5. re-create it from scratch with num_token=16 RF=3 > 6. switch production applications to "main DC" again > 7. drop "DC-temp", eventually integrate nodes into "main DC" > > I'd also like to migrate from 3.11.9 to 4.0-beta2 (I'm running on > FreeBSD so those are the options), does it make sense to do it during > the mentioned "num_tokens migration" (at step 1, or 5) or does it make > more sense to do it at step 8, as a in-place rolling upgrade of each of > the 6 (or 8) nodes? > > Did I get it correctly? > Can it be done "better"? > > Thanks in advance for any suggestion or correction! > > -- > Lapo Luchini > l...@lapo.it > > > - > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > >
Re: Changing num_tokens and migrating to 4.0
if the nodes are almost the same, except the disk space, then giving them more may make siltation worse - they will get more requests than other nodes, and won't have resources to process them. In Cassandra the disk size isn't the main "success" factor - it's a memory, CPU, disk type (SSD), etc. On Sat, Mar 20, 2021 at 5:26 PM Lapo Luchini wrote: > Hi, thanks for suggestions! > I'll definitely migrate to 4.0 after all this is done, then. > > Old prod DC I fear can't suffer losing a node right now (a few nodes > have the disk 70% full), but I can maybe find a third node for the new > DC right away. > > BTW the new nodes have got 3× the disk space, but are not so much > different regarding CPU and RAM: does it make any sense giving them a > bit more num_tokens (maybe 20-30 instead of 16) than the rest of the old > DC hosts or "asymmetrical" clusters lead to problems? > > No real need to do that anyways, moving from 6 nodes to (eventually) 8 > should be enough lessen the load on the disks, and before more space is > needed I will probably have more nodes. > > Lapo > > On 2021-03-20 16:23, Alex Ott wrote: > > I personally maybe would go following way (need to calculate how many > > joins/decommissions will be at the end): > > > > * Decommission one node from prod DC > > * Form new DC from two new machines and decommissioned one. > > * Rebuild DC from existing one, make sure that repair finished, etc. > > * Switch traffic > > * Remove old DC > > * Add nodes from old DC one by one into new DC > > > > > > - > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > > -- With best wishes,Alex Ott http://alexott.net/ Twitter: alexott_en (English), alexott (Russian)
Re: Changing num_tokens and migrating to 4.0
Hi, thanks for suggestions! I'll definitely migrate to 4.0 after all this is done, then. Old prod DC I fear can't suffer losing a node right now (a few nodes have the disk 70% full), but I can maybe find a third node for the new DC right away. BTW the new nodes have got 3× the disk space, but are not so much different regarding CPU and RAM: does it make any sense giving them a bit more num_tokens (maybe 20-30 instead of 16) than the rest of the old DC hosts or "asymmetrical" clusters lead to problems? No real need to do that anyways, moving from 6 nodes to (eventually) 8 should be enough lessen the load on the disks, and before more space is needed I will probably have more nodes. Lapo On 2021-03-20 16:23, Alex Ott wrote: I personally maybe would go following way (need to calculate how many joins/decommissions will be at the end): * Decommission one node from prod DC * Form new DC from two new machines and decommissioned one. * Rebuild DC from existing one, make sure that repair finished, etc. * Switch traffic * Remove old DC * Add nodes from old DC one by one into new DC - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: Changing num_tokens and migrating to 4.0
The are several things to consider here: - You can't have DC of two nodes with RF=3... - Are you sure that new DC will handle all production traffic? - if new nodes much more powerful than other (memory/CPU/disk type) that could also cause unpredictable spikes when request will hit the "smaller" node. I personally maybe would go following way (need to calculate how many joins/decommissions will be at the end): - Decommission one node from prod DC - Form new DC from two new machines and decommissioned one. - Rebuild DC from existing one, make sure that repair finished, etc. - Switch traffic - Remove old DC - Add nodes from old DC one by one into new DC Upgrade to Cassandra 4.0 should be done either prior to that, or after - you shouldn't do it when doing bootstrapping/decomissioning... On Sat, Mar 20, 2021 at 4:09 PM Lapo Luchini wrote: > I have a 6 nodes production cluster running 3.11.9 with the default > num_tokens=256… which is fine but I later discovered is a bit of a > hassle to do repairs and is probably better to lower that to 16. > > I'm adding two new nodes with much higher space storage and I was > wondering which migration strategy is better. > > If I got it correct I was thinking about this: > 1. add the 2 new nodes as a new "temporary DC", with num_token=16 RF=3 > 2. repair it all, then test it a bit > 3. switch production applications to "DC-temp" > 4. drop the old 6-node DC > 5. re-create it from scratch with num_token=16 RF=3 > 6. switch production applications to "main DC" again > 7. drop "DC-temp", eventually integrate nodes into "main DC" > > I'd also like to migrate from 3.11.9 to 4.0-beta2 (I'm running on > FreeBSD so those are the options), does it make sense to do it during > the mentioned "num_tokens migration" (at step 1, or 5) or does it make > more sense to do it at step 8, as a in-place rolling upgrade of each of > the 6 (or 8) nodes? > > Did I get it correctly? > Can it be done "better"? > > Thanks in advance for any suggestion or correction! > > -- > Lapo Luchini > l...@lapo.it > > > - > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > > -- With best wishes,Alex Ott http://alexott.net/ Twitter: alexott_en (English), alexott (Russian)