Re: Changing num_tokens and migrating to 4.0
if the nodes are almost the same, except the disk space, then giving them more may make siltation worse - they will get more requests than other nodes, and won't have resources to process them. In Cassandra the disk size isn't the main "success" factor - it's a memory, CPU, disk type (SSD), etc. On Sat, Mar 20, 2021 at 5:26 PM Lapo Luchini wrote: > Hi, thanks for suggestions! > I'll definitely migrate to 4.0 after all this is done, then. > > Old prod DC I fear can't suffer losing a node right now (a few nodes > have the disk 70% full), but I can maybe find a third node for the new > DC right away. > > BTW the new nodes have got 3× the disk space, but are not so much > different regarding CPU and RAM: does it make any sense giving them a > bit more num_tokens (maybe 20-30 instead of 16) than the rest of the old > DC hosts or "asymmetrical" clusters lead to problems? > > No real need to do that anyways, moving from 6 nodes to (eventually) 8 > should be enough lessen the load on the disks, and before more space is > needed I will probably have more nodes. > > Lapo > > On 2021-03-20 16:23, Alex Ott wrote: > > I personally maybe would go following way (need to calculate how many > > joins/decommissions will be at the end): > > > > * Decommission one node from prod DC > > * Form new DC from two new machines and decommissioned one. > > * Rebuild DC from existing one, make sure that repair finished, etc. > > * Switch traffic > > * Remove old DC > > * Add nodes from old DC one by one into new DC > > > > > > - > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > > -- With best wishes,Alex Ott http://alexott.net/ Twitter: alexott_en (English), alexott (Russian)
Re: Changing num_tokens and migrating to 4.0
Hi, thanks for suggestions! I'll definitely migrate to 4.0 after all this is done, then. Old prod DC I fear can't suffer losing a node right now (a few nodes have the disk 70% full), but I can maybe find a third node for the new DC right away. BTW the new nodes have got 3× the disk space, but are not so much different regarding CPU and RAM: does it make any sense giving them a bit more num_tokens (maybe 20-30 instead of 16) than the rest of the old DC hosts or "asymmetrical" clusters lead to problems? No real need to do that anyways, moving from 6 nodes to (eventually) 8 should be enough lessen the load on the disks, and before more space is needed I will probably have more nodes. Lapo On 2021-03-20 16:23, Alex Ott wrote: I personally maybe would go following way (need to calculate how many joins/decommissions will be at the end): * Decommission one node from prod DC * Form new DC from two new machines and decommissioned one. * Rebuild DC from existing one, make sure that repair finished, etc. * Switch traffic * Remove old DC * Add nodes from old DC one by one into new DC - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: Changing num_tokens and migrating to 4.0
The are several things to consider here: - You can't have DC of two nodes with RF=3... - Are you sure that new DC will handle all production traffic? - if new nodes much more powerful than other (memory/CPU/disk type) that could also cause unpredictable spikes when request will hit the "smaller" node. I personally maybe would go following way (need to calculate how many joins/decommissions will be at the end): - Decommission one node from prod DC - Form new DC from two new machines and decommissioned one. - Rebuild DC from existing one, make sure that repair finished, etc. - Switch traffic - Remove old DC - Add nodes from old DC one by one into new DC Upgrade to Cassandra 4.0 should be done either prior to that, or after - you shouldn't do it when doing bootstrapping/decomissioning... On Sat, Mar 20, 2021 at 4:09 PM Lapo Luchini wrote: > I have a 6 nodes production cluster running 3.11.9 with the default > num_tokens=256… which is fine but I later discovered is a bit of a > hassle to do repairs and is probably better to lower that to 16. > > I'm adding two new nodes with much higher space storage and I was > wondering which migration strategy is better. > > If I got it correct I was thinking about this: > 1. add the 2 new nodes as a new "temporary DC", with num_token=16 RF=3 > 2. repair it all, then test it a bit > 3. switch production applications to "DC-temp" > 4. drop the old 6-node DC > 5. re-create it from scratch with num_token=16 RF=3 > 6. switch production applications to "main DC" again > 7. drop "DC-temp", eventually integrate nodes into "main DC" > > I'd also like to migrate from 3.11.9 to 4.0-beta2 (I'm running on > FreeBSD so those are the options), does it make sense to do it during > the mentioned "num_tokens migration" (at step 1, or 5) or does it make > more sense to do it at step 8, as a in-place rolling upgrade of each of > the 6 (or 8) nodes? > > Did I get it correctly? > Can it be done "better"? > > Thanks in advance for any suggestion or correction! > > -- > Lapo Luchini > l...@lapo.it > > > - > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > > -- With best wishes,Alex Ott http://alexott.net/ Twitter: alexott_en (English), alexott (Russian)
Changing num_tokens and migrating to 4.0
I have a 6 nodes production cluster running 3.11.9 with the default num_tokens=256… which is fine but I later discovered is a bit of a hassle to do repairs and is probably better to lower that to 16. I'm adding two new nodes with much higher space storage and I was wondering which migration strategy is better. If I got it correct I was thinking about this: 1. add the 2 new nodes as a new "temporary DC", with num_token=16 RF=3 2. repair it all, then test it a bit 3. switch production applications to "DC-temp" 4. drop the old 6-node DC 5. re-create it from scratch with num_token=16 RF=3 6. switch production applications to "main DC" again 7. drop "DC-temp", eventually integrate nodes into "main DC" I'd also like to migrate from 3.11.9 to 4.0-beta2 (I'm running on FreeBSD so those are the options), does it make sense to do it during the mentioned "num_tokens migration" (at step 1, or 5) or does it make more sense to do it at step 8, as a in-place rolling upgrade of each of the 6 (or 8) nodes? Did I get it correctly? Can it be done "better"? Thanks in advance for any suggestion or correction! -- Lapo Luchini l...@lapo.it - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
