Ok. Have to psych myself up to the add node task a bit. Didn't go well the
first time round!
Tasks
- Make sure the new node is not in seeds list!
- Check cluster name, listen address, rpc address
- Give it its own rack in cassandra-rackdc.properties
- Delete cassandra-topology.properties if it
Because executing “removenode” streamed extra data from live nodes to the “gaining” replicaOversimplified (if you had one token per node) If you start with A B CThen add DD should bootstrap a range from each of A B and C, but at the end, some of the data that was A B C becomes B C DWhen you
Looks like the remove has sorted things out. Thanks.
One thing I am wondering about is why the nodes are carrying a lot more
data? The loads were about 2.7T before, now 3.4T.
# nodetool status
Datacenter: dc1
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address
Usually, it’s a good practice to resemble the real datacenter in the Cassandra
topology, thus nodes mounted to distinct racks are know with different rack
names to Cassandra. This is due to the usual datacenter infrastructure, having
a single point of failure in each rack - e.g. a network
That's correct. nodetool removenode is strongly preferred when your node
is already down. If the node is still functional, use nodetool
decommission on the node instead.
On 03/04/2023 16:32, Jeff Jirsa wrote:
FWIW, `nodetool decommission` is strongly preferred. `nodetool
removenode` is
I just want to mention that the "rack" in Cassandra don't need to match
the physical rack. As long as each "rack" in Cassandra fails independent
of each other, it is fine.
That means if you have 6 physical servers each in an unique physical
rack and Cassandra RF=3, you can have any of the
FWIW, `nodetool decommission` is strongly preferred. `nodetool removenode`
is designed to be run when a host is offline. Only decommission is
guaranteed to maintain consistency / correctness, and removemode probably
streams a lot more data around than decommission.
On Mon, Apr 3, 2023 at 6:47 AM
>
> I just asked that question on this list and the answer was that adding the
> new nodes as rack4, rack5 and rack6 is fine. They are all on
> separate physical racks. Is that ok?
>
Yes, Jeff is right, all 6 nodes each on their own rack will work just fine.
Should I do a full repair first or is
Thanks. Yes my big screwup here was to make the new node a seed node so it
didn't get any data. I am going to add 3 more nodes, one at a time when the
cluster has finished with the remove and everything seems stable. Should I
do a full repair first or is the remove node operation basically doing
The time it takes to stream data off of a node varies by network, cloud
region, and other factors. So it's not unheard of for it to take a bit to
finish.
Just thought I'd mention that auto_bootstrap is true by default. So if
you're not setting it, the node should bootstrap as long as it's not a
As long as the number of racks is already at/above the number of nodes /
replication factor, it's gonna be fine.
Where it tends to surprise people is if you have RF=3 and either 1 or 2
racks, and then you add a third, that third rack gets one copy of "all" of
the data, so you often run out of
I have a 3 node cluster using the GossipingPropertyFileSnitch and
replication factor of 3. All nodes are leased hardware and more or less the
same. The cassandra-rackdc.properties files look like this:
dc=dc1
rack=rack1
(rack2 and rack3 for the other nodes)
Now I need to expand the cluster. I
Thanks. Hmm, the remove has been busy for hours but seems to be progressing.
I have been running this on the nodes to monitor progress:
# nodetool netstats | grep Already
Receiving 92 files, 843934103369 bytes total. Already received 82
files (89.13%), 590204687299 bytes total (69.93%)
Use nodetool removenode is strongly preferred in most circumstances, and
only resort to assassinate if you do not care about data consistency or
you know there won't be any consistency issue (e.g. no new writes and
did not run nodetool cleanup).
Since the size of data on the new node is
14 matches
Mail list logo
