subject:"Re\: Reads not returning data after adding node"

Ok. Have to psych myself up to the add node task a bit. Didn't go well the
first time round!

Tasks
- Make sure the new node is not in seeds list!
- Check cluster name, listen address, rpc address
- Give it its own rack in cassandra-rackdc.properties
- Delete cassandra-topology.properties if it exists
- Make sure no compactions are on the go
- rm -rf /var/lib/cassandra/*
- rm /data/cassandra/commitlog/* (this is on different disk)
- systemctl start cassandra

And it should start streaming data from the other nodes and join the
cluster. Anything else I have to watch out for? Tx.


On Tue, Apr 4, 2023 at 5:25 AM Jeff Jirsa  wrote:

> Because executing “removenode” streamed extra data from live nodes to the
> “gaining” replica
>
> Oversimplified (if you had one token per node)
>
> If you  start with A B C
>
> Then add D
>
> D should bootstrap a range from each of A B and C, but at the end, some of
> the data that was A B C becomes B C D
>
> When you removenode, you tell B and C to send data back to A.
>
> A B and C will eventually contact that data away. Eventually.
>
> If you get around to adding D again, running “cleanup” when you’re done
> (successfully) will remove a lot of it.
>
>
>
> On Apr 3, 2023, at 8:14 PM, David Tinker  wrote:
>
> 
> Looks like the remove has sorted things out. Thanks.
>
> One thing I am wondering about is why the nodes are carrying a lot more
> data? The loads were about 2.7T before, now 3.4T.
>
> # nodetool status
> Datacenter: dc1
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address  Load  Tokens  Owns (effective)  Host ID
> Rack
> UN  xxx.xxx.xxx.105  3.4 TiB   256 100.0%
>  afd02287-3f88-4c6f-8b27-06f7a8192402  rack3
> UN  xxx.xxx.xxx.253  3.34 TiB  256 100.0%
>  e1af72be-e5df-4c6b-a124-c7bc48c6602a  rack2
> UN  xxx.xxx.xxx.107  3.44 TiB  256 100.0%
>  ab72f017-be96-41d2-9bef-a551dec2c7b5  rack1
>
> On Mon, Apr 3, 2023 at 5:42 PM Bowen Song via user <
> user@cassandra.apache.org> wrote:
>
>> That's correct. nodetool removenode is strongly preferred when your node
>> is already down. If the node is still functional, use nodetool
>> decommission on the node instead.
>> On 03/04/2023 16:32, Jeff Jirsa wrote:
>>
>> FWIW, `nodetool decommission` is strongly preferred. `nodetool
>> removenode` is designed to be run when a host is offline. Only decommission
>> is guaranteed to maintain consistency / correctness, and removemode
>> probably streams a lot more data around than decommission.
>>
>>
>> On Mon, Apr 3, 2023 at 6:47 AM Bowen Song via user <
>> user@cassandra.apache.org> wrote:
>>
>>> Use nodetool removenode is strongly preferred in most circumstances,
>>> and only resort to assassinate if you do not care about data
>>> consistency or you know there won't be any consistency issue (e.g. no new
>>> writes and did not run nodetool cleanup).
>>>
>>> Since the size of data on the new node is small, nodetool removenode
>>> should finish fairly quickly and bring your cluster back.
>>>
>>> Next time when you are doing something like this again, please test it
>>> out on a non-production environment, make sure everything works as expected
>>> before moving onto the production.
>>>
>>>
>>> On 03/04/2023 06:28, David Tinker wrote:
>>>
>>> Should I use assassinate or removenode? Given that there is some data on
>>> the node. Or will that be found on the other nodes? Sorry for all the
>>> questions but I really don't want to mess up.
>>>
>>> On Mon, Apr 3, 2023 at 7:21 AM Carlos Diaz  wrote:
>>>
 That's what nodetool assassinte will do.

 On Sun, Apr 2, 2023 at 10:19 PM David Tinker 
 wrote:

> Is it possible for me to remove the node from the cluster i.e. to undo
> this mess and get the cluster operating again?
>
> On Mon, Apr 3, 2023 at 7:13 AM Carlos Diaz 
> wrote:
>
>> You can leave it in the seed list of the other nodes, just make sure
>> it's not included in this node's seed list.  However, if you do decide to
>> fix the issue with the racks first assassinate this node (nodetool
>> assassinate ), and update the rack name before you restart.
>>
>> On Sun, Apr 2, 2023 at 10:06 PM David Tinker 
>> wrote:
>>
>>> It is also in the seeds list for the other nodes. Should I remove it
>>> from those, restart them one at a time, then restart it?
>>>
>>> /etc/cassandra # grep -i bootstrap *
>>> doesn't show anything so I don't think I have auto_bootstrap false.
>>>
>>> Thanks very much for the help.
>>>
>>>
>>> On Mon, Apr 3, 2023 at 7:01 AM Carlos Diaz 
>>> wrote:
>>>
 Just remove it from the seed list in the cassandra.yaml file and
 restart the node.  Make sure that auto_bootstrap is set to true first
 though.

 On Sun, Apr 2, 2023 at 9:59 PM David Tinker 
 wrote:

> So likely because I made it a seed node when I added it to

Re: Reads not returning data after adding node

2023-04-03 Thread Jeff Jirsa

Because executing “removenode” streamed extra data from live nodes to the “gaining” replicaOversimplified (if you had one token per node) If you start with A B CThen add DD should bootstrap a range from each of A B and C, but at the end, some of the data that was A B C becomes B C DWhen you removenode, you tell B and C to send data back to A. A B and C will eventually contact that data away. Eventually. If you get around to adding D again, running “cleanup” when you’re done (successfully) will remove a lot of it. On Apr 3, 2023, at 8:14 PM, David Tinker wrote:Looks like the remove has sorted things out. Thanks.One thing I am wondering about is why the nodes are carrying a lot more data? The loads were about 2.7T before, now 3.4T. # nodetool statusDatacenter: dc1===Status=Up/Down|/ State=Normal/Leaving/Joining/Moving-- Address Load Tokens Owns (effective) Host ID RackUN xxx.xxx.xxx.105 3.4 TiB 256 100.0% afd02287-3f88-4c6f-8b27-06f7a8192402 rack3UN xxx.xxx.xxx.253 3.34 TiB 256 100.0% e1af72be-e5df-4c6b-a124-c7bc48c6602a rack2UN xxx.xxx.xxx.107 3.44 TiB 256 100.0% ab72f017-be96-41d2-9bef-a551dec2c7b5 rack1On Mon, Apr 3, 2023 at 5:42 PM Bowen Song via user wrote:

That's correct. nodetool removenode
is strongly preferred when your node is already down. If the node
is still functional, use nodetool
decommission on the node instead.

On 03/04/2023 16:32, Jeff Jirsa wrote:

FWIW, `nodetool decommission` is strongly
preferred. `nodetool removenode` is designed to be run when a
host is offline. Only decommission is guaranteed to maintain
consistency / correctness, and removemode probably streams a
lot more data around than decommission.

On Mon, Apr 3, 2023 at
6:47 AM Bowen Song via user
wrote:

Use nodetool removenode
is strongly preferred in most circumstances, and only
resort to assassinate if
you do not care about data consistency or you know there
won't be any consistency issue (e.g. no new writes and
did not run nodetool cleanup).
Since the size of data on the new node is small, nodetool removenode should
finish fairly quickly and bring your cluster back.
Next time when you are doing something like this again,
please test it out on a non-production environment, make
sure everything works as expected before moving onto the
production.

On 03/04/2023 06:28, David Tinker wrote:

Should I use assassinate or removenode?
Given that there is some data on the node. Or will
that be found on the other nodes? Sorry for all the
questions but I really don't want to mess up.

On Mon, Apr 3, 2023
at 7:21 AM Carlos Diaz
wrote:

That's what nodetool assassinte will
do.

On Sun, Apr 2,
2023 at 10:19 PM David Tinker
wrote:

Is it possible for me to remove
the node from the cluster i.e. to undo this
mess and get the cluster operating again?

On Mon, Apr
3, 2023 at 7:13 AM Carlos Diaz
wrote:

You can leave it in the seed
list of the other nodes, just make sure
it's not included in this node's seed
list. However, if you do decide to fix
the issue with the racks first assassinate
this node (nodetool assassinate
), and update the rack name
before you restart.

On Sun,
Apr 2, 2023 at 10:06 PM David Tinker

Re: Reads not returning data after adding node

Looks like the remove has sorted things out. Thanks.

One thing I am wondering about is why the nodes are carrying a lot more
data? The loads were about 2.7T before, now 3.4T.

# nodetool status
Datacenter: dc1
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address  Load  Tokens  Owns (effective)  Host ID
Rack
UN  xxx.xxx.xxx.105  3.4 TiB   256 100.0%
 afd02287-3f88-4c6f-8b27-06f7a8192402  rack3
UN  xxx.xxx.xxx.253  3.34 TiB  256 100.0%
 e1af72be-e5df-4c6b-a124-c7bc48c6602a  rack2
UN  xxx.xxx.xxx.107  3.44 TiB  256 100.0%
 ab72f017-be96-41d2-9bef-a551dec2c7b5  rack1

On Mon, Apr 3, 2023 at 5:42 PM Bowen Song via user <
user@cassandra.apache.org> wrote:

> That's correct. nodetool removenode is strongly preferred when your node
> is already down. If the node is still functional, use nodetool
> decommission on the node instead.
> On 03/04/2023 16:32, Jeff Jirsa wrote:
>
> FWIW, `nodetool decommission` is strongly preferred. `nodetool removenode`
> is designed to be run when a host is offline. Only decommission is
> guaranteed to maintain consistency / correctness, and removemode probably
> streams a lot more data around than decommission.
>
>
> On Mon, Apr 3, 2023 at 6:47 AM Bowen Song via user <
> user@cassandra.apache.org> wrote:
>
>> Use nodetool removenode is strongly preferred in most circumstances, and
>> only resort to assassinate if you do not care about data consistency or
>> you know there won't be any consistency issue (e.g. no new writes and did
>> not run nodetool cleanup).
>>
>> Since the size of data on the new node is small, nodetool removenode
>> should finish fairly quickly and bring your cluster back.
>>
>> Next time when you are doing something like this again, please test it
>> out on a non-production environment, make sure everything works as expected
>> before moving onto the production.
>>
>>
>> On 03/04/2023 06:28, David Tinker wrote:
>>
>> Should I use assassinate or removenode? Given that there is some data on
>> the node. Or will that be found on the other nodes? Sorry for all the
>> questions but I really don't want to mess up.
>>
>> On Mon, Apr 3, 2023 at 7:21 AM Carlos Diaz  wrote:
>>
>>> That's what nodetool assassinte will do.
>>>
>>> On Sun, Apr 2, 2023 at 10:19 PM David Tinker 
>>> wrote:
>>>
 Is it possible for me to remove the node from the cluster i.e. to undo
 this mess and get the cluster operating again?

 On Mon, Apr 3, 2023 at 7:13 AM Carlos Diaz  wrote:

> You can leave it in the seed list of the other nodes, just make sure
> it's not included in this node's seed list.  However, if you do decide to
> fix the issue with the racks first assassinate this node (nodetool
> assassinate ), and update the rack name before you restart.
>
> On Sun, Apr 2, 2023 at 10:06 PM David Tinker 
> wrote:
>
>> It is also in the seeds list for the other nodes. Should I remove it
>> from those, restart them one at a time, then restart it?
>>
>> /etc/cassandra # grep -i bootstrap *
>> doesn't show anything so I don't think I have auto_bootstrap false.
>>
>> Thanks very much for the help.
>>
>>
>> On Mon, Apr 3, 2023 at 7:01 AM Carlos Diaz 
>> wrote:
>>
>>> Just remove it from the seed list in the cassandra.yaml file and
>>> restart the node.  Make sure that auto_bootstrap is set to true first
>>> though.
>>>
>>> On Sun, Apr 2, 2023 at 9:59 PM David Tinker 
>>> wrote:
>>>
 So likely because I made it a seed node when I added it to the
 cluster it didn't do the bootstrap process. How can I recover this?

 On Mon, Apr 3, 2023 at 6:41 AM David Tinker 
 wrote:

> Yes replication factor is 3.
>
> I ran nodetool repair -pr on all the nodes (one at a time) and am
> still having issues getting data back from queries.
>
> I did make the new node a seed node.
>
> Re "rack4": I assumed that was just an indication as to the
> physical location of the server for redundancy. This one is separate 
> from
> the others so I used rack4.
>
> On Mon, Apr 3, 2023 at 6:30 AM Carlos Diaz 
> wrote:
>
>> I'm assuming that your replication factor is 3.  If that's the
>> case, did you intentionally put this node in rack 4?  Typically, you 
>> want
>> to add nodes in multiples of your replication factor in order to 
>> keep the
>> "racks" balanced.  In other words, this node should have been added 
>> to rack
>> 1, 2 or 3.
>>
>> Having said that, you should be able to easily fix your problem
>> by running a nodetool repair -pr on the new node.
>>
>> On Sun, Apr 2, 2023 at 8:16 PM David Tinker <
>> david.tin...@gmail.com> wrote:
>>

Re: Reads not returning data after adding node

2023-04-03 Thread Bowen Song via user

That's correct. nodetool removenode is strongly preferred when your node 
is already down. If the node is still functional, use nodetool 
decommission on the node instead.


On 03/04/2023 16:32, Jeff Jirsa wrote:
FWIW, `nodetool decommission` is strongly preferred. `nodetool 
removenode` is designed to be run when a host is offline. Only 
decommission is guaranteed to maintain consistency / correctness, and 
removemode probably streams a lot more data around than decommission.



On Mon, Apr 3, 2023 at 6:47 AM Bowen Song via user 
 wrote:


Use nodetool removenode is strongly preferred in most
circumstances, and only resort to assassinate if you do not care
about data consistency or you know there won't be any consistency
issue (e.g. no new writes and did not run nodetool cleanup).

Since the size of data on the new node is small, nodetool
removenode should finish fairly quickly and bring your cluster back.

Next time when you are doing something like this again, please
test it out on a non-production environment, make sure everything
works as expected before moving onto the production.


On 03/04/2023 06:28, David Tinker wrote:

Should I use assassinate or removenode? Given that there is some
data on the node. Or will that be found on the other nodes? Sorry
for all the questions but I really don't want to mess up.

On Mon, Apr 3, 2023 at 7:21 AM Carlos Diaz 
wrote:

That's what nodetool assassinte will do.

On Sun, Apr 2, 2023 at 10:19 PM David Tinker
 wrote:

Is it possible for me to remove the node from the cluster
i.e. to undo this mess and get the cluster operating again?

On Mon, Apr 3, 2023 at 7:13 AM Carlos Diaz
 wrote:

You can leave it in the seed list of the other nodes,
just make sure it's not included in this node's seed
list.  However, if you do decide to fix the issue
with the racks first assassinate this node (nodetool
assassinate ), and update the rack name before
you restart.

On Sun, Apr 2, 2023 at 10:06 PM David Tinker
 wrote:

It is also in the seeds list for the other nodes.
Should I remove it from those, restart them one
at a time, then restart it?

/etc/cassandra # grep -i bootstrap *
doesn't show anything so I don't think I have
auto_bootstrap false.

Thanks very much for the help.


On Mon, Apr 3, 2023 at 7:01 AM Carlos Diaz
 wrote:

Just remove it from the seed list in the
cassandra.yaml file and restart the node. 
Make sure that auto_bootstrap is set to true
first though.

On Sun, Apr 2, 2023 at 9:59 PM David Tinker
 wrote:

So likely because I made it a seed node
when I added it to the cluster it didn't
do the bootstrap process. How can I
recover this?

On Mon, Apr 3, 2023 at 6:41 AM David
Tinker  wrote:

Yes replication factor is 3.

I ran nodetool repair -pr on all the
nodes (one at a time) and am still
having issues getting data back from
queries.

I did make the new node a seed node.

Re "rack4": I assumed that was just
an indication as to the physical
location of the server for
redundancy. This one is separate from
the others so I used rack4.

On Mon, Apr 3, 2023 at 6:30 AM Carlos
Diaz  wrote:

I'm assuming that your
replication factor is 3.  If
that's the case, did you
intentionally put this node in
rack 4?  Typically, you want to
add nodes in multiples of your
replication factor in order to
keep the "racks" balanced.  In
other words, this node should
have been added to rack 1, 2 or 3.

Having said that, you should be

Re: Reads not returning data after adding node

2023-04-03 Thread Jeff Jirsa

FWIW, `nodetool decommission` is strongly preferred. `nodetool removenode`
is designed to be run when a host is offline. Only decommission is
guaranteed to maintain consistency / correctness, and removemode probably
streams a lot more data around than decommission.


On Mon, Apr 3, 2023 at 6:47 AM Bowen Song via user <
user@cassandra.apache.org> wrote:

> Use nodetool removenode is strongly preferred in most circumstances, and
> only resort to assassinate if you do not care about data consistency or
> you know there won't be any consistency issue (e.g. no new writes and did
> not run nodetool cleanup).
>
> Since the size of data on the new node is small, nodetool removenode
> should finish fairly quickly and bring your cluster back.
>
> Next time when you are doing something like this again, please test it out
> on a non-production environment, make sure everything works as expected
> before moving onto the production.
>
>
> On 03/04/2023 06:28, David Tinker wrote:
>
> Should I use assassinate or removenode? Given that there is some data on
> the node. Or will that be found on the other nodes? Sorry for all the
> questions but I really don't want to mess up.
>
> On Mon, Apr 3, 2023 at 7:21 AM Carlos Diaz  wrote:
>
>> That's what nodetool assassinte will do.
>>
>> On Sun, Apr 2, 2023 at 10:19 PM David Tinker 
>> wrote:
>>
>>> Is it possible for me to remove the node from the cluster i.e. to undo
>>> this mess and get the cluster operating again?
>>>
>>> On Mon, Apr 3, 2023 at 7:13 AM Carlos Diaz  wrote:
>>>
 You can leave it in the seed list of the other nodes, just make sure
 it's not included in this node's seed list.  However, if you do decide to
 fix the issue with the racks first assassinate this node (nodetool
 assassinate ), and update the rack name before you restart.

 On Sun, Apr 2, 2023 at 10:06 PM David Tinker 
 wrote:

> It is also in the seeds list for the other nodes. Should I remove it
> from those, restart them one at a time, then restart it?
>
> /etc/cassandra # grep -i bootstrap *
> doesn't show anything so I don't think I have auto_bootstrap false.
>
> Thanks very much for the help.
>
>
> On Mon, Apr 3, 2023 at 7:01 AM Carlos Diaz 
> wrote:
>
>> Just remove it from the seed list in the cassandra.yaml file and
>> restart the node.  Make sure that auto_bootstrap is set to true first
>> though.
>>
>> On Sun, Apr 2, 2023 at 9:59 PM David Tinker 
>> wrote:
>>
>>> So likely because I made it a seed node when I added it to the
>>> cluster it didn't do the bootstrap process. How can I recover this?
>>>
>>> On Mon, Apr 3, 2023 at 6:41 AM David Tinker 
>>> wrote:
>>>
 Yes replication factor is 3.

 I ran nodetool repair -pr on all the nodes (one at a time) and am
 still having issues getting data back from queries.

 I did make the new node a seed node.

 Re "rack4": I assumed that was just an indication as to the
 physical location of the server for redundancy. This one is separate 
 from
 the others so I used rack4.

 On Mon, Apr 3, 2023 at 6:30 AM Carlos Diaz 
 wrote:

> I'm assuming that your replication factor is 3.  If that's the
> case, did you intentionally put this node in rack 4?  Typically, you 
> want
> to add nodes in multiples of your replication factor in order to keep 
> the
> "racks" balanced.  In other words, this node should have been added 
> to rack
> 1, 2 or 3.
>
> Having said that, you should be able to easily fix your problem by
> running a nodetool repair -pr on the new node.
>
> On Sun, Apr 2, 2023 at 8:16 PM David Tinker <
> david.tin...@gmail.com> wrote:
>
>> Hi All
>>
>> I recently added a node to my 3 node Cassandra 4.0.5 cluster and
>> now many reads are not returning rows! What do I need to do to fix 
>> this?
>> There weren't any errors in the logs or other problems that I could 
>> see. I
>> expected the cluster to balance itself but this hasn't happened 
>> (yet?). The
>> nodes are similar so I have num_tokens=256 for each. I am using the
>> Murmur3Partitioner.
>>
>> # nodetool status
>> Datacenter: dc1
>> ===
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> --  Address  Load   Tokens  Owns (effective)  Host ID
>>   Rack
>> UN  xxx.xxx.xxx.105  2.65 TiB   256 72.9%
>> afd02287-3f88-4c6f-8b27-06f7a8192402  rack3
>> UN  xxx.xxx.xxx.253  2.6 TiB256 73.9%
>> e1af72be-e5df-4c6b-a124-c7bc48c6602a  rack2
>> UN  xxx.xxx.xxx.24   93.82 KiB

Re: Reads not returning data after adding node

2023-04-03 Thread Aaron Ploetz

>
> I just asked that question on this list and the answer was that adding the
> new nodes as rack4, rack5 and rack6 is fine. They are all on
> separate physical racks. Is that ok?
>

Yes, Jeff is right, all 6 nodes each on their own rack will work just fine.

Should I do a full repair first or is the remove node operation basically
> doing that?
>

I don't think you'll need a full repair.  Removenode should be taking care
of streaming that node's data to where it needs to go.

On Mon, Apr 3, 2023 at 10:26 AM David Tinker  wrote:

> Thanks. Yes my big screwup here was to make the new node a seed node so it
> didn't get any data. I am going to add 3 more nodes, one at a time when the
> cluster has finished with the remove and everything seems stable. Should I
> do a full repair first or is the remove node operation basically doing that?
>
> Re the racks. I just asked that question on this list and the answer was
> that adding the new nodes as rack4, rack5 and rack6 is fine. They are all
> on separate physical racks. Is that ok?
>
> On Mon, Apr 3, 2023 at 5:16 PM Aaron Ploetz  wrote:
>
>> The time it takes to stream data off of a node varies by network, cloud
>> region, and other factors.  So it's not unheard of for it to take a bit to
>> finish.
>>
>> Just thought I'd mention that auto_bootstrap is true by default.  So if
>> you're not setting it, the node should bootstrap as long as it's not a seed
>> node.
>>
>> As for the rack issue, yes, it's a good idea to keep your racks in
>> multiples of your RF.  When performing token ownership calculations,
>> Cassandra takes rack designation into consideration.  It tries to ensure
>> that multiple replicas for a row are not placed in the same rack.  TBH -
>> I'd build out two more nodes to have 6 nodes across 3 racks (2 in each),
>> just to ensure even distribution.  Otherwise, you might notice that the
>> nodes sharing a rack will consume disk at a different rate than the nodes
>> which have their own rack.
>>
>> On Mon, Apr 3, 2023 at 8:57 AM David Tinker 
>> wrote:
>>
>>> Thanks. Hmm, the remove has been busy for hours but seems to be
>>> progressing.
>>>
>>> I have been running this on the nodes to monitor progress:
>>> # nodetool netstats | grep Already
>>> Receiving 92 files, 843934103369 bytes total. Already received
>>> 82 files (89.13%), 590204687299 bytes total (69.93%)
>>> Sending 84 files, 860198753783 bytes total. Already sent 56
>>> files (66.67%), 307038785732 bytes total (35.69%)
>>> Sending 78 files, 815573435637 bytes total. Already sent 56
>>> files (71.79%), 313079823738 bytes total (38.39%)
>>>
>>> The percentages are ticking up.
>>>
>>> # nodetool ring | head -20
>>> Datacenter: dc1
>>> ==
>>> Address   RackStatus State   LoadOwns
>>>  Token
>>>
>>>  9189523899826545641
>>> xxx.xxx.xxx..24rack4   Down   Leaving 26.62 GiB   79.95%
>>>  -9194674091837769168
>>> xxx.xxx.xxx.107   rack1   Up Normal  2.68 TiB73.25%
>>>  -9168781258594813088
>>> xxx.xxx.xxx.253   rack2   Up Normal  2.63 TiB73.92%
>>>  -9163037340977721917
>>> xxx.xxx.xxx.105   rack3   Up Normal  2.68 TiB72.88%
>>>  -9148860739730046229
>>>
>>>
>>> On Mon, Apr 3, 2023 at 3:46 PM Bowen Song via user <
>>> user@cassandra.apache.org> wrote:
>>>
 Use nodetool removenode is strongly preferred in most circumstances,
 and only resort to assassinate if you do not care about data
 consistency or you know there won't be any consistency issue (e.g. no new
 writes and did not run nodetool cleanup).

 Since the size of data on the new node is small, nodetool removenode
 should finish fairly quickly and bring your cluster back.

 Next time when you are doing something like this again, please test it
 out on a non-production environment, make sure everything works as expected
 before moving onto the production.


 On 03/04/2023 06:28, David Tinker wrote:

 Should I use assassinate or removenode? Given that there is some data
 on the node. Or will that be found on the other nodes? Sorry for all the
 questions but I really don't want to mess up.

 On Mon, Apr 3, 2023 at 7:21 AM Carlos Diaz  wrote:

> That's what nodetool assassinte will do.
>
> On Sun, Apr 2, 2023 at 10:19 PM David Tinker 
> wrote:
>
>> Is it possible for me to remove the node from the cluster i.e. to
>> undo this mess and get the cluster operating again?
>>
>> On Mon, Apr 3, 2023 at 7:13 AM Carlos Diaz 
>> wrote:
>>
>>> You can leave it in the seed list of the other nodes, just make sure
>>> it's not included in this node's seed list.  However, if you do decide 
>>> to
>>> fix the issue with the racks first assassinate this node (nodetool
>>> assassinate ), and update the rack

Re: Reads not returning data after adding node

Thanks. Yes my big screwup here was to make the new node a seed node so it
didn't get any data. I am going to add 3 more nodes, one at a time when the
cluster has finished with the remove and everything seems stable. Should I
do a full repair first or is the remove node operation basically doing that?

Re the racks. I just asked that question on this list and the answer was
that adding the new nodes as rack4, rack5 and rack6 is fine. They are all
on separate physical racks. Is that ok?

On Mon, Apr 3, 2023 at 5:16 PM Aaron Ploetz  wrote:

> The time it takes to stream data off of a node varies by network, cloud
> region, and other factors.  So it's not unheard of for it to take a bit to
> finish.
>
> Just thought I'd mention that auto_bootstrap is true by default.  So if
> you're not setting it, the node should bootstrap as long as it's not a seed
> node.
>
> As for the rack issue, yes, it's a good idea to keep your racks in
> multiples of your RF.  When performing token ownership calculations,
> Cassandra takes rack designation into consideration.  It tries to ensure
> that multiple replicas for a row are not placed in the same rack.  TBH -
> I'd build out two more nodes to have 6 nodes across 3 racks (2 in each),
> just to ensure even distribution.  Otherwise, you might notice that the
> nodes sharing a rack will consume disk at a different rate than the nodes
> which have their own rack.
>
> On Mon, Apr 3, 2023 at 8:57 AM David Tinker 
> wrote:
>
>> Thanks. Hmm, the remove has been busy for hours but seems to be
>> progressing.
>>
>> I have been running this on the nodes to monitor progress:
>> # nodetool netstats | grep Already
>> Receiving 92 files, 843934103369 bytes total. Already received 82
>> files (89.13%), 590204687299 bytes total (69.93%)
>> Sending 84 files, 860198753783 bytes total. Already sent 56 files
>> (66.67%), 307038785732 bytes total (35.69%)
>> Sending 78 files, 815573435637 bytes total. Already sent 56 files
>> (71.79%), 313079823738 bytes total (38.39%)
>>
>> The percentages are ticking up.
>>
>> # nodetool ring | head -20
>> Datacenter: dc1
>> ==
>> Address   RackStatus State   LoadOwns
>>Token
>>
>>9189523899826545641
>> xxx.xxx.xxx..24rack4   Down   Leaving 26.62 GiB   79.95%
>>  -9194674091837769168
>> xxx.xxx.xxx.107   rack1   Up Normal  2.68 TiB73.25%
>>-9168781258594813088
>> xxx.xxx.xxx.253   rack2   Up Normal  2.63 TiB73.92%
>>-9163037340977721917
>> xxx.xxx.xxx.105   rack3   Up Normal  2.68 TiB72.88%
>>-9148860739730046229
>>
>>
>> On Mon, Apr 3, 2023 at 3:46 PM Bowen Song via user <
>> user@cassandra.apache.org> wrote:
>>
>>> Use nodetool removenode is strongly preferred in most circumstances,
>>> and only resort to assassinate if you do not care about data
>>> consistency or you know there won't be any consistency issue (e.g. no new
>>> writes and did not run nodetool cleanup).
>>>
>>> Since the size of data on the new node is small, nodetool removenode
>>> should finish fairly quickly and bring your cluster back.
>>>
>>> Next time when you are doing something like this again, please test it
>>> out on a non-production environment, make sure everything works as expected
>>> before moving onto the production.
>>>
>>>
>>> On 03/04/2023 06:28, David Tinker wrote:
>>>
>>> Should I use assassinate or removenode? Given that there is some data on
>>> the node. Or will that be found on the other nodes? Sorry for all the
>>> questions but I really don't want to mess up.
>>>
>>> On Mon, Apr 3, 2023 at 7:21 AM Carlos Diaz  wrote:
>>>
 That's what nodetool assassinte will do.

 On Sun, Apr 2, 2023 at 10:19 PM David Tinker 
 wrote:

> Is it possible for me to remove the node from the cluster i.e. to undo
> this mess and get the cluster operating again?
>
> On Mon, Apr 3, 2023 at 7:13 AM Carlos Diaz 
> wrote:
>
>> You can leave it in the seed list of the other nodes, just make sure
>> it's not included in this node's seed list.  However, if you do decide to
>> fix the issue with the racks first assassinate this node (nodetool
>> assassinate ), and update the rack name before you restart.
>>
>> On Sun, Apr 2, 2023 at 10:06 PM David Tinker 
>> wrote:
>>
>>> It is also in the seeds list for the other nodes. Should I remove it
>>> from those, restart them one at a time, then restart it?
>>>
>>> /etc/cassandra # grep -i bootstrap *
>>> doesn't show anything so I don't think I have auto_bootstrap false.
>>>
>>> Thanks very much for the help.
>>>
>>>
>>> On Mon, Apr 3, 2023 at 7:01 AM Carlos Diaz 
>>> wrote:
>>>
 Just remove it from the seed list in the cassandra.yaml file and
 restart the node.  Make sure that auto_bootstrap is set to true

Re: Reads not returning data after adding node

2023-04-03 Thread Aaron Ploetz

The time it takes to stream data off of a node varies by network, cloud
region, and other factors.  So it's not unheard of for it to take a bit to
finish.

Just thought I'd mention that auto_bootstrap is true by default.  So if
you're not setting it, the node should bootstrap as long as it's not a seed
node.

As for the rack issue, yes, it's a good idea to keep your racks in
multiples of your RF.  When performing token ownership calculations,
Cassandra takes rack designation into consideration.  It tries to ensure
that multiple replicas for a row are not placed in the same rack.  TBH -
I'd build out two more nodes to have 6 nodes across 3 racks (2 in each),
just to ensure even distribution.  Otherwise, you might notice that the
nodes sharing a rack will consume disk at a different rate than the nodes
which have their own rack.

On Mon, Apr 3, 2023 at 8:57 AM David Tinker  wrote:

> Thanks. Hmm, the remove has been busy for hours but seems to be
> progressing.
>
> I have been running this on the nodes to monitor progress:
> # nodetool netstats | grep Already
> Receiving 92 files, 843934103369 bytes total. Already received 82
> files (89.13%), 590204687299 bytes total (69.93%)
> Sending 84 files, 860198753783 bytes total. Already sent 56 files
> (66.67%), 307038785732 bytes total (35.69%)
> Sending 78 files, 815573435637 bytes total. Already sent 56 files
> (71.79%), 313079823738 bytes total (38.39%)
>
> The percentages are ticking up.
>
> # nodetool ring | head -20
> Datacenter: dc1
> ==
> Address   RackStatus State   LoadOwns
>Token
>
>9189523899826545641
> xxx.xxx.xxx..24rack4   Down   Leaving 26.62 GiB   79.95%
>-9194674091837769168
> xxx.xxx.xxx.107   rack1   Up Normal  2.68 TiB73.25%
>-9168781258594813088
> xxx.xxx.xxx.253   rack2   Up Normal  2.63 TiB73.92%
>-9163037340977721917
> xxx.xxx.xxx.105   rack3   Up Normal  2.68 TiB72.88%
>-9148860739730046229
>
>
> On Mon, Apr 3, 2023 at 3:46 PM Bowen Song via user <
> user@cassandra.apache.org> wrote:
>
>> Use nodetool removenode is strongly preferred in most circumstances, and
>> only resort to assassinate if you do not care about data consistency or
>> you know there won't be any consistency issue (e.g. no new writes and did
>> not run nodetool cleanup).
>>
>> Since the size of data on the new node is small, nodetool removenode
>> should finish fairly quickly and bring your cluster back.
>>
>> Next time when you are doing something like this again, please test it
>> out on a non-production environment, make sure everything works as expected
>> before moving onto the production.
>>
>>
>> On 03/04/2023 06:28, David Tinker wrote:
>>
>> Should I use assassinate or removenode? Given that there is some data on
>> the node. Or will that be found on the other nodes? Sorry for all the
>> questions but I really don't want to mess up.
>>
>> On Mon, Apr 3, 2023 at 7:21 AM Carlos Diaz  wrote:
>>
>>> That's what nodetool assassinte will do.
>>>
>>> On Sun, Apr 2, 2023 at 10:19 PM David Tinker 
>>> wrote:
>>>
 Is it possible for me to remove the node from the cluster i.e. to undo
 this mess and get the cluster operating again?

 On Mon, Apr 3, 2023 at 7:13 AM Carlos Diaz  wrote:

> You can leave it in the seed list of the other nodes, just make sure
> it's not included in this node's seed list.  However, if you do decide to
> fix the issue with the racks first assassinate this node (nodetool
> assassinate ), and update the rack name before you restart.
>
> On Sun, Apr 2, 2023 at 10:06 PM David Tinker 
> wrote:
>
>> It is also in the seeds list for the other nodes. Should I remove it
>> from those, restart them one at a time, then restart it?
>>
>> /etc/cassandra # grep -i bootstrap *
>> doesn't show anything so I don't think I have auto_bootstrap false.
>>
>> Thanks very much for the help.
>>
>>
>> On Mon, Apr 3, 2023 at 7:01 AM Carlos Diaz 
>> wrote:
>>
>>> Just remove it from the seed list in the cassandra.yaml file and
>>> restart the node.  Make sure that auto_bootstrap is set to true first
>>> though.
>>>
>>> On Sun, Apr 2, 2023 at 9:59 PM David Tinker 
>>> wrote:
>>>
 So likely because I made it a seed node when I added it to the
 cluster it didn't do the bootstrap process. How can I recover this?

 On Mon, Apr 3, 2023 at 6:41 AM David Tinker 
 wrote:

> Yes replication factor is 3.
>
> I ran nodetool repair -pr on all the nodes (one at a time) and am
> still having issues getting data back from queries.
>
> I did make the new node a seed node.
>
> Re "rack4": I assumed that was just an indication as to the
>

Re: Reads not returning data after adding node

Thanks. Hmm, the remove has been busy for hours but seems to be progressing.

I have been running this on the nodes to monitor progress:
# nodetool netstats | grep Already
Receiving 92 files, 843934103369 bytes total. Already received 82
files (89.13%), 590204687299 bytes total (69.93%)
Sending 84 files, 860198753783 bytes total. Already sent 56 files
(66.67%), 307038785732 bytes total (35.69%)
Sending 78 files, 815573435637 bytes total. Already sent 56 files
(71.79%), 313079823738 bytes total (38.39%)

The percentages are ticking up.

# nodetool ring | head -20
Datacenter: dc1
==
Address   RackStatus State   LoadOwns
 Token

 9189523899826545641
xxx.xxx.xxx..24rack4   Down   Leaving 26.62 GiB   79.95%
   -9194674091837769168
xxx.xxx.xxx.107   rack1   Up Normal  2.68 TiB73.25%
 -9168781258594813088
xxx.xxx.xxx.253   rack2   Up Normal  2.63 TiB73.92%
 -9163037340977721917
xxx.xxx.xxx.105   rack3   Up Normal  2.68 TiB72.88%
 -9148860739730046229


On Mon, Apr 3, 2023 at 3:46 PM Bowen Song via user <
user@cassandra.apache.org> wrote:

> Use nodetool removenode is strongly preferred in most circumstances, and
> only resort to assassinate if you do not care about data consistency or
> you know there won't be any consistency issue (e.g. no new writes and did
> not run nodetool cleanup).
>
> Since the size of data on the new node is small, nodetool removenode
> should finish fairly quickly and bring your cluster back.
>
> Next time when you are doing something like this again, please test it out
> on a non-production environment, make sure everything works as expected
> before moving onto the production.
>
>
> On 03/04/2023 06:28, David Tinker wrote:
>
> Should I use assassinate or removenode? Given that there is some data on
> the node. Or will that be found on the other nodes? Sorry for all the
> questions but I really don't want to mess up.
>
> On Mon, Apr 3, 2023 at 7:21 AM Carlos Diaz  wrote:
>
>> That's what nodetool assassinte will do.
>>
>> On Sun, Apr 2, 2023 at 10:19 PM David Tinker 
>> wrote:
>>
>>> Is it possible for me to remove the node from the cluster i.e. to undo
>>> this mess and get the cluster operating again?
>>>
>>> On Mon, Apr 3, 2023 at 7:13 AM Carlos Diaz  wrote:
>>>
 You can leave it in the seed list of the other nodes, just make sure
 it's not included in this node's seed list.  However, if you do decide to
 fix the issue with the racks first assassinate this node (nodetool
 assassinate ), and update the rack name before you restart.

 On Sun, Apr 2, 2023 at 10:06 PM David Tinker 
 wrote:

> It is also in the seeds list for the other nodes. Should I remove it
> from those, restart them one at a time, then restart it?
>
> /etc/cassandra # grep -i bootstrap *
> doesn't show anything so I don't think I have auto_bootstrap false.
>
> Thanks very much for the help.
>
>
> On Mon, Apr 3, 2023 at 7:01 AM Carlos Diaz 
> wrote:
>
>> Just remove it from the seed list in the cassandra.yaml file and
>> restart the node.  Make sure that auto_bootstrap is set to true first
>> though.
>>
>> On Sun, Apr 2, 2023 at 9:59 PM David Tinker 
>> wrote:
>>
>>> So likely because I made it a seed node when I added it to the
>>> cluster it didn't do the bootstrap process. How can I recover this?
>>>
>>> On Mon, Apr 3, 2023 at 6:41 AM David Tinker 
>>> wrote:
>>>
 Yes replication factor is 3.

 I ran nodetool repair -pr on all the nodes (one at a time) and am
 still having issues getting data back from queries.

 I did make the new node a seed node.

 Re "rack4": I assumed that was just an indication as to the
 physical location of the server for redundancy. This one is separate 
 from
 the others so I used rack4.

 On Mon, Apr 3, 2023 at 6:30 AM Carlos Diaz 
 wrote:

> I'm assuming that your replication factor is 3.  If that's the
> case, did you intentionally put this node in rack 4?  Typically, you 
> want
> to add nodes in multiples of your replication factor in order to keep 
> the
> "racks" balanced.  In other words, this node should have been added 
> to rack
> 1, 2 or 3.
>
> Having said that, you should be able to easily fix your problem by
> running a nodetool repair -pr on the new node.
>
> On Sun, Apr 2, 2023 at 8:16 PM David Tinker <
> david.tin...@gmail.com> wrote:
>
>> Hi All
>>
>> I recently added a node to my 3 node Cassandra 4.0.5 cluster and
>> now many reads are not returning rows! What do I need to do to fix

Re: Reads not returning data after adding node

2023-04-03 Thread Bowen Song via user

Use nodetool removenode is strongly preferred in most circumstances, and 
only resort to assassinate if you do not care about data consistency or 
you know there won't be any consistency issue (e.g. no new writes and 
did not run nodetool cleanup).


Since the size of data on the new node is small, nodetool removenode 
should finish fairly quickly and bring your cluster back.


Next time when you are doing something like this again, please test it 
out on a non-production environment, make sure everything works as 
expected before moving onto the production.



On 03/04/2023 06:28, David Tinker wrote:
Should I use assassinate or removenode? Given that there is some data 
on the node. Or will that be found on the other nodes? Sorry for all 
the questions but I really don't want to mess up.


On Mon, Apr 3, 2023 at 7:21 AM Carlos Diaz  wrote:

That's what nodetool assassinte will do.

On Sun, Apr 2, 2023 at 10:19 PM David Tinker
 wrote:

Is it possible for me to remove the node from the cluster i.e.
to undo this mess and get the cluster operating again?

On Mon, Apr 3, 2023 at 7:13 AM Carlos Diaz
 wrote:

You can leave it in the seed list of the other nodes, just
make sure it's not included in this node's seed list. 
However, if you do decide to fix the issue with the racks
first assassinate this node (nodetool assassinate ),
and update the rack name before you restart.

On Sun, Apr 2, 2023 at 10:06 PM David Tinker
 wrote:

It is also in the seeds list for the other nodes.
Should I remove it from those, restart them one at a
time, then restart it?

/etc/cassandra # grep -i bootstrap *
doesn't show anything so I don't think I have
auto_bootstrap false.

Thanks very much for the help.


On Mon, Apr 3, 2023 at 7:01 AM Carlos Diaz
 wrote:

Just remove it from the seed list in the
cassandra.yaml file and restart the node.  Make
sure that auto_bootstrap is set to true first though.

On Sun, Apr 2, 2023 at 9:59 PM David Tinker
 wrote:

So likely because I made it a seed node when I
added it to the cluster it didn't do the
bootstrap process. How can I recover this?

On Mon, Apr 3, 2023 at 6:41 AM David Tinker
 wrote:

Yes replication factor is 3.

I ran nodetool repair -pr on all the nodes
(one at a time) and am still having issues
getting data back from queries.

I did make the new node a seed node.

Re "rack4": I assumed that was just an
indication as to the physical location of
the server for redundancy. This one is
separate from the others so I used rack4.

On Mon, Apr 3, 2023 at 6:30 AM Carlos Diaz
 wrote:

I'm assuming that your replication
factor is 3.  If that's the case, did
you intentionally put this node in
rack 4?  Typically, you want to add
nodes in multiples of your replication
factor in order to keep the "racks"
balanced.  In other words, this node
should have been added to rack 1, 2 or 3.

Having said that, you should be able
to easily fix your problem by running
a nodetool repair -pr on the new node.

On Sun, Apr 2, 2023 at 8:16 PM David
Tinker  wrote:

Hi All

I recently added a node to my 3
node Cassandra 4.0.5 cluster and
now many reads are not returning
rows! What do I need to do to fix
this? There weren't any errors in
the logs or other problems that I
could see. I expected the cluster
to balance itself but this hasn't
happened (yet?). The nodes are
similar so I have num_tokens=256

Re: Reads not returning data after adding node

If I have messed up with the rack thing I would like to get this node out
of the cluster so the cluster is functioning as quickly as possible. Then
do some more research and try again. So I am looking for the safest way to
do that.

On Mon, Apr 3, 2023 at 7:27 AM Jeff Jirsa  wrote:

> Just run nodetool rebuild on the new node
>
> If you assassinate it now you violate consistency for your most recent
> writes
>
>
>
> On Apr 2, 2023, at 10:22 PM, Carlos Diaz  wrote:
>
> 
> That's what nodetool assassinte will do.
>
> On Sun, Apr 2, 2023 at 10:19 PM David Tinker 
> wrote:
>
>> Is it possible for me to remove the node from the cluster i.e. to undo
>> this mess and get the cluster operating again?
>>
>> On Mon, Apr 3, 2023 at 7:13 AM Carlos Diaz  wrote:
>>
>>> You can leave it in the seed list of the other nodes, just make sure
>>> it's not included in this node's seed list.  However, if you do decide to
>>> fix the issue with the racks first assassinate this node (nodetool
>>> assassinate ), and update the rack name before you restart.
>>>
>>> On Sun, Apr 2, 2023 at 10:06 PM David Tinker 
>>> wrote:
>>>
 It is also in the seeds list for the other nodes. Should I remove it
 from those, restart them one at a time, then restart it?

 /etc/cassandra # grep -i bootstrap *
 doesn't show anything so I don't think I have auto_bootstrap false.

 Thanks very much for the help.


 On Mon, Apr 3, 2023 at 7:01 AM Carlos Diaz  wrote:

> Just remove it from the seed list in the cassandra.yaml file and
> restart the node.  Make sure that auto_bootstrap is set to true first
> though.
>
> On Sun, Apr 2, 2023 at 9:59 PM David Tinker 
> wrote:
>
>> So likely because I made it a seed node when I added it to the
>> cluster it didn't do the bootstrap process. How can I recover this?
>>
>> On Mon, Apr 3, 2023 at 6:41 AM David Tinker 
>> wrote:
>>
>>> Yes replication factor is 3.
>>>
>>> I ran nodetool repair -pr on all the nodes (one at a time) and am
>>> still having issues getting data back from queries.
>>>
>>> I did make the new node a seed node.
>>>
>>> Re "rack4": I assumed that was just an indication as to the physical
>>> location of the server for redundancy. This one is separate from the 
>>> others
>>> so I used rack4.
>>>
>>> On Mon, Apr 3, 2023 at 6:30 AM Carlos Diaz 
>>> wrote:
>>>
 I'm assuming that your replication factor is 3.  If that's the
 case, did you intentionally put this node in rack 4?  Typically, you 
 want
 to add nodes in multiples of your replication factor in order to keep 
 the
 "racks" balanced.  In other words, this node should have been added to 
 rack
 1, 2 or 3.

 Having said that, you should be able to easily fix your problem by
 running a nodetool repair -pr on the new node.

 On Sun, Apr 2, 2023 at 8:16 PM David Tinker 
 wrote:

> Hi All
>
> I recently added a node to my 3 node Cassandra 4.0.5 cluster and
> now many reads are not returning rows! What do I need to do to fix 
> this?
> There weren't any errors in the logs or other problems that I could 
> see. I
> expected the cluster to balance itself but this hasn't happened 
> (yet?). The
> nodes are similar so I have num_tokens=256 for each. I am using the
> Murmur3Partitioner.
>
> # nodetool status
> Datacenter: dc1
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address  Load   Tokens  Owns (effective)  Host ID
>   Rack
> UN  xxx.xxx.xxx.105  2.65 TiB   256 72.9%
> afd02287-3f88-4c6f-8b27-06f7a8192402  rack3
> UN  xxx.xxx.xxx.253  2.6 TiB256 73.9%
> e1af72be-e5df-4c6b-a124-c7bc48c6602a  rack2
> UN  xxx.xxx.xxx.24   93.82 KiB  256 80.0%
> c4e8b4a0-f014-45e6-afb4-648aad4f8500  rack4
> UN  xxx.xxx.xxx.107  2.65 TiB   256 73.2%
> ab72f017-be96-41d2-9bef-a551dec2c7b5  rack1
>
> # nodetool netstats
> Mode: NORMAL
> Not sending any streams.
> Read Repair Statistics:
> Attempted: 0
> Mismatch (Blocking): 0
> Mismatch (Background): 0
> Pool NameActive   Pending  Completed
> Dropped
> Large messages  n/a 0  71754
>   0
> Small messages  n/a 08398184
>  14
> Gossip messages n/a 01303634
>   0
>
> # nodetool ring
> Datacenter: dc1
> ==
> Address   RackStatus State

Re: Reads not returning data after adding node

Should I use assassinate or removenode? Given that there is some data on
the node. Or will that be found on the other nodes? Sorry for all the
questions but I really don't want to mess up.

On Mon, Apr 3, 2023 at 7:21 AM Carlos Diaz  wrote:

> That's what nodetool assassinte will do.
>
> On Sun, Apr 2, 2023 at 10:19 PM David Tinker 
> wrote:
>
>> Is it possible for me to remove the node from the cluster i.e. to undo
>> this mess and get the cluster operating again?
>>
>> On Mon, Apr 3, 2023 at 7:13 AM Carlos Diaz  wrote:
>>
>>> You can leave it in the seed list of the other nodes, just make sure
>>> it's not included in this node's seed list.  However, if you do decide to
>>> fix the issue with the racks first assassinate this node (nodetool
>>> assassinate ), and update the rack name before you restart.
>>>
>>> On Sun, Apr 2, 2023 at 10:06 PM David Tinker 
>>> wrote:
>>>
 It is also in the seeds list for the other nodes. Should I remove it
 from those, restart them one at a time, then restart it?

 /etc/cassandra # grep -i bootstrap *
 doesn't show anything so I don't think I have auto_bootstrap false.

 Thanks very much for the help.


 On Mon, Apr 3, 2023 at 7:01 AM Carlos Diaz  wrote:

> Just remove it from the seed list in the cassandra.yaml file and
> restart the node.  Make sure that auto_bootstrap is set to true first
> though.
>
> On Sun, Apr 2, 2023 at 9:59 PM David Tinker 
> wrote:
>
>> So likely because I made it a seed node when I added it to the
>> cluster it didn't do the bootstrap process. How can I recover this?
>>
>> On Mon, Apr 3, 2023 at 6:41 AM David Tinker 
>> wrote:
>>
>>> Yes replication factor is 3.
>>>
>>> I ran nodetool repair -pr on all the nodes (one at a time) and am
>>> still having issues getting data back from queries.
>>>
>>> I did make the new node a seed node.
>>>
>>> Re "rack4": I assumed that was just an indication as to the physical
>>> location of the server for redundancy. This one is separate from the 
>>> others
>>> so I used rack4.
>>>
>>> On Mon, Apr 3, 2023 at 6:30 AM Carlos Diaz 
>>> wrote:
>>>
 I'm assuming that your replication factor is 3.  If that's the
 case, did you intentionally put this node in rack 4?  Typically, you 
 want
 to add nodes in multiples of your replication factor in order to keep 
 the
 "racks" balanced.  In other words, this node should have been added to 
 rack
 1, 2 or 3.

 Having said that, you should be able to easily fix your problem by
 running a nodetool repair -pr on the new node.

 On Sun, Apr 2, 2023 at 8:16 PM David Tinker 
 wrote:

> Hi All
>
> I recently added a node to my 3 node Cassandra 4.0.5 cluster and
> now many reads are not returning rows! What do I need to do to fix 
> this?
> There weren't any errors in the logs or other problems that I could 
> see. I
> expected the cluster to balance itself but this hasn't happened 
> (yet?). The
> nodes are similar so I have num_tokens=256 for each. I am using the
> Murmur3Partitioner.
>
> # nodetool status
> Datacenter: dc1
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address  Load   Tokens  Owns (effective)  Host ID
>   Rack
> UN  xxx.xxx.xxx.105  2.65 TiB   256 72.9%
> afd02287-3f88-4c6f-8b27-06f7a8192402  rack3
> UN  xxx.xxx.xxx.253  2.6 TiB256 73.9%
> e1af72be-e5df-4c6b-a124-c7bc48c6602a  rack2
> UN  xxx.xxx.xxx.24   93.82 KiB  256 80.0%
> c4e8b4a0-f014-45e6-afb4-648aad4f8500  rack4
> UN  xxx.xxx.xxx.107  2.65 TiB   256 73.2%
> ab72f017-be96-41d2-9bef-a551dec2c7b5  rack1
>
> # nodetool netstats
> Mode: NORMAL
> Not sending any streams.
> Read Repair Statistics:
> Attempted: 0
> Mismatch (Blocking): 0
> Mismatch (Background): 0
> Pool NameActive   Pending  Completed
> Dropped
> Large messages  n/a 0  71754
>   0
> Small messages  n/a 08398184
>  14
> Gossip messages n/a 01303634
>   0
>
> # nodetool ring
> Datacenter: dc1
> ==
> Address   RackStatus State   Load
>  OwnsToken
>
>9189523899826545641
> xxx.xxx.xxx.24rack4   Up Normal  93.82 KiB
> 79.95%  -9194674091837769168
>

Re: Reads not returning data after adding node

2023-04-02 Thread Jeff Jirsa

Just run nodetool rebuild on the new node If you assassinate it now you violate consistency for your most recent writes On Apr 2, 2023, at 10:22 PM, Carlos Diaz  wrote:That's what nodetool assassinte will do.On Sun, Apr 2, 2023 at 10:19 PM David Tinker  wrote:Is it possible for me to remove the node from the cluster i.e. to undo this mess and get the cluster operating again?On Mon, Apr 3, 2023 at 7:13 AM Carlos Diaz  wrote:You can leave it in the seed list of the other nodes, just make sure it's not included in this node's seed list.  However, if you do decide to fix the issue with the racks first assassinate this node (nodetool assassinate ), and update the rack name before you restart.  On Sun, Apr 2, 2023 at 10:06 PM David Tinker  wrote:It is also in the seeds list for the other nodes. Should I remove it from those, restart them one at a time, then restart it?/etc/cassandra # grep -i bootstrap *doesn't show anything so I don't think I have auto_bootstrap false.Thanks very much for the help.On Mon, Apr 3, 2023 at 7:01 AM Carlos Diaz  wrote:Just remove it from the seed list in the cassandra.yaml file and restart the node.  Make sure that auto_bootstrap is set to true first though.  On Sun, Apr 2, 2023 at 9:59 PM David Tinker  wrote:So likely because I made it a seed node when I added it to the cluster it didn't do the bootstrap process. How can I recover this?On Mon, Apr 3, 2023 at 6:41 AM David Tinker  wrote:Yes replication factor is 3.I ran nodetool repair -pr on all the nodes (one at a time) and am still having issues getting data back from queries.I did make the new node a seed node.Re "rack4": I assumed that was just an indication as to the physical location of the server for redundancy. This one is separate from the others so I used rack4.On Mon, Apr 3, 2023 at 6:30 AM Carlos Diaz  wrote:I'm assuming that your replication factor is 3.  If that's the case, did you intentionally put this node in rack 4?  Typically, you want to add nodes in multiples of your replication factor in order to keep the "racks" balanced.  In other words, this node should have been added to rack 1, 2 or 3.  Having said that, you should be able to easily fix your problem by running a nodetool repair -pr on the new node.  On Sun, Apr 2, 2023 at 8:16 PM David Tinker  wrote:Hi AllI recently added a node to my 3 node Cassandra 4.0.5 cluster and now many reads are not returning rows! What do I need to do to fix this? There weren't any errors in the logs or other problems that I could see. I expected the cluster to balance itself but this hasn't happened (yet?). The nodes are similar so I have num_tokens=256 for each. I am using the Murmur3Partitioner.# nodetool statusDatacenter: dc1===Status=Up/Down|/ State=Normal/Leaving/Joining/Moving--  Address          Load       Tokens  Owns (effective)  Host ID                               RackUN  xxx.xxx.xxx.105  2.65 TiB   256     72.9%             afd02287-3f88-4c6f-8b27-06f7a8192402  rack3UN  xxx.xxx.xxx.253  2.6 TiB    256     73.9%             e1af72be-e5df-4c6b-a124-c7bc48c6602a  rack2UN  xxx.xxx.xxx.24   93.82 KiB  256     80.0%             c4e8b4a0-f014-45e6-afb4-648aad4f8500  rack4UN  xxx.xxx.xxx.107  2.65 TiB   256     73.2%             ab72f017-be96-41d2-9bef-a551dec2c7b5  rack1# nodetool netstatsMode: NORMALNot sending any streams.Read Repair Statistics:Attempted: 0Mismatch (Blocking): 0Mismatch (Background): 0Pool Name                    Active   Pending      Completed   DroppedLarge messages                  n/a         0          71754         0Small messages                  n/a         0        8398184        14Gossip messages                 n/a         0        1303634         0# nodetool ringDatacenter: dc1==Address               Rack        Status State   Load            Owns                Token                                                                                     9189523899826545641xxx.xxx.xxx.24        rack4       Up     Normal  93.82 KiB       79.95%              -9194674091837769168xxx.xxx.xxx.107       rack1       Up     Normal  2.65 TiB        73.25%              -9168781258594813088xxx.xxx.xxx.253       rack2       Up     Normal  2.6 TiB         73.92%              -9163037340977721917xxx.xxx.xxx.105       rack3       Up     Normal  2.65 TiB        72.88%              -9148860739730046229xxx.xxx.xxx.107       rack1       Up     Normal  2.65 TiB        73.25%              -9125240034139323535xxx.xxx.xxx.253       rack2       Up     Normal  2.6 TiB         73.92%              -9112518853051755414xxx.xxx.xxx.105       rack3       Up     Normal  2.65 TiB        72.88%              -9100516173422432134...This is causing a serious production issue. Please help if you can.ThanksDavid

Re: Reads not returning data after adding node

That's what nodetool assassinte will do.

On Sun, Apr 2, 2023 at 10:19 PM David Tinker  wrote:

> Is it possible for me to remove the node from the cluster i.e. to undo
> this mess and get the cluster operating again?
>
> On Mon, Apr 3, 2023 at 7:13 AM Carlos Diaz  wrote:
>
>> You can leave it in the seed list of the other nodes, just make sure it's
>> not included in this node's seed list.  However, if you do decide to fix
>> the issue with the racks first assassinate this node (nodetool assassinate
>> ), and update the rack name before you restart.
>>
>> On Sun, Apr 2, 2023 at 10:06 PM David Tinker 
>> wrote:
>>
>>> It is also in the seeds list for the other nodes. Should I remove it
>>> from those, restart them one at a time, then restart it?
>>>
>>> /etc/cassandra # grep -i bootstrap *
>>> doesn't show anything so I don't think I have auto_bootstrap false.
>>>
>>> Thanks very much for the help.
>>>
>>>
>>> On Mon, Apr 3, 2023 at 7:01 AM Carlos Diaz  wrote:
>>>
 Just remove it from the seed list in the cassandra.yaml file and
 restart the node.  Make sure that auto_bootstrap is set to true first
 though.

 On Sun, Apr 2, 2023 at 9:59 PM David Tinker 
 wrote:

> So likely because I made it a seed node when I added it to the cluster
> it didn't do the bootstrap process. How can I recover this?
>
> On Mon, Apr 3, 2023 at 6:41 AM David Tinker 
> wrote:
>
>> Yes replication factor is 3.
>>
>> I ran nodetool repair -pr on all the nodes (one at a time) and am
>> still having issues getting data back from queries.
>>
>> I did make the new node a seed node.
>>
>> Re "rack4": I assumed that was just an indication as to the physical
>> location of the server for redundancy. This one is separate from the 
>> others
>> so I used rack4.
>>
>> On Mon, Apr 3, 2023 at 6:30 AM Carlos Diaz 
>> wrote:
>>
>>> I'm assuming that your replication factor is 3.  If that's the case,
>>> did you intentionally put this node in rack 4?  Typically, you want to 
>>> add
>>> nodes in multiples of your replication factor in order to keep the 
>>> "racks"
>>> balanced.  In other words, this node should have been added to rack 1, 
>>> 2 or
>>> 3.
>>>
>>> Having said that, you should be able to easily fix your problem by
>>> running a nodetool repair -pr on the new node.
>>>
>>> On Sun, Apr 2, 2023 at 8:16 PM David Tinker 
>>> wrote:
>>>
 Hi All

 I recently added a node to my 3 node Cassandra 4.0.5 cluster and
 now many reads are not returning rows! What do I need to do to fix 
 this?
 There weren't any errors in the logs or other problems that I could 
 see. I
 expected the cluster to balance itself but this hasn't happened 
 (yet?). The
 nodes are similar so I have num_tokens=256 for each. I am using the
 Murmur3Partitioner.

 # nodetool status
 Datacenter: dc1
 ===
 Status=Up/Down
 |/ State=Normal/Leaving/Joining/Moving
 --  Address  Load   Tokens  Owns (effective)  Host ID
 Rack
 UN  xxx.xxx.xxx.105  2.65 TiB   256 72.9%
 afd02287-3f88-4c6f-8b27-06f7a8192402  rack3
 UN  xxx.xxx.xxx.253  2.6 TiB256 73.9%
 e1af72be-e5df-4c6b-a124-c7bc48c6602a  rack2
 UN  xxx.xxx.xxx.24   93.82 KiB  256 80.0%
 c4e8b4a0-f014-45e6-afb4-648aad4f8500  rack4
 UN  xxx.xxx.xxx.107  2.65 TiB   256 73.2%
 ab72f017-be96-41d2-9bef-a551dec2c7b5  rack1

 # nodetool netstats
 Mode: NORMAL
 Not sending any streams.
 Read Repair Statistics:
 Attempted: 0
 Mismatch (Blocking): 0
 Mismatch (Background): 0
 Pool NameActive   Pending  Completed
 Dropped
 Large messages  n/a 0  71754
   0
 Small messages  n/a 08398184
  14
 Gossip messages n/a 01303634
   0

 # nodetool ring
 Datacenter: dc1
 ==
 Address   RackStatus State   Load
  OwnsToken

  9189523899826545641
 xxx.xxx.xxx.24rack4   Up Normal  93.82 KiB
 79.95%  -9194674091837769168
 xxx.xxx.xxx.107   rack1   Up Normal  2.65 TiB
  73.25%  -9168781258594813088
 xxx.xxx.xxx.253   rack2   Up Normal  2.6 TiB
 73.92%  -9163037340977721917
 xxx.xxx.xxx.105   rack3   Up Normal  2.65 TiB
  72.88%  -9148860739730046229
 xxx.xxx.xxx.107

Re: Reads not returning data after adding node

Is it possible for me to remove the node from the cluster i.e. to undo this
mess and get the cluster operating again?

On Mon, Apr 3, 2023 at 7:13 AM Carlos Diaz  wrote:

> You can leave it in the seed list of the other nodes, just make sure it's
> not included in this node's seed list.  However, if you do decide to fix
> the issue with the racks first assassinate this node (nodetool assassinate
> ), and update the rack name before you restart.
>
> On Sun, Apr 2, 2023 at 10:06 PM David Tinker 
> wrote:
>
>> It is also in the seeds list for the other nodes. Should I remove it from
>> those, restart them one at a time, then restart it?
>>
>> /etc/cassandra # grep -i bootstrap *
>> doesn't show anything so I don't think I have auto_bootstrap false.
>>
>> Thanks very much for the help.
>>
>>
>> On Mon, Apr 3, 2023 at 7:01 AM Carlos Diaz  wrote:
>>
>>> Just remove it from the seed list in the cassandra.yaml file and restart
>>> the node.  Make sure that auto_bootstrap is set to true first though.
>>>
>>> On Sun, Apr 2, 2023 at 9:59 PM David Tinker 
>>> wrote:
>>>
 So likely because I made it a seed node when I added it to the cluster
 it didn't do the bootstrap process. How can I recover this?

 On Mon, Apr 3, 2023 at 6:41 AM David Tinker 
 wrote:

> Yes replication factor is 3.
>
> I ran nodetool repair -pr on all the nodes (one at a time) and am
> still having issues getting data back from queries.
>
> I did make the new node a seed node.
>
> Re "rack4": I assumed that was just an indication as to the physical
> location of the server for redundancy. This one is separate from the 
> others
> so I used rack4.
>
> On Mon, Apr 3, 2023 at 6:30 AM Carlos Diaz 
> wrote:
>
>> I'm assuming that your replication factor is 3.  If that's the case,
>> did you intentionally put this node in rack 4?  Typically, you want to 
>> add
>> nodes in multiples of your replication factor in order to keep the 
>> "racks"
>> balanced.  In other words, this node should have been added to rack 1, 2 
>> or
>> 3.
>>
>> Having said that, you should be able to easily fix your problem by
>> running a nodetool repair -pr on the new node.
>>
>> On Sun, Apr 2, 2023 at 8:16 PM David Tinker 
>> wrote:
>>
>>> Hi All
>>>
>>> I recently added a node to my 3 node Cassandra 4.0.5 cluster and now
>>> many reads are not returning rows! What do I need to do to fix this? 
>>> There
>>> weren't any errors in the logs or other problems that I could see. I
>>> expected the cluster to balance itself but this hasn't happened (yet?). 
>>> The
>>> nodes are similar so I have num_tokens=256 for each. I am using the
>>> Murmur3Partitioner.
>>>
>>> # nodetool status
>>> Datacenter: dc1
>>> ===
>>> Status=Up/Down
>>> |/ State=Normal/Leaving/Joining/Moving
>>> --  Address  Load   Tokens  Owns (effective)  Host ID
>>> Rack
>>> UN  xxx.xxx.xxx.105  2.65 TiB   256 72.9%
>>> afd02287-3f88-4c6f-8b27-06f7a8192402  rack3
>>> UN  xxx.xxx.xxx.253  2.6 TiB256 73.9%
>>> e1af72be-e5df-4c6b-a124-c7bc48c6602a  rack2
>>> UN  xxx.xxx.xxx.24   93.82 KiB  256 80.0%
>>> c4e8b4a0-f014-45e6-afb4-648aad4f8500  rack4
>>> UN  xxx.xxx.xxx.107  2.65 TiB   256 73.2%
>>> ab72f017-be96-41d2-9bef-a551dec2c7b5  rack1
>>>
>>> # nodetool netstats
>>> Mode: NORMAL
>>> Not sending any streams.
>>> Read Repair Statistics:
>>> Attempted: 0
>>> Mismatch (Blocking): 0
>>> Mismatch (Background): 0
>>> Pool NameActive   Pending  Completed
>>> Dropped
>>> Large messages  n/a 0  71754
>>> 0
>>> Small messages  n/a 08398184
>>>  14
>>> Gossip messages n/a 01303634
>>> 0
>>>
>>> # nodetool ring
>>> Datacenter: dc1
>>> ==
>>> Address   RackStatus State   Load
>>>  OwnsToken
>>>
>>>  9189523899826545641
>>> xxx.xxx.xxx.24rack4   Up Normal  93.82 KiB
>>> 79.95%  -9194674091837769168
>>> xxx.xxx.xxx.107   rack1   Up Normal  2.65 TiB
>>>  73.25%  -9168781258594813088
>>> xxx.xxx.xxx.253   rack2   Up Normal  2.6 TiB
>>> 73.92%  -9163037340977721917
>>> xxx.xxx.xxx.105   rack3   Up Normal  2.65 TiB
>>>  72.88%  -9148860739730046229
>>> xxx.xxx.xxx.107   rack1   Up Normal  2.65 TiB
>>>  73.25%  -9125240034139323535
>>> xxx.xxx.xxx.253   rack2   Up Normal  2.6 TiB
>>> 73.92%  -9112518853051755414
>>> xxx.xxx.xxx.105   rack3   Up

Re: Reads not returning data after adding node

You can leave it in the seed list of the other nodes, just make sure it's
not included in this node's seed list.  However, if you do decide to fix
the issue with the racks first assassinate this node (nodetool assassinate
), and update the rack name before you restart.

On Sun, Apr 2, 2023 at 10:06 PM David Tinker  wrote:

> It is also in the seeds list for the other nodes. Should I remove it from
> those, restart them one at a time, then restart it?
>
> /etc/cassandra # grep -i bootstrap *
> doesn't show anything so I don't think I have auto_bootstrap false.
>
> Thanks very much for the help.
>
>
> On Mon, Apr 3, 2023 at 7:01 AM Carlos Diaz  wrote:
>
>> Just remove it from the seed list in the cassandra.yaml file and restart
>> the node.  Make sure that auto_bootstrap is set to true first though.
>>
>> On Sun, Apr 2, 2023 at 9:59 PM David Tinker 
>> wrote:
>>
>>> So likely because I made it a seed node when I added it to the cluster
>>> it didn't do the bootstrap process. How can I recover this?
>>>
>>> On Mon, Apr 3, 2023 at 6:41 AM David Tinker 
>>> wrote:
>>>
 Yes replication factor is 3.

 I ran nodetool repair -pr on all the nodes (one at a time) and am
 still having issues getting data back from queries.

 I did make the new node a seed node.

 Re "rack4": I assumed that was just an indication as to the physical
 location of the server for redundancy. This one is separate from the others
 so I used rack4.

 On Mon, Apr 3, 2023 at 6:30 AM Carlos Diaz  wrote:

> I'm assuming that your replication factor is 3.  If that's the case,
> did you intentionally put this node in rack 4?  Typically, you want to add
> nodes in multiples of your replication factor in order to keep the "racks"
> balanced.  In other words, this node should have been added to rack 1, 2 
> or
> 3.
>
> Having said that, you should be able to easily fix your problem by
> running a nodetool repair -pr on the new node.
>
> On Sun, Apr 2, 2023 at 8:16 PM David Tinker 
> wrote:
>
>> Hi All
>>
>> I recently added a node to my 3 node Cassandra 4.0.5 cluster and now
>> many reads are not returning rows! What do I need to do to fix this? 
>> There
>> weren't any errors in the logs or other problems that I could see. I
>> expected the cluster to balance itself but this hasn't happened (yet?). 
>> The
>> nodes are similar so I have num_tokens=256 for each. I am using the
>> Murmur3Partitioner.
>>
>> # nodetool status
>> Datacenter: dc1
>> ===
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> --  Address  Load   Tokens  Owns (effective)  Host ID
>>   Rack
>> UN  xxx.xxx.xxx.105  2.65 TiB   256 72.9%
>> afd02287-3f88-4c6f-8b27-06f7a8192402  rack3
>> UN  xxx.xxx.xxx.253  2.6 TiB256 73.9%
>> e1af72be-e5df-4c6b-a124-c7bc48c6602a  rack2
>> UN  xxx.xxx.xxx.24   93.82 KiB  256 80.0%
>> c4e8b4a0-f014-45e6-afb4-648aad4f8500  rack4
>> UN  xxx.xxx.xxx.107  2.65 TiB   256 73.2%
>> ab72f017-be96-41d2-9bef-a551dec2c7b5  rack1
>>
>> # nodetool netstats
>> Mode: NORMAL
>> Not sending any streams.
>> Read Repair Statistics:
>> Attempted: 0
>> Mismatch (Blocking): 0
>> Mismatch (Background): 0
>> Pool NameActive   Pending  Completed   Dropped
>> Large messages  n/a 0  71754 0
>> Small messages  n/a 0839818414
>> Gossip messages n/a 01303634 0
>>
>> # nodetool ring
>> Datacenter: dc1
>> ==
>> Address   RackStatus State   LoadOwns
>>Token
>>
>>9189523899826545641
>> xxx.xxx.xxx.24rack4   Up Normal  93.82 KiB
>> 79.95%  -9194674091837769168
>> xxx.xxx.xxx.107   rack1   Up Normal  2.65 TiB
>>  73.25%  -9168781258594813088
>> xxx.xxx.xxx.253   rack2   Up Normal  2.6 TiB
>> 73.92%  -9163037340977721917
>> xxx.xxx.xxx.105   rack3   Up Normal  2.65 TiB
>>  72.88%  -9148860739730046229
>> xxx.xxx.xxx.107   rack1   Up Normal  2.65 TiB
>>  73.25%  -9125240034139323535
>> xxx.xxx.xxx.253   rack2   Up Normal  2.6 TiB
>> 73.92%  -9112518853051755414
>> xxx.xxx.xxx.105   rack3   Up Normal  2.65 TiB
>>  72.88%  -9100516173422432134
>> ...
>>
>> This is causing a serious production issue. Please help if you can.
>>
>> Thanks
>> David
>>
>>
>>
>>

Re: Reads not returning data after adding node

I should add that the new node does have some data.

On Mon, Apr 3, 2023 at 7:04 AM David Tinker  wrote:

> It is also in the seeds list for the other nodes. Should I remove it from
> those, restart them one at a time, then restart it?
>
> /etc/cassandra # grep -i bootstrap *
> doesn't show anything so I don't think I have auto_bootstrap false.
>
> Thanks very much for the help.
>
>
> On Mon, Apr 3, 2023 at 7:01 AM Carlos Diaz  wrote:
>
>> Just remove it from the seed list in the cassandra.yaml file and restart
>> the node.  Make sure that auto_bootstrap is set to true first though.
>>
>> On Sun, Apr 2, 2023 at 9:59 PM David Tinker 
>> wrote:
>>
>>> So likely because I made it a seed node when I added it to the cluster
>>> it didn't do the bootstrap process. How can I recover this?
>>>
>>> On Mon, Apr 3, 2023 at 6:41 AM David Tinker 
>>> wrote:
>>>
 Yes replication factor is 3.

 I ran nodetool repair -pr on all the nodes (one at a time) and am
 still having issues getting data back from queries.

 I did make the new node a seed node.

 Re "rack4": I assumed that was just an indication as to the physical
 location of the server for redundancy. This one is separate from the others
 so I used rack4.

 On Mon, Apr 3, 2023 at 6:30 AM Carlos Diaz  wrote:

> I'm assuming that your replication factor is 3.  If that's the case,
> did you intentionally put this node in rack 4?  Typically, you want to add
> nodes in multiples of your replication factor in order to keep the "racks"
> balanced.  In other words, this node should have been added to rack 1, 2 
> or
> 3.
>
> Having said that, you should be able to easily fix your problem by
> running a nodetool repair -pr on the new node.
>
> On Sun, Apr 2, 2023 at 8:16 PM David Tinker 
> wrote:
>
>> Hi All
>>
>> I recently added a node to my 3 node Cassandra 4.0.5 cluster and now
>> many reads are not returning rows! What do I need to do to fix this? 
>> There
>> weren't any errors in the logs or other problems that I could see. I
>> expected the cluster to balance itself but this hasn't happened (yet?). 
>> The
>> nodes are similar so I have num_tokens=256 for each. I am using the
>> Murmur3Partitioner.
>>
>> # nodetool status
>> Datacenter: dc1
>> ===
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> --  Address  Load   Tokens  Owns (effective)  Host ID
>>   Rack
>> UN  xxx.xxx.xxx.105  2.65 TiB   256 72.9%
>> afd02287-3f88-4c6f-8b27-06f7a8192402  rack3
>> UN  xxx.xxx.xxx.253  2.6 TiB256 73.9%
>> e1af72be-e5df-4c6b-a124-c7bc48c6602a  rack2
>> UN  xxx.xxx.xxx.24   93.82 KiB  256 80.0%
>> c4e8b4a0-f014-45e6-afb4-648aad4f8500  rack4
>> UN  xxx.xxx.xxx.107  2.65 TiB   256 73.2%
>> ab72f017-be96-41d2-9bef-a551dec2c7b5  rack1
>>
>> # nodetool netstats
>> Mode: NORMAL
>> Not sending any streams.
>> Read Repair Statistics:
>> Attempted: 0
>> Mismatch (Blocking): 0
>> Mismatch (Background): 0
>> Pool NameActive   Pending  Completed   Dropped
>> Large messages  n/a 0  71754 0
>> Small messages  n/a 0839818414
>> Gossip messages n/a 01303634 0
>>
>> # nodetool ring
>> Datacenter: dc1
>> ==
>> Address   RackStatus State   LoadOwns
>>Token
>>
>>9189523899826545641
>> xxx.xxx.xxx.24rack4   Up Normal  93.82 KiB
>> 79.95%  -9194674091837769168
>> xxx.xxx.xxx.107   rack1   Up Normal  2.65 TiB
>>  73.25%  -9168781258594813088
>> xxx.xxx.xxx.253   rack2   Up Normal  2.6 TiB
>> 73.92%  -9163037340977721917
>> xxx.xxx.xxx.105   rack3   Up Normal  2.65 TiB
>>  72.88%  -9148860739730046229
>> xxx.xxx.xxx.107   rack1   Up Normal  2.65 TiB
>>  73.25%  -9125240034139323535
>> xxx.xxx.xxx.253   rack2   Up Normal  2.6 TiB
>> 73.92%  -9112518853051755414
>> xxx.xxx.xxx.105   rack3   Up Normal  2.65 TiB
>>  72.88%  -9100516173422432134
>> ...
>>
>> This is causing a serious production issue. Please help if you can.
>>
>> Thanks
>> David
>>
>>
>>
>>

Re: Reads not returning data after adding node

It is also in the seeds list for the other nodes. Should I remove it from
those, restart them one at a time, then restart it?

/etc/cassandra # grep -i bootstrap *
doesn't show anything so I don't think I have auto_bootstrap false.

Thanks very much for the help.


On Mon, Apr 3, 2023 at 7:01 AM Carlos Diaz  wrote:

> Just remove it from the seed list in the cassandra.yaml file and restart
> the node.  Make sure that auto_bootstrap is set to true first though.
>
> On Sun, Apr 2, 2023 at 9:59 PM David Tinker 
> wrote:
>
>> So likely because I made it a seed node when I added it to the cluster it
>> didn't do the bootstrap process. How can I recover this?
>>
>> On Mon, Apr 3, 2023 at 6:41 AM David Tinker 
>> wrote:
>>
>>> Yes replication factor is 3.
>>>
>>> I ran nodetool repair -pr on all the nodes (one at a time) and am still
>>> having issues getting data back from queries.
>>>
>>> I did make the new node a seed node.
>>>
>>> Re "rack4": I assumed that was just an indication as to the physical
>>> location of the server for redundancy. This one is separate from the others
>>> so I used rack4.
>>>
>>> On Mon, Apr 3, 2023 at 6:30 AM Carlos Diaz  wrote:
>>>
 I'm assuming that your replication factor is 3.  If that's the case,
 did you intentionally put this node in rack 4?  Typically, you want to add
 nodes in multiples of your replication factor in order to keep the "racks"
 balanced.  In other words, this node should have been added to rack 1, 2 or
 3.

 Having said that, you should be able to easily fix your problem by
 running a nodetool repair -pr on the new node.

 On Sun, Apr 2, 2023 at 8:16 PM David Tinker 
 wrote:

> Hi All
>
> I recently added a node to my 3 node Cassandra 4.0.5 cluster and now
> many reads are not returning rows! What do I need to do to fix this? There
> weren't any errors in the logs or other problems that I could see. I
> expected the cluster to balance itself but this hasn't happened (yet?). 
> The
> nodes are similar so I have num_tokens=256 for each. I am using the
> Murmur3Partitioner.
>
> # nodetool status
> Datacenter: dc1
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address  Load   Tokens  Owns (effective)  Host ID
>   Rack
> UN  xxx.xxx.xxx.105  2.65 TiB   256 72.9%
> afd02287-3f88-4c6f-8b27-06f7a8192402  rack3
> UN  xxx.xxx.xxx.253  2.6 TiB256 73.9%
> e1af72be-e5df-4c6b-a124-c7bc48c6602a  rack2
> UN  xxx.xxx.xxx.24   93.82 KiB  256 80.0%
> c4e8b4a0-f014-45e6-afb4-648aad4f8500  rack4
> UN  xxx.xxx.xxx.107  2.65 TiB   256 73.2%
> ab72f017-be96-41d2-9bef-a551dec2c7b5  rack1
>
> # nodetool netstats
> Mode: NORMAL
> Not sending any streams.
> Read Repair Statistics:
> Attempted: 0
> Mismatch (Blocking): 0
> Mismatch (Background): 0
> Pool NameActive   Pending  Completed   Dropped
> Large messages  n/a 0  71754 0
> Small messages  n/a 0839818414
> Gossip messages n/a 01303634 0
>
> # nodetool ring
> Datacenter: dc1
> ==
> Address   RackStatus State   LoadOwns
>Token
>
>9189523899826545641
> xxx.xxx.xxx.24rack4   Up Normal  93.82 KiB
> 79.95%  -9194674091837769168
> xxx.xxx.xxx.107   rack1   Up Normal  2.65 TiB
>  73.25%  -9168781258594813088
> xxx.xxx.xxx.253   rack2   Up Normal  2.6 TiB
> 73.92%  -9163037340977721917
> xxx.xxx.xxx.105   rack3   Up Normal  2.65 TiB
>  72.88%  -9148860739730046229
> xxx.xxx.xxx.107   rack1   Up Normal  2.65 TiB
>  73.25%  -9125240034139323535
> xxx.xxx.xxx.253   rack2   Up Normal  2.6 TiB
> 73.92%  -9112518853051755414
> xxx.xxx.xxx.105   rack3   Up Normal  2.65 TiB
>  72.88%  -9100516173422432134
> ...
>
> This is causing a serious production issue. Please help if you can.
>
> Thanks
> David
>
>
>
>

Re: Reads not returning data after adding node

Just remove it from the seed list in the cassandra.yaml file and restart
the node.  Make sure that auto_bootstrap is set to true first though.

On Sun, Apr 2, 2023 at 9:59 PM David Tinker  wrote:

> So likely because I made it a seed node when I added it to the cluster it
> didn't do the bootstrap process. How can I recover this?
>
> On Mon, Apr 3, 2023 at 6:41 AM David Tinker 
> wrote:
>
>> Yes replication factor is 3.
>>
>> I ran nodetool repair -pr on all the nodes (one at a time) and am still
>> having issues getting data back from queries.
>>
>> I did make the new node a seed node.
>>
>> Re "rack4": I assumed that was just an indication as to the physical
>> location of the server for redundancy. This one is separate from the others
>> so I used rack4.
>>
>> On Mon, Apr 3, 2023 at 6:30 AM Carlos Diaz  wrote:
>>
>>> I'm assuming that your replication factor is 3.  If that's the case, did
>>> you intentionally put this node in rack 4?  Typically, you want to add
>>> nodes in multiples of your replication factor in order to keep the "racks"
>>> balanced.  In other words, this node should have been added to rack 1, 2 or
>>> 3.
>>>
>>> Having said that, you should be able to easily fix your problem by
>>> running a nodetool repair -pr on the new node.
>>>
>>> On Sun, Apr 2, 2023 at 8:16 PM David Tinker 
>>> wrote:
>>>
 Hi All

 I recently added a node to my 3 node Cassandra 4.0.5 cluster and now
 many reads are not returning rows! What do I need to do to fix this? There
 weren't any errors in the logs or other problems that I could see. I
 expected the cluster to balance itself but this hasn't happened (yet?). The
 nodes are similar so I have num_tokens=256 for each. I am using the
 Murmur3Partitioner.

 # nodetool status
 Datacenter: dc1
 ===
 Status=Up/Down
 |/ State=Normal/Leaving/Joining/Moving
 --  Address  Load   Tokens  Owns (effective)  Host ID
 Rack
 UN  xxx.xxx.xxx.105  2.65 TiB   256 72.9%
 afd02287-3f88-4c6f-8b27-06f7a8192402  rack3
 UN  xxx.xxx.xxx.253  2.6 TiB256 73.9%
 e1af72be-e5df-4c6b-a124-c7bc48c6602a  rack2
 UN  xxx.xxx.xxx.24   93.82 KiB  256 80.0%
 c4e8b4a0-f014-45e6-afb4-648aad4f8500  rack4
 UN  xxx.xxx.xxx.107  2.65 TiB   256 73.2%
 ab72f017-be96-41d2-9bef-a551dec2c7b5  rack1

 # nodetool netstats
 Mode: NORMAL
 Not sending any streams.
 Read Repair Statistics:
 Attempted: 0
 Mismatch (Blocking): 0
 Mismatch (Background): 0
 Pool NameActive   Pending  Completed   Dropped
 Large messages  n/a 0  71754 0
 Small messages  n/a 0839818414
 Gossip messages n/a 01303634 0

 # nodetool ring
 Datacenter: dc1
 ==
 Address   RackStatus State   LoadOwns
  Token

  9189523899826545641
 xxx.xxx.xxx.24rack4   Up Normal  93.82 KiB   79.95%
  -9194674091837769168
 xxx.xxx.xxx.107   rack1   Up Normal  2.65 TiB73.25%
  -9168781258594813088
 xxx.xxx.xxx.253   rack2   Up Normal  2.6 TiB 73.92%
  -9163037340977721917
 xxx.xxx.xxx.105   rack3   Up Normal  2.65 TiB72.88%
  -9148860739730046229
 xxx.xxx.xxx.107   rack1   Up Normal  2.65 TiB73.25%
  -9125240034139323535
 xxx.xxx.xxx.253   rack2   Up Normal  2.6 TiB 73.92%
  -9112518853051755414
 xxx.xxx.xxx.105   rack3   Up Normal  2.65 TiB72.88%
  -9100516173422432134
 ...

 This is causing a serious production issue. Please help if you can.

 Thanks
 David

Re: Reads not returning data after adding node

So likely because I made it a seed node when I added it to the cluster it
didn't do the bootstrap process. How can I recover this?

On Mon, Apr 3, 2023 at 6:41 AM David Tinker  wrote:

> Yes replication factor is 3.
>
> I ran nodetool repair -pr on all the nodes (one at a time) and am still
> having issues getting data back from queries.
>
> I did make the new node a seed node.
>
> Re "rack4": I assumed that was just an indication as to the physical
> location of the server for redundancy. This one is separate from the others
> so I used rack4.
>
> On Mon, Apr 3, 2023 at 6:30 AM Carlos Diaz  wrote:
>
>> I'm assuming that your replication factor is 3.  If that's the case, did
>> you intentionally put this node in rack 4?  Typically, you want to add
>> nodes in multiples of your replication factor in order to keep the "racks"
>> balanced.  In other words, this node should have been added to rack 1, 2 or
>> 3.
>>
>> Having said that, you should be able to easily fix your problem by
>> running a nodetool repair -pr on the new node.
>>
>> On Sun, Apr 2, 2023 at 8:16 PM David Tinker 
>> wrote:
>>
>>> Hi All
>>>
>>> I recently added a node to my 3 node Cassandra 4.0.5 cluster and now
>>> many reads are not returning rows! What do I need to do to fix this? There
>>> weren't any errors in the logs or other problems that I could see. I
>>> expected the cluster to balance itself but this hasn't happened (yet?). The
>>> nodes are similar so I have num_tokens=256 for each. I am using the
>>> Murmur3Partitioner.
>>>
>>> # nodetool status
>>> Datacenter: dc1
>>> ===
>>> Status=Up/Down
>>> |/ State=Normal/Leaving/Joining/Moving
>>> --  Address  Load   Tokens  Owns (effective)  Host ID
>>> Rack
>>> UN  xxx.xxx.xxx.105  2.65 TiB   256 72.9%
>>> afd02287-3f88-4c6f-8b27-06f7a8192402  rack3
>>> UN  xxx.xxx.xxx.253  2.6 TiB256 73.9%
>>> e1af72be-e5df-4c6b-a124-c7bc48c6602a  rack2
>>> UN  xxx.xxx.xxx.24   93.82 KiB  256 80.0%
>>> c4e8b4a0-f014-45e6-afb4-648aad4f8500  rack4
>>> UN  xxx.xxx.xxx.107  2.65 TiB   256 73.2%
>>> ab72f017-be96-41d2-9bef-a551dec2c7b5  rack1
>>>
>>> # nodetool netstats
>>> Mode: NORMAL
>>> Not sending any streams.
>>> Read Repair Statistics:
>>> Attempted: 0
>>> Mismatch (Blocking): 0
>>> Mismatch (Background): 0
>>> Pool NameActive   Pending  Completed   Dropped
>>> Large messages  n/a 0  71754 0
>>> Small messages  n/a 0839818414
>>> Gossip messages n/a 01303634 0
>>>
>>> # nodetool ring
>>> Datacenter: dc1
>>> ==
>>> Address   RackStatus State   LoadOwns
>>>  Token
>>>
>>>  9189523899826545641
>>> xxx.xxx.xxx.24rack4   Up Normal  93.82 KiB   79.95%
>>>  -9194674091837769168
>>> xxx.xxx.xxx.107   rack1   Up Normal  2.65 TiB73.25%
>>>  -9168781258594813088
>>> xxx.xxx.xxx.253   rack2   Up Normal  2.6 TiB 73.92%
>>>  -9163037340977721917
>>> xxx.xxx.xxx.105   rack3   Up Normal  2.65 TiB72.88%
>>>  -9148860739730046229
>>> xxx.xxx.xxx.107   rack1   Up Normal  2.65 TiB73.25%
>>>  -9125240034139323535
>>> xxx.xxx.xxx.253   rack2   Up Normal  2.6 TiB 73.92%
>>>  -9112518853051755414
>>> xxx.xxx.xxx.105   rack3   Up Normal  2.65 TiB72.88%
>>>  -9100516173422432134
>>> ...
>>>
>>> This is causing a serious production issue. Please help if you can.
>>>
>>> Thanks
>>> David
>>>
>>>
>>>
>>>

Re: Reads not returning data after adding node

Yes replication factor is 3.

I ran nodetool repair -pr on all the nodes (one at a time) and am still
having issues getting data back from queries.

I did make the new node a seed node.

Re "rack4": I assumed that was just an indication as to the physical
location of the server for redundancy. This one is separate from the others
so I used rack4.

On Mon, Apr 3, 2023 at 6:30 AM Carlos Diaz  wrote:

> I'm assuming that your replication factor is 3.  If that's the case, did
> you intentionally put this node in rack 4?  Typically, you want to add
> nodes in multiples of your replication factor in order to keep the "racks"
> balanced.  In other words, this node should have been added to rack 1, 2 or
> 3.
>
> Having said that, you should be able to easily fix your problem by running
> a nodetool repair -pr on the new node.
>
> On Sun, Apr 2, 2023 at 8:16 PM David Tinker 
> wrote:
>
>> Hi All
>>
>> I recently added a node to my 3 node Cassandra 4.0.5 cluster and now many
>> reads are not returning rows! What do I need to do to fix this? There
>> weren't any errors in the logs or other problems that I could see. I
>> expected the cluster to balance itself but this hasn't happened (yet?). The
>> nodes are similar so I have num_tokens=256 for each. I am using the
>> Murmur3Partitioner.
>>
>> # nodetool status
>> Datacenter: dc1
>> ===
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> --  Address  Load   Tokens  Owns (effective)  Host ID
>>   Rack
>> UN  xxx.xxx.xxx.105  2.65 TiB   256 72.9%
>> afd02287-3f88-4c6f-8b27-06f7a8192402  rack3
>> UN  xxx.xxx.xxx.253  2.6 TiB256 73.9%
>> e1af72be-e5df-4c6b-a124-c7bc48c6602a  rack2
>> UN  xxx.xxx.xxx.24   93.82 KiB  256 80.0%
>> c4e8b4a0-f014-45e6-afb4-648aad4f8500  rack4
>> UN  xxx.xxx.xxx.107  2.65 TiB   256 73.2%
>> ab72f017-be96-41d2-9bef-a551dec2c7b5  rack1
>>
>> # nodetool netstats
>> Mode: NORMAL
>> Not sending any streams.
>> Read Repair Statistics:
>> Attempted: 0
>> Mismatch (Blocking): 0
>> Mismatch (Background): 0
>> Pool NameActive   Pending  Completed   Dropped
>> Large messages  n/a 0  71754 0
>> Small messages  n/a 0839818414
>> Gossip messages n/a 01303634 0
>>
>> # nodetool ring
>> Datacenter: dc1
>> ==
>> Address   RackStatus State   LoadOwns
>>Token
>>
>>9189523899826545641
>> xxx.xxx.xxx.24rack4   Up Normal  93.82 KiB   79.95%
>>-9194674091837769168
>> xxx.xxx.xxx.107   rack1   Up Normal  2.65 TiB73.25%
>>-9168781258594813088
>> xxx.xxx.xxx.253   rack2   Up Normal  2.6 TiB 73.92%
>>-9163037340977721917
>> xxx.xxx.xxx.105   rack3   Up Normal  2.65 TiB72.88%
>>-9148860739730046229
>> xxx.xxx.xxx.107   rack1   Up Normal  2.65 TiB73.25%
>>-9125240034139323535
>> xxx.xxx.xxx.253   rack2   Up Normal  2.6 TiB 73.92%
>>-9112518853051755414
>> xxx.xxx.xxx.105   rack3   Up Normal  2.65 TiB72.88%
>>-9100516173422432134
>> ...
>>
>> This is causing a serious production issue. Please help if you can.
>>
>> Thanks
>> David
>>
>>
>>
>>

Re: Reads not returning data after adding node