Should I use assassinate or removenode? Given that there is some data on the node. Or will that be found on the other nodes? Sorry for all the questions but I really don't want to mess up.
On Mon, Apr 3, 2023 at 7:21 AM Carlos Diaz <crdiaz...@gmail.com> wrote: > That's what nodetool assassinte will do. > > On Sun, Apr 2, 2023 at 10:19 PM David Tinker <david.tin...@gmail.com> > wrote: > >> Is it possible for me to remove the node from the cluster i.e. to undo >> this mess and get the cluster operating again? >> >> On Mon, Apr 3, 2023 at 7:13 AM Carlos Diaz <crdiaz...@gmail.com> wrote: >> >>> You can leave it in the seed list of the other nodes, just make sure >>> it's not included in this node's seed list. However, if you do decide to >>> fix the issue with the racks first assassinate this node (nodetool >>> assassinate <ip>), and update the rack name before you restart. >>> >>> On Sun, Apr 2, 2023 at 10:06 PM David Tinker <david.tin...@gmail.com> >>> wrote: >>> >>>> It is also in the seeds list for the other nodes. Should I remove it >>>> from those, restart them one at a time, then restart it? >>>> >>>> /etc/cassandra # grep -i bootstrap * >>>> doesn't show anything so I don't think I have auto_bootstrap false. >>>> >>>> Thanks very much for the help. >>>> >>>> >>>> On Mon, Apr 3, 2023 at 7:01 AM Carlos Diaz <crdiaz...@gmail.com> wrote: >>>> >>>>> Just remove it from the seed list in the cassandra.yaml file and >>>>> restart the node. Make sure that auto_bootstrap is set to true first >>>>> though. >>>>> >>>>> On Sun, Apr 2, 2023 at 9:59 PM David Tinker <david.tin...@gmail.com> >>>>> wrote: >>>>> >>>>>> So likely because I made it a seed node when I added it to the >>>>>> cluster it didn't do the bootstrap process. How can I recover this? >>>>>> >>>>>> On Mon, Apr 3, 2023 at 6:41 AM David Tinker <david.tin...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Yes replication factor is 3. >>>>>>> >>>>>>> I ran nodetool repair -pr on all the nodes (one at a time) and am >>>>>>> still having issues getting data back from queries. >>>>>>> >>>>>>> I did make the new node a seed node. >>>>>>> >>>>>>> Re "rack4": I assumed that was just an indication as to the physical >>>>>>> location of the server for redundancy. This one is separate from the >>>>>>> others >>>>>>> so I used rack4. >>>>>>> >>>>>>> On Mon, Apr 3, 2023 at 6:30 AM Carlos Diaz <crdiaz...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> I'm assuming that your replication factor is 3. If that's the >>>>>>>> case, did you intentionally put this node in rack 4? Typically, you >>>>>>>> want >>>>>>>> to add nodes in multiples of your replication factor in order to keep >>>>>>>> the >>>>>>>> "racks" balanced. In other words, this node should have been added to >>>>>>>> rack >>>>>>>> 1, 2 or 3. >>>>>>>> >>>>>>>> Having said that, you should be able to easily fix your problem by >>>>>>>> running a nodetool repair -pr on the new node. >>>>>>>> >>>>>>>> On Sun, Apr 2, 2023 at 8:16 PM David Tinker <david.tin...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi All >>>>>>>>> >>>>>>>>> I recently added a node to my 3 node Cassandra 4.0.5 cluster and >>>>>>>>> now many reads are not returning rows! What do I need to do to fix >>>>>>>>> this? >>>>>>>>> There weren't any errors in the logs or other problems that I could >>>>>>>>> see. I >>>>>>>>> expected the cluster to balance itself but this hasn't happened >>>>>>>>> (yet?). The >>>>>>>>> nodes are similar so I have num_tokens=256 for each. I am using the >>>>>>>>> Murmur3Partitioner. >>>>>>>>> >>>>>>>>> # nodetool status >>>>>>>>> Datacenter: dc1 >>>>>>>>> =============== >>>>>>>>> Status=Up/Down >>>>>>>>> |/ State=Normal/Leaving/Joining/Moving >>>>>>>>> -- Address Load Tokens Owns (effective) Host ID >>>>>>>>> Rack >>>>>>>>> UN xxx.xxx.xxx.105 2.65 TiB 256 72.9% >>>>>>>>> afd02287-3f88-4c6f-8b27-06f7a8192402 rack3 >>>>>>>>> UN xxx.xxx.xxx.253 2.6 TiB 256 73.9% >>>>>>>>> e1af72be-e5df-4c6b-a124-c7bc48c6602a rack2 >>>>>>>>> UN xxx.xxx.xxx.24 93.82 KiB 256 80.0% >>>>>>>>> c4e8b4a0-f014-45e6-afb4-648aad4f8500 rack4 >>>>>>>>> UN xxx.xxx.xxx.107 2.65 TiB 256 73.2% >>>>>>>>> ab72f017-be96-41d2-9bef-a551dec2c7b5 rack1 >>>>>>>>> >>>>>>>>> # nodetool netstats >>>>>>>>> Mode: NORMAL >>>>>>>>> Not sending any streams. >>>>>>>>> Read Repair Statistics: >>>>>>>>> Attempted: 0 >>>>>>>>> Mismatch (Blocking): 0 >>>>>>>>> Mismatch (Background): 0 >>>>>>>>> Pool Name Active Pending Completed >>>>>>>>> Dropped >>>>>>>>> Large messages n/a 0 71754 >>>>>>>>> 0 >>>>>>>>> Small messages n/a 0 8398184 >>>>>>>>> 14 >>>>>>>>> Gossip messages n/a 0 1303634 >>>>>>>>> 0 >>>>>>>>> >>>>>>>>> # nodetool ring >>>>>>>>> Datacenter: dc1 >>>>>>>>> ========== >>>>>>>>> Address Rack Status State Load >>>>>>>>> Owns Token >>>>>>>>> >>>>>>>>> 9189523899826545641 >>>>>>>>> xxx.xxx.xxx.24 rack4 Up Normal 93.82 KiB >>>>>>>>> 79.95% -9194674091837769168 >>>>>>>>> xxx.xxx.xxx.107 rack1 Up Normal 2.65 TiB >>>>>>>>> 73.25% -9168781258594813088 >>>>>>>>> xxx.xxx.xxx.253 rack2 Up Normal 2.6 TiB >>>>>>>>> 73.92% -9163037340977721917 >>>>>>>>> xxx.xxx.xxx.105 rack3 Up Normal 2.65 TiB >>>>>>>>> 72.88% -9148860739730046229 >>>>>>>>> xxx.xxx.xxx.107 rack1 Up Normal 2.65 TiB >>>>>>>>> 73.25% -9125240034139323535 >>>>>>>>> xxx.xxx.xxx.253 rack2 Up Normal 2.6 TiB >>>>>>>>> 73.92% -9112518853051755414 >>>>>>>>> xxx.xxx.xxx.105 rack3 Up Normal 2.65 TiB >>>>>>>>> 72.88% -9100516173422432134 >>>>>>>>> ... >>>>>>>>> >>>>>>>>> This is causing a serious production issue. Please help if you can. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> David >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>
