Looks like it joined with no data. Did you set auto_bootstrap to false? Or does the node think it’s a seed?
You want to use “nodetool rebuild” to stream data to that host. You can potentially end the production outage / incident by taking the host offline, or making it less likely to be queried (disable binary on that host and if you know how, use jmx to set severity to an arbitrarily high number) > On Apr 2, 2023, at 8:16 PM, David Tinker <david.tin...@gmail.com> wrote: > > > Hi All > > I recently added a node to my 3 node Cassandra 4.0.5 cluster and now many > reads are not returning rows! What do I need to do to fix this? There weren't > any errors in the logs or other problems that I could see. I expected the > cluster to balance itself but this hasn't happened (yet?). The nodes are > similar so I have num_tokens=256 for each. I am using the Murmur3Partitioner. > > # nodetool status > Datacenter: dc1 > =============== > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- Address Load Tokens Owns (effective) Host ID > Rack > UN xxx.xxx.xxx.105 2.65 TiB 256 72.9% > afd02287-3f88-4c6f-8b27-06f7a8192402 rack3 > UN xxx.xxx.xxx.253 2.6 TiB 256 73.9% > e1af72be-e5df-4c6b-a124-c7bc48c6602a rack2 > UN xxx.xxx.xxx.24 93.82 KiB 256 80.0% > c4e8b4a0-f014-45e6-afb4-648aad4f8500 rack4 > UN xxx.xxx.xxx.107 2.65 TiB 256 73.2% > ab72f017-be96-41d2-9bef-a551dec2c7b5 rack1 > > # nodetool netstats > Mode: NORMAL > Not sending any streams. > Read Repair Statistics: > Attempted: 0 > Mismatch (Blocking): 0 > Mismatch (Background): 0 > Pool Name Active Pending Completed Dropped > Large messages n/a 0 71754 0 > Small messages n/a 0 8398184 14 > Gossip messages n/a 0 1303634 0 > > # nodetool ring > Datacenter: dc1 > ========== > Address Rack Status State Load Owns > Token > > 9189523899826545641 > xxx.xxx.xxx.24 rack4 Up Normal 93.82 KiB 79.95% > -9194674091837769168 > xxx.xxx.xxx.107 rack1 Up Normal 2.65 TiB 73.25% > -9168781258594813088 > xxx.xxx.xxx.253 rack2 Up Normal 2.6 TiB 73.92% > -9163037340977721917 > xxx.xxx.xxx.105 rack3 Up Normal 2.65 TiB 72.88% > -9148860739730046229 > xxx.xxx.xxx.107 rack1 Up Normal 2.65 TiB 73.25% > -9125240034139323535 > xxx.xxx.xxx.253 rack2 Up Normal 2.6 TiB 73.92% > -9112518853051755414 > xxx.xxx.xxx.105 rack3 Up Normal 2.65 TiB 72.88% > -9100516173422432134 > ... > > This is causing a serious production issue. Please help if you can. > > Thanks > David > > >
