Can you drop as much info as possible into a JIRA? Include the output of `nodetool gossipinfo` if at all possible
On Fri, Sep 10, 2021 at 7:58 AM Joe Obernberger < joseph.obernber...@gmail.com> wrote: > Thank you Jeff - yes, this is on the latest 4.0.1 > > nodetool version > ReleaseVersion: 4.0.1 > nodetool status > Datacenter: datacenter1 > ======================= > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- Address Load Tokens Owns (effective) Host > ID Rack > UN 172.16.100.251 488.38 GiB 200 38.7% > 660f476c-a124-4ca0-b55f-75efe56370da rack1 > UN 172.16.100.208 76.02 GiB 30 5.8% > 2529b6ed-cdb2-43c2-bdd7-171cfe308bd3 rack1 > UN 172.16.100.252 501.88 GiB 200 38.6% > e83aa851-69b4-478f-88f6-60e657ea6539 rack1 > UN 172.16.100.249 517.27 GiB 200 38.6% > 49e4f571-7d1c-4e1e-aca7-5bbe076596f7 rack1 > UN 172.16.100.36 524.45 GiB 200 38.6% > d9702f96-256e-45ae-8e12-69a42712be50 rack1 > UN 172.16.100.39 521.05 GiB 200 38.6% > 93f9cb0f-ea71-4e3d-b62a-f0ea0e888c47 rack1 > UN 172.16.100.253 11.39 GiB 4 0.8% > a1a16910-9167-4174-b34b-eb859d36347e rack1 > UN 172.16.100.248 524.46 GiB 200 38.7% > 4bbbe57c-6219-41e5-bbac-de92a9594d53 rack1 > UN 172.16.100.37 314.67 GiB 120 23.2% > 08a19658-40be-4e55-8709-812b3d4ac750 rack1 > UN 172.16.100.250 464.23 GiB 200 38.6% > b74b6e65-af63-486a-b07f-9e304ec30a39 rack1 > > yum list installed | grep cass > cassandra.noarch 4.0.1-1 > @cassandra > > -Joe > On 9/10/2021 10:54 AM, Jeff Jirsa wrote: > > Is this on 4.0.0 ? 4.0.1 fixes an issue where the gossip result is too > large for the urgent message queue, causing this stack trace, and was > released 3 days ago. I've never seen it on a 10 node cluster before, but > I'd be trying that. > > On Fri, Sep 10, 2021 at 7:50 AM Joe Obernberger < > joseph.obernber...@gmail.com> wrote: > >> I have a 10 node cluster and am trying to add another node. The new >> node is running Rocky Linux and I'm getting the unable to gossip with >> any peers error. Firewall and SELinux are off. I can ping all the >> other nodes OK. I've checked everything I can think of (/etc/hosts, >> listen_address, broadcast etc..). It all looks correct to me. >> Any ideas? Could it be an incompatibility with Rocky? >> >> DEBUG [main] 2021-09-10 06:45:24,846 YamlConfigurationLoader.java:112 - >> Loading settings from file:/etc/cassandra/default.conf/cassandra.yaml >> INFO [Messaging-EventLoop-3-6] 2021-09-10 06:45:24,921 >> OutboundConnection.java:1150 - >> /172.16.100.44:7000(/172.16.100.44:45934)->/172.16.100.253:7000-URGENT_MESSAGES-90efbb9e >> >> successfully connected, version = 12, framing = LZ4, encryption = >> unencrypted >> INFO [Messaging-EventLoop-3-3] 2021-09-10 06:45:24,930 >> OutboundConnection.java:1150 - >> /172.16.100.44:7000(/172.16.100.44:44320)->/172.16.100.37:7000-URGENT_MESSAGES-eae47864 >> >> successfully connected, version = 12, framing = LZ4, encryption = >> unencrypted >> INFO [ScheduledTasks:1] 2021-09-10 06:45:27,648 TokenMetadata.java:525 >> - Updating topology for all endpoints that have changed >> DEBUG [OptionalTasks:1] 2021-09-10 06:45:54,644 >> SizeEstimatesRecorder.java:65 - Node is not part of the ring; not >> recording size estimates >> ERROR [main] 2021-09-10 06:46:25,891 CassandraDaemon.java:909 - >> Exception encountered during startup >> java.lang.RuntimeException: Unable to gossip with any peers >> at >> org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1805) >> at >> >> org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:648) >> at >> >> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:934) >> at >> >> org.apache.cassandra.service.StorageService.initServer(StorageService.java:784) >> at >> >> org.apache.cassandra.service.StorageService.initServer(StorageService.java:729) >> at >> >> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:420) >> at >> >> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:763) >> at >> >> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:887) >> DEBUG [StorageServiceShutdownHook] 2021-09-10 06:46:25,896 >> StorageService.java:1621 - DRAINING: starting drain process >> INFO [StorageServiceShutdownHook] 2021-09-10 06:46:25,898 >> HintsService.java:220 - Paused hints dispatch >> WARN [StorageServiceShutdownHook] 2021-09-10 06:46:25,899 >> Gossiper.java:1993 - No local state, state is in silent shutdown, or >> node hasn't joined, not announcing shutdown >> >> Thank you! >> >> -Joe >> >> > > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> > Virus-free. > www.avg.com > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> > <#m_-4486529990198249803_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> > >
