Thank you Jeff - yes, this is on the latest 4.0.1
nodetool version
ReleaseVersion: 4.0.1
nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host
ID Rack
UN 172.16.100.251 488.38 GiB 200 38.7%
660f476c-a124-4ca0-b55f-75efe56370da rack1
UN 172.16.100.208 76.02 GiB 30 5.8%
2529b6ed-cdb2-43c2-bdd7-171cfe308bd3 rack1
UN 172.16.100.252 501.88 GiB 200 38.6%
e83aa851-69b4-478f-88f6-60e657ea6539 rack1
UN 172.16.100.249 517.27 GiB 200 38.6%
49e4f571-7d1c-4e1e-aca7-5bbe076596f7 rack1
UN 172.16.100.36 524.45 GiB 200 38.6%
d9702f96-256e-45ae-8e12-69a42712be50 rack1
UN 172.16.100.39 521.05 GiB 200 38.6%
93f9cb0f-ea71-4e3d-b62a-f0ea0e888c47 rack1
UN 172.16.100.253 11.39 GiB 4 0.8%
a1a16910-9167-4174-b34b-eb859d36347e rack1
UN 172.16.100.248 524.46 GiB 200 38.7%
4bbbe57c-6219-41e5-bbac-de92a9594d53 rack1
UN 172.16.100.37 314.67 GiB 120 23.2%
08a19658-40be-4e55-8709-812b3d4ac750 rack1
UN 172.16.100.250 464.23 GiB 200 38.6%
b74b6e65-af63-486a-b07f-9e304ec30a39 rack1
yum list installed | grep cass
cassandra.noarch 4.0.1-1 @cassandra
-Joe
On 9/10/2021 10:54 AM, Jeff Jirsa wrote:
Is this on 4.0.0 ? 4.0.1 fixes an issue where the gossip result is too
large for the urgent message queue, causing this stack trace, and was
released 3 days ago. I've never seen it on a 10 node cluster before,
but I'd be trying that.
On Fri, Sep 10, 2021 at 7:50 AM Joe Obernberger
<joseph.obernber...@gmail.com> wrote:
I have a 10 node cluster and am trying to add another node. The new
node is running Rocky Linux and I'm getting the unable to gossip with
any peers error. Firewall and SELinux are off. I can ping all the
other nodes OK. I've checked everything I can think of (/etc/hosts,
listen_address, broadcast etc..). It all looks correct to me.
Any ideas? Could it be an incompatibility with Rocky?
DEBUG [main] 2021-09-10 06:45:24,846
YamlConfigurationLoader.java:112 -
Loading settings from file:/etc/cassandra/default.conf/cassandra.yaml
INFO [Messaging-EventLoop-3-6] 2021-09-10 06:45:24,921
OutboundConnection.java:1150 -
/172.16.100.44:7000(/172.16.100.44:45934)->/172.16.100.
<http://172.16.100.>253:7000-URGENT_MESSAGES-90efbb9e
successfully connected, version = 12, framing = LZ4, encryption =
unencrypted
INFO [Messaging-EventLoop-3-3] 2021-09-10 06:45:24,930
OutboundConnection.java:1150 -
/172.16.100.44:7000(/172.16.100.44:44320)->/172.16.100.37
<http://172.16.100.37>:7000-URGENT_MESSAGES-eae47864
successfully connected, version = 12, framing = LZ4, encryption =
unencrypted
INFO [ScheduledTasks:1] 2021-09-10 06:45:27,648
TokenMetadata.java:525
- Updating topology for all endpoints that have changed
DEBUG [OptionalTasks:1] 2021-09-10 06:45:54,644
SizeEstimatesRecorder.java:65 - Node is not part of the ring; not
recording size estimates
ERROR [main] 2021-09-10 06:46:25,891 CassandraDaemon.java:909 -
Exception encountered during startup
java.lang.RuntimeException: Unable to gossip with any peers
at
org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1805)
at
org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:648)
at
org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:934)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:784)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:729)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:420)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:763)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:887)
DEBUG [StorageServiceShutdownHook] 2021-09-10 06:46:25,896
StorageService.java:1621 - DRAINING: starting drain process
INFO [StorageServiceShutdownHook] 2021-09-10 06:46:25,898
HintsService.java:220 - Paused hints dispatch
WARN [StorageServiceShutdownHook] 2021-09-10 06:46:25,899
Gossiper.java:1993 - No local state, state is in silent shutdown, or
node hasn't joined, not announcing shutdown
Thank you!
-Joe
<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
Virus-free. www.avg.com
<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
