Oh! Excellent! Doh! That was it. So when we add a new system, we use
puppet to push things out...like NTP...well this is our first Rocky
Linux install and guess what I didn't do?
Thank you Song. The new machine is now joining the cluster.
nodetool status
Datacenter: datacenter1
===
Hello Joe,
These logs indicate the clocks are out of sync (by over 4.2 hours)
between the new node and the seed nodes:
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,567
MessagingMetrics.java:206 - GOSSIP_DIGEST_SYN messages were dropped
in last 5000 ms: 0 internal and 1 cross node. Me
Hmm. are the ports open on the `new` server?
Looks like it can connect to other nodes but other nodes can't connect to
it.
-Vy
On Fri, Sep 10, 2021 at 10:20 AM Joe Obernberger <
joseph.obernber...@gmail.com> wrote:
> Good idea.
> There are two seed nodes:
> I see this on one (note 172.16.100.44
Good idea.
There are two seed nodes:
I see this on one (note 172.16.100.44 is the new node):
DEBUG [CompactionExecutor:1345] 2021-09-10 11:13:49,569
TimeWindowCompactionStrategy.java:129 - TWCS skipping check for fully
expired SSTables
INFO [Messaging-EventLoop-3-10] 2021-09-10 11:14:22,810
I
Can you drop as much info as possible into a JIRA?
Include the output of `nodetool gossipinfo` if at all possible
On Fri, Sep 10, 2021 at 7:58 AM Joe Obernberger <
joseph.obernber...@gmail.com> wrote:
> Thank you Jeff - yes, this is on the latest 4.0.1
>
> nodetool version
> ReleaseVersion: 4.
Thank you Jeff - yes, this is on the latest 4.0.1
nodetool version
ReleaseVersion: 4.0.1
nodetool status
Datacenter: datacenter1
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host
ID
Is this on 4.0.0 ? 4.0.1 fixes an issue where the gossip result is too
large for the urgent message queue, causing this stack trace, and was
released 3 days ago. I've never seen it on a 10 node cluster before, but
I'd be trying that.
On Fri, Sep 10, 2021 at 7:50 AM Joe Obernberger <
joseph.obernbe
I have a 10 node cluster and am trying to add another node. The new
node is running Rocky Linux and I'm getting the unable to gossip with
any peers error. Firewall and SELinux are off. I can ping all the
other nodes OK. I've checked everything I can think of (/etc/hosts,
listen_address, bro
