Re: Issue replacing a dead node

2025-05-27 Thread Courtney

One last update:

After kicking it more, it finally fully joined the cluster. The third 
time the server was rebooted and after that it eventually reached the UN 
state. I wish I had kept the link, but I had read that someone had a 
similar issue joining a node to a cluster with 4.1.x and the answer was 
to restart the service.


On 5/23/25 10:46 PM, Courtney wrote:
Some updates after getting back to this. I did hardware tests and 
could not find any hardware issues. Instead of trying a replace, I 
went the route of removing the dead node entirely and then adding in a 
new node.


The new node is still joining, but I am hitting some oddities in the 
log. When joining, the process seems to "rescheduling" around the time 
that these logs appear:


INFO  [Messaging-EventLoop-3-25] 2025-05-24 05:20:38,203 
NoSpamLogger.java:105 - 
/:7000->/:7000-SMALL_MESSAGES-[no-channel] failed 
to connect
io.netty.channel.ConnectTimeoutException: connection timed out: 
/:7000
WARN  [Messaging-EventLoop-3-25] 2025-05-24 05:20:46,200 
NoSpamLogger.java:108 - 
/:7000->/:7000-SMALL_MESSAGES-[no-channel] 
dropping message of type PING_REQ whose timeout expired before 
reaching the network
INFO  [Messaging-EventLoop-3-26] 2025-05-24 05:32:27,606 
NoSpamLogger.java:105 - 
/:7000->/:7000-LARGE_MESSAGES-[no-channel] failed 
to connect
io.netty.channel.ConnectTimeoutException: connection timed out: 
/:7000
INFO  [GossipStage:1] 2025-05-24 05:33:15,538 Gossiper.java:1428 - 
InetAddress /:7000 is now DOWN


The cluster knows 0 about the old node, the old node is completely 
powered off. I don't know why it is attempting to connect to the dead 
node that has no presence in the cluster.


Beforehand, I start to get these messages:

INFO  [OptionalTasks:1] 2025-05-24 05:27:46,189 NoSpamLogger.java:105 
- "Cannot read from a bootstrapping node" while executing SELECT * 
FROM system_auth.roles WHERE role = 'cassandra' ALLOW FILTERING
WARN  [OptionalTasks:1] 2025-05-24 05:32:16,240 
CassandraRoleManager.java:359 - CassandraRoleManager skipped default 
role setup: some nodes were not ready
INFO  [OptionalTasks:1] 2025-05-24 05:32:16,240 
CassandraRoleManager.java:395 - Setup task failed with error, 
rescheduling


I've had to stop and start cassandra to get passed this, but I am 
afraid I will hit this again soon.


On 5/16/25 11:54 PM, Sebastian Marsching wrote:
To add on to what Bowen already wrote, if you cannot find any reason 
in the logs at all, I would retry using different hardware.


In the recent past I have seen two cases where strange Cassandra 
problems were actually caused by broken hardware (in both cases, a 
faulty memory module caused the issues). In one case, there were log 
messages, but misleading ones about SSTable corruption (the SSTables 
were fine, but when loaded into memory, the data got corrupted). In 
the other case, there were no log messages at all. The Cassandra 
process simply stopped without a good reason. Eventually I found the 
crash dumps in the Cassandra data directory (whether they are written 
depends on the JVM setting), which indicated that the JVM experienced 
a segmentation fault.


So, if you have any spare hardware lying around (it’s best if it is 
from a different batch to exclude common-mode failures), using a 
different piece of hardware and trying with that one might make 
sense. If the problem stays, you can at least be sure that it isn’t 
related to the hardware, and if it vanishes, you can further inspect 
the original hardware.



Am 17.05.2025 um 04:27 schrieb Bowen Song via user 
:
In my experience, failed bootstrap / node replacement always leave 
some traces in the logs. At the very minimal, there's going to be 
logs about streaming sessions failing or aborting. I have never seen 
it silently fails or stops without leaving any traces in the log. I 
can't think of anything that can cause the process to fail and 
doesn't leave a trace in the log. BTW, the relevant logs can be 
hours before the symptom becomes visible, because a failed streaming 
session does not cause Cassandra to immediately abort other active 
streaming sessions, and the remaining active sessions can take a 
while to complete.


If the process repeatedly fails at a certain place, I would suspect 
some sort of data corruption or disk error, resulting in the data 
cannot be read or deserialised correctly. But this is just a guess, 
and I could be wrong.


On 16/05/2025 01:14, Courtney wrote:
I checked all the logs and really couldn't find anything. I 
couldn't find any sort of errors in dmesg, system.log, debug.log, 
gc.log (maybe up the log level?), systemd journal...the logs are 
totally clean. It just stops gossiping all of a sudden at 22GB of 
data each time, then the old node returning to DN state. What is 
`nodetool bootstrap resume` going to do? Is there a risk to running 
resume when the replacement node is no longer in the cluster? Could 
too high of a tombstone ratio cause this?


On 5/15/25 5:08 PM, Bowen 

Re: Issue replacing a dead node

2025-05-23 Thread Courtney
Some updates after getting back to this. I did hardware tests and could 
not find any hardware issues. Instead of trying a replace, I went the 
route of removing the dead node entirely and then adding in a new node.


The new node is still joining, but I am hitting some oddities in the 
log. When joining, the process seems to "rescheduling" around the time 
that these logs appear:


INFO  [Messaging-EventLoop-3-25] 2025-05-24 05:20:38,203 
NoSpamLogger.java:105 - 
/:7000->/:7000-SMALL_MESSAGES-[no-channel] failed 
to connect
io.netty.channel.ConnectTimeoutException: connection timed out: 
/:7000
WARN  [Messaging-EventLoop-3-25] 2025-05-24 05:20:46,200 
NoSpamLogger.java:108 - 
/:7000->/:7000-SMALL_MESSAGES-[no-channel] dropping 
message of type PING_REQ whose timeout expired before reaching the network
INFO  [Messaging-EventLoop-3-26] 2025-05-24 05:32:27,606 
NoSpamLogger.java:105 - 
/:7000->/:7000-LARGE_MESSAGES-[no-channel] failed 
to connect
io.netty.channel.ConnectTimeoutException: connection timed out: 
/:7000
INFO  [GossipStage:1] 2025-05-24 05:33:15,538 Gossiper.java:1428 - 
InetAddress /:7000 is now DOWN


The cluster knows 0 about the old node, the old node is completely 
powered off. I don't know why it is attempting to connect to the dead 
node that has no presence in the cluster.


Beforehand, I start to get these messages:

INFO  [OptionalTasks:1] 2025-05-24 05:27:46,189 NoSpamLogger.java:105 - 
"Cannot read from a bootstrapping node" while executing SELECT * FROM 
system_auth.roles WHERE role = 'cassandra' ALLOW FILTERING
WARN  [OptionalTasks:1] 2025-05-24 05:32:16,240 
CassandraRoleManager.java:359 - CassandraRoleManager skipped default 
role setup: some nodes were not ready
INFO  [OptionalTasks:1] 2025-05-24 05:32:16,240 
CassandraRoleManager.java:395 - Setup task failed with error, rescheduling


I've had to stop and start cassandra to get passed this, but I am afraid 
I will hit this again soon.


On 5/16/25 11:54 PM, Sebastian Marsching wrote:

To add on to what Bowen already wrote, if you cannot find any reason in the 
logs at all, I would retry using different hardware.

In the recent past I have seen two cases where strange Cassandra problems were 
actually caused by broken hardware (in both cases, a faulty memory module 
caused the issues). In one case, there were log messages, but misleading ones 
about SSTable corruption (the SSTables were fine, but when loaded into memory, 
the data got corrupted). In the other case, there were no log messages at all. 
The Cassandra process simply stopped without a good reason. Eventually I found 
the crash dumps in the Cassandra data directory (whether they are written 
depends on the JVM setting), which indicated that the JVM experienced a 
segmentation fault.

So, if you have any spare hardware lying around (it’s best if it is from a 
different batch to exclude common-mode failures), using a different piece of 
hardware and trying with that one might make sense. If the problem stays, you 
can at least be sure that it isn’t related to the hardware, and if it vanishes, 
you can further inspect the original hardware.


Am 17.05.2025 um 04:27 schrieb Bowen Song via user :

In my experience, failed bootstrap / node replacement always leave some traces 
in the logs. At the very minimal, there's going to be logs about streaming 
sessions failing or aborting. I have never seen it silently fails or stops 
without leaving any traces in the log. I can't think of anything that can cause 
the process to fail and doesn't leave a trace in the log. BTW, the relevant 
logs can be hours before the symptom becomes visible, because a failed 
streaming session does not cause Cassandra to immediately abort other active 
streaming sessions, and the remaining active sessions can take a while to 
complete.

If the process repeatedly fails at a certain place, I would suspect some sort 
of data corruption or disk error, resulting in the data cannot be read or 
deserialised correctly. But this is just a guess, and I could be wrong.

On 16/05/2025 01:14, Courtney wrote:

I checked all the logs and really couldn't find anything. I couldn't find any 
sort of errors in dmesg, system.log, debug.log, gc.log (maybe up the log 
level?), systemd journal...the logs are totally clean. It just stops gossiping 
all of a sudden at 22GB of data each time, then the old node returning to DN 
state. What is `nodetool bootstrap resume` going to do? Is there a risk to 
running resume when the replacement node is no longer in the cluster? Could too 
high of a tombstone ratio cause this?

On 5/15/25 5:08 PM, Bowen Song via user wrote:

The dead node being replaced went back to DN state indicating the new 
replacement node failed to join the cluster, usually because the streaming was 
interrupted (e.g. by network issues, or long STW GC pauses). I would start 
looking for red flags in the logs, including Cassandra's logs, GC logs, dmesg, 
systemd journal, etc., on the new node, and other nodes 

Re: Issue replacing a dead node

2025-05-16 Thread Sebastian Marsching

To add on to what Bowen already wrote, if you cannot find any reason in the 
logs at all, I would retry using different hardware.

In the recent past I have seen two cases where strange Cassandra problems were 
actually caused by broken hardware (in both cases, a faulty memory module 
caused the issues). In one case, there were log messages, but misleading ones 
about SSTable corruption (the SSTables were fine, but when loaded into memory, 
the data got corrupted). In the other case, there were no log messages at all. 
The Cassandra process simply stopped without a good reason. Eventually I found 
the crash dumps in the Cassandra data directory (whether they are written 
depends on the JVM setting), which indicated that the JVM experienced a 
segmentation fault.

So, if you have any spare hardware lying around (it’s best if it is from a 
different batch to exclude common-mode failures), using a different piece of 
hardware and trying with that one might make sense. If the problem stays, you 
can at least be sure that it isn’t related to the hardware, and if it vanishes, 
you can further inspect the original hardware.


Am 17.05.2025 um 04:27 schrieb Bowen Song via user :
>
> In my experience, failed bootstrap / node replacement always leave some 
> traces in the logs. At the very minimal, there's going to be logs about 
> streaming sessions failing or aborting. I have never seen it silently fails 
> or stops without leaving any traces in the log. I can't think of anything 
> that can cause the process to fail and doesn't leave a trace in the log. BTW, 
> the relevant logs can be hours before the symptom becomes visible, because a 
> failed streaming session does not cause Cassandra to immediately abort other 
> active streaming sessions, and the remaining active sessions can take a while 
> to complete.
>
> If the process repeatedly fails at a certain place, I would suspect some sort 
> of data corruption or disk error, resulting in the data cannot be read or 
> deserialised correctly. But this is just a guess, and I could be wrong.
>
> On 16/05/2025 01:14, Courtney wrote:
>> I checked all the logs and really couldn't find anything. I couldn't find 
>> any sort of errors in dmesg, system.log, debug.log, gc.log (maybe up the log 
>> level?), systemd journal...the logs are totally clean. It just stops 
>> gossiping all of a sudden at 22GB of data each time, then the old node 
>> returning to DN state. What is `nodetool bootstrap resume` going to do? Is 
>> there a risk to running resume when the replacement node is no longer in the 
>> cluster? Could too high of a tombstone ratio cause this?
>>
>> On 5/15/25 5:08 PM, Bowen Song via user wrote:
>>> The dead node being replaced went back to DN state indicating the new 
>>> replacement node failed to join the cluster, usually because the streaming 
>>> was interrupted (e.g. by network issues, or long STW GC pauses). I would 
>>> start looking for red flags in the logs, including Cassandra's logs, GC 
>>> logs, dmesg, systemd journal, etc., on the new node, and other nodes in the 
>>> cluster too. Also, I would try `nodetool bootstrap resume` on the 
>>> replacement node.
>>>
>>>
>>> On 12/05/2025 09:53, Courtney wrote:
 Hello everyone,

 I have a cluster with 2 datacenters. I am using 
 GossipingPropertyFileSnitch as my endpoint snitch. Cassandra version 
 4.1.8. One datacenter is fully Ubuntu 24.04 and OpenJDK 11 and another is 
 Ubuntu 20.04 on OpenJDK 8. A seed node died in my second DC running Ubuntu 
 20.04 hosts. I ordered a new dedicated server. I updated my seeds to 
 forget the dead seed node. I did the steps to replace a dead node

 JVM_OPTS="$JVM_OPTS $JVM_EXTRA_OPTS 
 -Dcassandra.replace_address_first_boot="

 Configs between the old/new node are identical minus IP addresses and that 
 line above in the env file to replace the dead node. I started the node 
 and it started replacing the old node and was in the `UJ` state. Not long 
 into the process, the new node stops processing data and the cluster 
 forgets the new node and remembers the old one in its `DN` state (which is 
 turned off, no power). There are no errors in the logs. I've tried 
 different times hoping to solve the issue. I upped my ROOT logging level 
 to DEBUG, I also set "org.apache.cassandra.gms.Gossiper TRACE". No errors.

 With TRACE set for the Gossiper, I notice gossiping stops and data 
 stopping streaming about the same time. I cannot run any nodetool commands 
 on the new node. The process doesn't die, it leaves open connections to 
 nodes that are streaming data, but I don't see any data streaming.

 I've thought through a lot. Space isn't an issue, ulimits are set high in 
 /etc/security/limits.conf. Checking /proc//limits shows the values 
 are high. I've replaced nodes before like this without issue, but this one 
 is causing me grief. Is there 

Re: Issue replacing a dead node

2025-05-16 Thread Courtney
Is it bad to leave the replacement node up and running for hours even 
when the cluster forgets it for the old node being replaced? I'll have 
to set the logging to trace. debug produced nothing. I did stop the 
service, which produced errors in the other nodes in the datacenter 
since they had open connections to the server still. The hardware is 
new. A disk issue already would be odd, but perhaps not unsurprising if 
that was the case.


On 5/16/25 7:27 PM, Bowen Song via user wrote:
In my experience, failed bootstrap / node replacement always leave 
some traces in the logs. At the very minimal, there's going to be logs 
about streaming sessions failing or aborting. I have never seen it 
silently fails or stops without leaving any traces in the log. I can't 
think of anything that can cause the process to fail and doesn't leave 
a trace in the log. BTW, the relevant logs can be hours before the 
symptom becomes visible, because a failed streaming session does not 
cause Cassandra to immediately abort other active streaming sessions, 
and the remaining active sessions can take a while to complete.


If the process repeatedly fails at a certain place, I would suspect 
some sort of data corruption or disk error, resulting in the data 
cannot be read or deserialised correctly. But this is just a guess, 
and I could be wrong.


On 16/05/2025 01:14, Courtney wrote:
I checked all the logs and really couldn't find anything. I couldn't 
find any sort of errors in dmesg, system.log, debug.log, gc.log 
(maybe up the log level?), systemd journal...the logs are totally 
clean. It just stops gossiping all of a sudden at 22GB of data each 
time, then the old node returning to DN state. What is `nodetool 
bootstrap resume` going to do? Is there a risk to running resume when 
the replacement node is no longer in the cluster? Could too high of a 
tombstone ratio cause this?


On 5/15/25 5:08 PM, Bowen Song via user wrote:
The dead node being replaced went back to DN state indicating the 
new replacement node failed to join the cluster, usually because the 
streaming was interrupted (e.g. by network issues, or long STW GC 
pauses). I would start looking for red flags in the logs, including 
Cassandra's logs, GC logs, dmesg, systemd journal, etc., on the new 
node, and other nodes in the cluster too. Also, I would try 
`nodetool bootstrap resume` on the replacement node.



On 12/05/2025 09:53, Courtney wrote:

Hello everyone,

I have a cluster with 2 datacenters. I am using 
GossipingPropertyFileSnitch as my endpoint snitch. Cassandra 
version 4.1.8. One datacenter is fully Ubuntu 24.04 and OpenJDK 11 
and another is Ubuntu 20.04 on OpenJDK 8. A seed node died in my 
second DC running Ubuntu 20.04 hosts. I ordered a new dedicated 
server. I updated my seeds to forget the dead seed node. I did the 
steps to replace a dead node


JVM_OPTS="$JVM_OPTS $JVM_EXTRA_OPTS 
-Dcassandra.replace_address_first_boot="


Configs between the old/new node are identical minus IP addresses 
and that line above in the env file to replace the dead node. I 
started the node and it started replacing the old node and was in 
the `UJ` state. Not long into the process, the new node stops 
processing data and the cluster forgets the new node and remembers 
the old one in its `DN` state (which is turned off, no power). 
There are no errors in the logs. I've tried different times hoping 
to solve the issue. I upped my ROOT logging level to DEBUG, I also 
set "org.apache.cassandra.gms.Gossiper TRACE". No errors.


With TRACE set for the Gossiper, I notice gossiping stops and data 
stopping streaming about the same time. I cannot run any nodetool 
commands on the new node. The process doesn't die, it leaves open 
connections to nodes that are streaming data, but I don't see any 
data streaming.


I've thought through a lot. Space isn't an issue, ulimits are set 
high in /etc/security/limits.conf. Checking /proc//limits 
shows the values are high. I've replaced nodes before like this 
without issue, but this one is causing me grief. Is there anything 
more I can do?


Courtney






Re: Issue replacing a dead node

2025-05-16 Thread Bowen Song via user
In my experience, failed bootstrap / node replacement always leave some 
traces in the logs. At the very minimal, there's going to be logs about 
streaming sessions failing or aborting. I have never seen it silently 
fails or stops without leaving any traces in the log. I can't think of 
anything that can cause the process to fail and doesn't leave a trace in 
the log. BTW, the relevant logs can be hours before the symptom becomes 
visible, because a failed streaming session does not cause Cassandra to 
immediately abort other active streaming sessions, and the remaining 
active sessions can take a while to complete.


If the process repeatedly fails at a certain place, I would suspect some 
sort of data corruption or disk error, resulting in the data cannot be 
read or deserialised correctly. But this is just a guess, and I could be 
wrong.


On 16/05/2025 01:14, Courtney wrote:
I checked all the logs and really couldn't find anything. I couldn't 
find any sort of errors in dmesg, system.log, debug.log, gc.log (maybe 
up the log level?), systemd journal...the logs are totally clean. It 
just stops gossiping all of a sudden at 22GB of data each time, then 
the old node returning to DN state. What is `nodetool bootstrap 
resume` going to do? Is there a risk to running resume when the 
replacement node is no longer in the cluster? Could too high of a 
tombstone ratio cause this?


On 5/15/25 5:08 PM, Bowen Song via user wrote:
The dead node being replaced went back to DN state indicating the new 
replacement node failed to join the cluster, usually because the 
streaming was interrupted (e.g. by network issues, or long STW GC 
pauses). I would start looking for red flags in the logs, including 
Cassandra's logs, GC logs, dmesg, systemd journal, etc., on the new 
node, and other nodes in the cluster too. Also, I would try `nodetool 
bootstrap resume` on the replacement node.



On 12/05/2025 09:53, Courtney wrote:

Hello everyone,

I have a cluster with 2 datacenters. I am using 
GossipingPropertyFileSnitch as my endpoint snitch. Cassandra version 
4.1.8. One datacenter is fully Ubuntu 24.04 and OpenJDK 11 and 
another is Ubuntu 20.04 on OpenJDK 8. A seed node died in my second 
DC running Ubuntu 20.04 hosts. I ordered a new dedicated server. I 
updated my seeds to forget the dead seed node. I did the steps to 
replace a dead node


JVM_OPTS="$JVM_OPTS $JVM_EXTRA_OPTS 
-Dcassandra.replace_address_first_boot="


Configs between the old/new node are identical minus IP addresses 
and that line above in the env file to replace the dead node. I 
started the node and it started replacing the old node and was in 
the `UJ` state. Not long into the process, the new node stops 
processing data and the cluster forgets the new node and remembers 
the old one in its `DN` state (which is turned off, no power). There 
are no errors in the logs. I've tried different times hoping to 
solve the issue. I upped my ROOT logging level to DEBUG, I also set 
"org.apache.cassandra.gms.Gossiper TRACE". No errors.


With TRACE set for the Gossiper, I notice gossiping stops and data 
stopping streaming about the same time. I cannot run any nodetool 
commands on the new node. The process doesn't die, it leaves open 
connections to nodes that are streaming data, but I don't see any 
data streaming.


I've thought through a lot. Space isn't an issue, ulimits are set 
high in /etc/security/limits.conf. Checking /proc//limits shows 
the values are high. I've replaced nodes before like this without 
issue, but this one is causing me grief. Is there anything more I 
can do?


Courtney




Re: Issue replacing a dead node

2025-05-15 Thread Courtney
I checked all the logs and really couldn't find anything. I couldn't 
find any sort of errors in dmesg, system.log, debug.log, gc.log (maybe 
up the log level?), systemd journal...the logs are totally clean. It 
just stops gossiping all of a sudden at 22GB of data each time, then the 
old node returning to DN state. What is `nodetool bootstrap resume` 
going to do? Is there a risk to running resume when the replacement node 
is no longer in the cluster? Could too high of a tombstone ratio cause this?


On 5/15/25 5:08 PM, Bowen Song via user wrote:
The dead node being replaced went back to DN state indicating the new 
replacement node failed to join the cluster, usually because the 
streaming was interrupted (e.g. by network issues, or long STW GC 
pauses). I would start looking for red flags in the logs, including 
Cassandra's logs, GC logs, dmesg, systemd journal, etc., on the new 
node, and other nodes in the cluster too. Also, I would try `nodetool 
bootstrap resume` on the replacement node.



On 12/05/2025 09:53, Courtney wrote:

Hello everyone,

I have a cluster with 2 datacenters. I am using 
GossipingPropertyFileSnitch as my endpoint snitch. Cassandra version 
4.1.8. One datacenter is fully Ubuntu 24.04 and OpenJDK 11 and 
another is Ubuntu 20.04 on OpenJDK 8. A seed node died in my second 
DC running Ubuntu 20.04 hosts. I ordered a new dedicated server. I 
updated my seeds to forget the dead seed node. I did the steps to 
replace a dead node


JVM_OPTS="$JVM_OPTS $JVM_EXTRA_OPTS 
-Dcassandra.replace_address_first_boot="


Configs between the old/new node are identical minus IP addresses and 
that line above in the env file to replace the dead node. I started 
the node and it started replacing the old node and was in the `UJ` 
state. Not long into the process, the new node stops processing data 
and the cluster forgets the new node and remembers the old one in its 
`DN` state (which is turned off, no power). There are no errors in 
the logs. I've tried different times hoping to solve the issue. I 
upped my ROOT logging level to DEBUG, I also set 
"org.apache.cassandra.gms.Gossiper TRACE". No errors.


With TRACE set for the Gossiper, I notice gossiping stops and data 
stopping streaming about the same time. I cannot run any nodetool 
commands on the new node. The process doesn't die, it leaves open 
connections to nodes that are streaming data, but I don't see any 
data streaming.


I've thought through a lot. Space isn't an issue, ulimits are set 
high in /etc/security/limits.conf. Checking /proc//limits shows 
the values are high. I've replaced nodes before like this without 
issue, but this one is causing me grief. Is there anything more I can 
do?


Courtney




Re: Issue replacing a dead node

2025-05-15 Thread Bowen Song via user
The dead node being replaced went back to DN state indicating the new 
replacement node failed to join the cluster, usually because the 
streaming was interrupted (e.g. by network issues, or long STW GC 
pauses). I would start looking for red flags in the logs, including 
Cassandra's logs, GC logs, dmesg, systemd journal, etc., on the new 
node, and other nodes in the cluster too. Also, I would try `nodetool 
bootstrap resume` on the replacement node.



On 12/05/2025 09:53, Courtney wrote:

Hello everyone,

I have a cluster with 2 datacenters. I am using 
GossipingPropertyFileSnitch as my endpoint snitch. Cassandra version 
4.1.8. One datacenter is fully Ubuntu 24.04 and OpenJDK 11 and another 
is Ubuntu 20.04 on OpenJDK 8. A seed node died in my second DC running 
Ubuntu 20.04 hosts. I ordered a new dedicated server. I updated my 
seeds to forget the dead seed node. I did the steps to replace a dead 
node


JVM_OPTS="$JVM_OPTS $JVM_EXTRA_OPTS 
-Dcassandra.replace_address_first_boot="


Configs between the old/new node are identical minus IP addresses and 
that line above in the env file to replace the dead node. I started 
the node and it started replacing the old node and was in the `UJ` 
state. Not long into the process, the new node stops processing data 
and the cluster forgets the new node and remembers the old one in its 
`DN` state (which is turned off, no power). There are no errors in the 
logs. I've tried different times hoping to solve the issue. I upped my 
ROOT logging level to DEBUG, I also set 
"org.apache.cassandra.gms.Gossiper TRACE". No errors.


With TRACE set for the Gossiper, I notice gossiping stops and data 
stopping streaming about the same time. I cannot run any nodetool 
commands on the new node. The process doesn't die, it leaves open 
connections to nodes that are streaming data, but I don't see any data 
streaming.


I've thought through a lot. Space isn't an issue, ulimits are set high 
in /etc/security/limits.conf. Checking /proc//limits shows the 
values are high. I've replaced nodes before like this without issue, 
but this one is causing me grief. Is there anything more I can do?


Courtney


Issue replacing a dead node

2025-05-12 Thread Courtney

Hello everyone,

I have a cluster with 2 datacenters. I am using 
GossipingPropertyFileSnitch as my endpoint snitch. Cassandra version 
4.1.8. One datacenter is fully Ubuntu 24.04 and OpenJDK 11 and another 
is Ubuntu 20.04 on OpenJDK 8. A seed node died in my second DC running 
Ubuntu 20.04 hosts. I ordered a new dedicated server. I updated my seeds 
to forget the dead seed node. I did the steps to replace a dead node


JVM_OPTS="$JVM_OPTS $JVM_EXTRA_OPTS 
-Dcassandra.replace_address_first_boot="


Configs between the old/new node are identical minus IP addresses and 
that line above in the env file to replace the dead node. I started the 
node and it started replacing the old node and was in the `UJ` state. 
Not long into the process, the new node stops processing data and the 
cluster forgets the new node and remembers the old one in its `DN` state 
(which is turned off, no power). There are no errors in the logs. I've 
tried different times hoping to solve the issue. I upped my ROOT logging 
level to DEBUG, I also set "org.apache.cassandra.gms.Gossiper TRACE". No 
errors.


With TRACE set for the Gossiper, I notice gossiping stops and data 
stopping streaming about the same time. I cannot run any nodetool 
commands on the new node. The process doesn't die, it leaves open 
connections to nodes that are streaming data, but I don't see any data 
streaming.


I've thought through a lot. Space isn't an issue, ulimits are set high 
in /etc/security/limits.conf. Checking /proc//limits shows the 
values are high. I've replaced nodes before like this without issue, but 
this one is causing me grief. Is there anything more I can do?


Courtney