Hi Generally a decommissioned node goes in hibernate state and if not wrong it is there for 72 hours. Not seen same for dead node replacement though not tested 5.0.5 as well.
Regards Manish On Tue, 11 Nov 2025 at 3:13 AM, dbms-tech <[email protected]> wrote: > We are testing 5.0.5 as a viable upgrade target in our lab environment. We > noticed that when we replace a dead node, bootstrapping fails with the > error below. > > Some facts ... > 1- We didn't see this failure with 5.0.4. Started with 5.0.5. > 2- Ruled out networking issues. Seeds IP's are reachable by the node. > > This issue looks very similar to CASSANDRA-19580. Other nodes will not > send SYN messages to a node in hibernate state. > > > https://issues.apache.org/jira/browse/CASSANDRA-19580?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&focusedCommentId=17930471 > > > The Jira above describes some workarounds, but they appear unreasonable > like copying system.peers table. > > 1- Is there a behavioral change in 5.0.5? > 2- Is there an operational workaround? > > Thanks > > ############################ > Bootstrap failure messages > ############################ > INFO [MemtableFlushWriter:3] 2025-11-10 20:01:06,450 > LogTransaction.java:249 - Unfinished transaction log, deleting > /data/cassandra/system/peers-37f71aca7dc2383ba70672528af04d4f/bti-da_txn_flush_fe335670-be6f-11f0-87fb-35b5e3b8e93a.log > Exception (java.lang.IllegalStateException) encountered during startup: > Unable to contact any seeds: [/10.xxx:7000, /10.xxxx:7000, /10.xxx:7000] > java.lang.IllegalStateException: Unable to contact any seeds: > [/10.xxx:7000, /10.xxx:7000, /10.xxx:7000] > at > org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:2196) > at > org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1291) > at > org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1251) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:1032) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:947) > --------------------------------------------------------------------- > > ############################ > The boostrapping node is in hibernate state > ############################ > [ec2-user@ip-10-xxxx conf]$ nodetool gossipinfo > /10.xxx > generation:1762808084 > heartbeat:16 > STATUS:3:hibernate,true > SCHEMA:15:59adb24e-f3cd-3e02-97f0-5b395827453f > DC:11:us-east-1 > RACK:13:us-east-1a > RELEASE_VERSION:8:5.0.5 > RPC_ADDRESS:7:10.xxxx > NET_VERSION:4:13 > HOST_ID:5:28fcc0eb-f7aa-4626-a46b-5551438b6b4e > NATIVE_ADDRESS_AND_PORT:6:10.xxxxx:9042 > STATUS_WITH_PORT:2:hibernate,true > SSTABLE_VERSIONS:9:bti-da > TOKENS:1:<hidden> > --------------------------------------------------------------------- > > ---------------------------------------- > Thank you > > >
