We are testing 5.0.5 as a viable upgrade target in our lab environment. We noticed that when we replace a dead node, bootstrapping fails with the error below.
Some facts ... 1- We didn't see this failure with 5.0.4. Started with 5.0.5. 2- Ruled out networking issues. Seeds IP's are reachable by the node. This issue looks very similar to CASSANDRA-19580. Other nodes will not send SYN messages to a node in hibernate state. https://issues.apache.org/jira/browse/CASSANDRA-19580?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&focusedCommentId=17930471 The Jira above describes some workarounds, but they appear unreasonable like copying system.peers table. 1- Is there a behavioral change in 5.0.5? 2- Is there an operational workaround? Thanks ############################ Bootstrap failure messages ############################ INFO [MemtableFlushWriter:3] 2025-11-10 20:01:06,450 LogTransaction.java:249 - Unfinished transaction log, deleting /data/cassandra/system/peers-37f71aca7dc2383ba70672528af04d4f/bti-da_txn_flush_fe335670-be6f-11f0-87fb-35b5e3b8e93a.log Exception (java.lang.IllegalStateException) encountered during startup: Unable to contact any seeds: [/10.xxx:7000, /10.xxxx:7000, /10.xxx:7000] java.lang.IllegalStateException: Unable to contact any seeds: [/10.xxx:7000, /10.xxx:7000, /10.xxx:7000] at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:2196) at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1291) at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1251) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:1032) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:947) --------------------------------------------------------------------- ############################ The boostrapping node is in hibernate state ############################ [ec2-user@ip-10-xxxx conf]$ nodetool gossipinfo /10.xxx generation:1762808084 heartbeat:16 STATUS:3:hibernate,true SCHEMA:15:59adb24e-f3cd-3e02-97f0-5b395827453f DC:11:us-east-1 RACK:13:us-east-1a RELEASE_VERSION:8:5.0.5 RPC_ADDRESS:7:10.xxxx NET_VERSION:4:13 HOST_ID:5:28fcc0eb-f7aa-4626-a46b-5551438b6b4e NATIVE_ADDRESS_AND_PORT:6:10.xxxxx:9042 STATUS_WITH_PORT:2:hibernate,true SSTABLE_VERSIONS:9:bti-da TOKENS:1:<hidden> --------------------------------------------------------------------- ---------------------------------------- Thank you
