Anyone ran into this issue? Seems to be a known bug affecting multiple releases 4.x/5.x. I'm not sure how it is possible to bootstrap a node if it can't talk to the seed nodes because it is in a HIBERNATE state? Is there a way to force a bootstrapping node to be in a SHUTDOWN state so bootstrap can advance? This seems too critical of a bug to be left unresolved.
On Mon, Nov 10, 2025 at 4:42 PM dbms-tech <[email protected]> wrote: > We are testing 5.0.5 as a viable upgrade target in our lab environment. We > noticed that when we replace a dead node, bootstrapping fails with the > error below. > > Some facts ... > 1- We didn't see this failure with 5.0.4. Started with 5.0.5. > 2- Ruled out networking issues. Seeds IP's are reachable by the node. > > This issue looks very similar to CASSANDRA-19580. Other nodes will not > send SYN messages to a node in hibernate state. > > > https://issues.apache.org/jira/browse/CASSANDRA-19580?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&focusedCommentId=17930471 > > > The Jira above describes some workarounds, but they appear unreasonable > like copying system.peers table. > > 1- Is there a behavioral change in 5.0.5? > 2- Is there an operational workaround? > > Thanks > > ############################ > Bootstrap failure messages > ############################ > INFO [MemtableFlushWriter:3] 2025-11-10 20:01:06,450 > LogTransaction.java:249 - Unfinished transaction log, deleting > /data/cassandra/system/peers-37f71aca7dc2383ba70672528af04d4f/bti-da_txn_flush_fe335670-be6f-11f0-87fb-35b5e3b8e93a.log > Exception (java.lang.IllegalStateException) encountered during startup: > Unable to contact any seeds: [/10.xxx:7000, /10.xxxx:7000, /10.xxx:7000] > java.lang.IllegalStateException: Unable to contact any seeds: > [/10.xxx:7000, /10.xxx:7000, /10.xxx:7000] > at > org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:2196) > at > org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1291) > at > org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1251) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:1032) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:947) > --------------------------------------------------------------------- > > ############################ > The boostrapping node is in hibernate state > ############################ > [ec2-user@ip-10-xxxx conf]$ nodetool gossipinfo > /10.xxx > generation:1762808084 > heartbeat:16 > STATUS:3:hibernate,true > SCHEMA:15:59adb24e-f3cd-3e02-97f0-5b395827453f > DC:11:us-east-1 > RACK:13:us-east-1a > RELEASE_VERSION:8:5.0.5 > RPC_ADDRESS:7:10.xxxx > NET_VERSION:4:13 > HOST_ID:5:28fcc0eb-f7aa-4626-a46b-5551438b6b4e > NATIVE_ADDRESS_AND_PORT:6:10.xxxxx:9042 > STATUS_WITH_PORT:2:hibernate,true > SSTABLE_VERSIONS:9:bti-da > TOKENS:1:<hidden> > --------------------------------------------------------------------- > > ---------------------------------------- > Thank you > > > -- ---------------------------------------- Thank you
