[
https://issues.apache.org/jira/browse/CASSANDRA-21365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18136913#comment-18136913
]
Andrew Weaver commented on CASSANDRA-21365:
-------------------------------------------
We were hit by this again so I'm working towards a patch here. Confirmed that
this affects 4.0.x and 5.x. This behavior seems to more or less stem from our
use of {{cassandra.join_ring=false}} to start nodes up in {{HIBERNATE}} state
while they wait to be joined to the cluster. This is an artifact of the
automation framework we use that simply makes it easier to prepare and start
Cassandra on a batch of new VMs and join them sequentially rather than starting
and joining them individually.
We've somewhat overloaded the {{cassandra.join_ring=false}} behavior that was
intended for running coordinator-only nodes that would never be joined to the
ring hence some open edge cases in the code.
Here's pretty much what happens and proposed fixes.
h2. Inadequate check for {{disablegossip}}
{{StorageService.stopGossiping}} has a guard to prevent coordinator-only nodes
from disabling gossip:
{code:java}
public void stopGossiping()
{
if (gossipActive)
{
if (!isNormal() && joinRing)
throw new IllegalStateException("Unable to stop gossip because
the node is not in the normal state. Try to stop the node instead.");
logger.warn("Stopping gossip by operator request");
...
{code}
* {{joinRing}} is the static {{-Dcassandra.join_ring flag}} (:507). It stays
false for the life of the JVM, even if a join completes later.
* During bootstrap the mode is JOINING, so isNormal() is false. With the
default join_ring=true the exception is thrown. With {{join_ring=false}} the
guard is skipped.
* CASSANDRA-16146 added the isNormal() guard in 2020. CASSANDRA-17752
(a7b53217bd, 2022) added && joinRing so that gossip-only nodes could restart
services. That change left bootstrapping nodes started with join_ring=false
unguarded.
Fix:
{code:java}
if (!isNormal() && joinRing) --> if(!isNormal() && (joinRing || isJoining() ||
hasJoined())
{code}
h2. Stopping gossip doesn't announce shutdown
If we fix the {{stopGossiping}} guard to prevent stopping gossip on a node in
this state, this only might be a problem for something like a clean shutdown.
Should anything be announced here?
h2. Starting gossip publishes NORMAL
startGossiping() does the following:
* {{SystemKeyspace.getSavedTokens()}} returns tokens, because bootstrap() was
called
* {{validTokens}} is true, so it calls {{{}setGossipTokens(tokens){}}}.
* {{setGossipTokens}} unconditionally writes TOKENS, STATUS_WITH_PORT=normal
and STATUS=normal.
* Nothing consults isBootstrapMode or operationMode.
Fix: Set the right status
{code:java}
public void setGossipTokens(Collection<Token> tokens)
{
List<Pair<ApplicationState, VersionedValue>> states = new ArrayList<>();
VersionedValue status = isBootstrapMode() ?
valueFactory.bootstrapping(tokens)
: valueFactory.normal(tokens);
states.add(Pair.create(ApplicationState.TOKENS,
valueFactory.tokens(tokens)));
states.add(Pair.create(ApplicationState.STATUS_WITH_PORT, status));
states.add(Pair.create(ApplicationState.STATUS, status));
Gossiper.instance.addLocalApplicationStates(states);
}
{code}
> Premature UJ→UN transition when restarting gossip on joining node during in
> bootstrap
> -------------------------------------------------------------------------------------
>
> Key: CASSANDRA-21365
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21365
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Cluster/Gossip
> Reporter: Amit Kumar
> Priority: Normal
>
> We had a node joining the cluster with -Dcassandra.join_ring=false, the node
> was streaming data from other node (State: UJ and bootstrap = true). The node
> gossip was restarted (stop & start) due to a maintainance event which caused
> the node status to move the status from UJ -> UN without streaming all the
> data.
> Code Reference:
> [https://github.com/apache/cassandra/blob/cassandra-4.0/src/java/org/apache/cassandra/service/StorageService.java#L508]
> Version : 4.0.13
> Below are log (Node did not complete the streaming but moved the status from
> UJ -> UN after gossip restart):
> {{INFO [RMI TCP Connection(4)-10.134.160.29] 2026-04-09 20:45:48,005
> StorageService.java:1219 - Joining ring by operator request}}
> {{INFO [RMI TCP Connection(4)-10.134.160.29] 2026-04-09 20:45:48,060
> StorageService.java:1732 - JOINING: waiting for ring information}}
> {{INFO [RMI TCP Connection(4)-10.134.160.29] 2026-04-09 20:46:42,162
> StorageService.java:1732 - JOINING: Starting to bootstrap...}}
> {{WARN [RMI TCP Connection(2030)-10.134.163.199] 2026-04-13 03:02:37,604
> StorageService.java:447 - Stopping gossip by operator request}}
> {{WARN [RMI TCP Connection(2038)-10.134.163.199] 2026-04-13 03:15:38,965
> StorageService.java:466 - Starting gossip by operator request}}
> {{INFO [RMI TCP Connection(2038)-10.134.163.199] 2026-04-13 03:15:38,982
> StorageService.java:2913 - Node /10.xxx.xxx.xxx:7000 state jump to NORMAL}}
> Error logs:
> {{ERROR [ReadStage-1] 2026-04-13 15:56:21,967
> AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[ReadStage-1,5,main]}}
> {{java.lang.RuntimeException: Cannot service reads while bootstrapping!}}
> {{ at
> org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:46)}}
> {{ at
> org.apache.cassandra.net.InboundSink.lambda$new$0(InboundSink.java:78)}}
> {{ at
> org.apache.cassandra.net.InboundSink.accept(InboundSink.java:97)}}
> {{ at
> org.apache.cassandra.net.InboundSink.accept(InboundSink.java:45)}}
> {{ at
> org.apache.cassandra.net.InboundMessageHandler$ProcessMessage.run(InboundMessageHandler.java:432)}}
> {{ at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)}}
> {{ at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:165)}}
> {{ at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:137)}}
> {{ at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:119)}}
> {{ at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)}}
> {{ at java.base/java.lang.Thread.run(Thread.java:840)}}
> Possible solution: change the status of node to JOINING or Bootstrapping when
> enabling gossip if the node is in bootstrap mode.
> {code:java}
> public void setGossipTokens(Collection<Token> tokens){
> List<Pair<ApplicationState, VersionedValue>> states = new
> ArrayList<Pair<ApplicationState, VersionedValue>>();
> states.add(Pair.create(ApplicationState.TOKENS,
> valueFactory.tokens(tokens)));
> states.add(Pair.create(ApplicationState.STATUS_WITH_PORT,
> valueFactory.normal(tokens)));
> states.add(Pair.create(ApplicationState.STATUS, isBootstrapMode() :
> valueFactory.bootstrapping(tokens) ? valueFactory.normal(tokens)));
> Gossiper.instance.addLocalApplicationStates(states);
> } {code}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
