[
https://issues.apache.org/jira/browse/CASSANDRA-18555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17729637#comment-17729637
]
Stefan Miklosovic commented on CASSANDRA-18555:
-----------------------------------------------
Actually, as I looked more into it, the ability to retrieve the failed
decommission state in something like "nodetool status" would mean that we would
need to introduce this "state" into TokenMetadata and "topology" and this stuff
would need to be propagated via Gossip etc. Think about it ... if we are about
to know that decommission has failed on some node _from whatever node_, the
only way a node knows that such and such node failed to decommission is to get
this information from Gossip. I think that is way too much work ...
On the other hand, if I understand this correctly, this (1) is a Runnable which
is passed to this method (2) where the actual decommission is done and it is
run here (3) at the very end.
So, as native transport is being shut down as part of that Runnable here (4),
that means that if something goes wrong with decommission and it throw an
exception, thar Runnable is never called which means that NativeTransport / CQL
together with Gossip is never stopped either so we do have ability to connect
to CQL even if decommission fails, no?
That means that we might reflect the fact that decommission has failed by
writing it to some table - even a virtual one - and we do not need to depend on
JMX nor on any nodetool.
(1)
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/StorageService.java#L5189
(2)
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/StorageService.java#L5210
(3)
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/StorageService.java#L5293
(4)
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/StorageService.java#L5193-L5194
> A new nodetool/JMX command that tells whether node's decommission failed or
> not
> -------------------------------------------------------------------------------
>
> Key: CASSANDRA-18555
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18555
> Project: Cassandra
> Issue Type: Task
> Components: Observability/JMX
> Reporter: Jaydeepkumar Chovatia
> Assignee: Jaydeepkumar Chovatia
> Priority: Normal
>
> Currently, when a node is being decommissioned and if any failure happens,
> then an exception is thrown back to the caller.
> But Cassandra's decommission takes considerable time ranging from minutes to
> hours to days. There are various scenarios in that the caller may need to
> probe the status again:
> * The caller times out
> * It is not possible to keep the caller hanging for such a long time
> And If the caller does not know what happened internally, then it cannot
> retry, etc., leading to other issues.
> So, in this ticket, I am going to add a new nodetool/JMX command that can be
> invoked by the caller anytime, and it will return the correct status.
> It might look like a smaller change, but when we need to operate Cassandra at
> scale in a large-scale fleet, then this becomes a bottleneck and require
> constant operator intervention.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
