Another request to the community to see if this is feasible or not:
Can we not wait for (CEP-21), and do the necessary cleanup as part of
regular compaction itself to avoid running *cleanup* manually? For now, we
can control through a flag, which is *false* by default. Whosoever wants to
do the
Because an operator will need to check and ensure the schema is
consistent across the cluster before running "nodetool cleanup". At the
moment, it's the operator's responsibility to ensure bad things don't
happen.
On 09/05/2023 06:20, Jaydeep Chovatia wrote:
One clarification question Jeff.
One clarification question Jeff.
AFAIK, the *nodetool cleanup* also internally goes through the same
compaction path as the regular compaction. Then why do we have to wait for
CEP-21 to clean up unowned data in the regular compaction path? Wouldn't it
be as simple as regular compaction just invoke
Thanks, Jeff, for the detailed steps and summary.
We will keep the community (this thread) up to date on how it plays out in
our fleet.
Jaydeep
On Fri, May 5, 2023 at 9:10 AM Jeff Jirsa wrote:
> Lots of caveats on these suggestions, let me try to hit most of them.
>
> Cleanup in parallel is
Lots of caveats on these suggestions, let me try to hit most of them.
Cleanup in parallel is good and fine and common. Limit number of threads in
cleanup if you're using lots of vnodes, so each node runs one at a time and
not all nodes use all your cores at the same time.
If a host is fully
Thanks all for your valuable inputs. We will try some of the suggested
methods in this thread, and see how it goes. We will keep you updated on
our progress.
Thanks a lot once again!
Jaydeep
On Fri, May 5, 2023 at 8:55 AM Bowen Song via user <
user@cassandra.apache.org> wrote:
> Depending on
Depending on the number of vnodes per server, the probability and
severity (i.e. the size of the affected token ranges) of an availability
degradation due to a server failure during node replacement may be
small. You also have the choice of increasing the RF if that's still not
acceptable.
We are doing the "adding a node then decommissioning a node" to
achieve better availability. Replacing a node need to shut down one node
first, if another node is down during the node replacement period, we will
get availability drop because most of our use case is local_quorum with
replication
Have you thought of using "-Dcassandra.replace_address_first_boot=..."
(or "-Dcassandra.replace_address=..." if you are using an older
version)? This will not result in a topology change, which means
"nodetool cleanup" is not needed after the operation is completed.
On 05/05/2023 05:24,
Durity
From: manish khandelwal
Sent: Friday, May 5, 2023 4:52 AM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Re: Is cleanup is required if cluster topology changes
You can replace the node directly why to add a node and decommission the
another node. Just replace the node with the new node
You can replace the node directly why to add a node and decommission the
another node. Just replace the node with the new node and your topology
remains the same so no need to run the cleanup .
On Fri, May 5, 2023 at 10:26 AM Jaydeep Chovatia
wrote:
> We use STCS, and our experience with
We use STCS, and our experience with *cleanup* is that it takes a long time
to run in a 100-node cluster. We would like to replace one node every day
for various purposes in our fleet.
If we run *cleanup* after each node replacement, then it might take, say,
15 days to complete, and that hinders
You should 100% trigger cleanup each time or you’ll almost certainly resurrect data sooner or laterIf you’re using leveled compaction it’s especially cheap. Stcs and twcs are worse, but if you’re really scaling that often, I’d be considering lcs and running cleanup just before or just after each
Thanks, Jeff!
But in our environment we replace nodes quite often for various
optimization purposes, etc. say, almost 1 node per day (node *addition*
followed by node *decommission*, which of course changes the topology), and
we have a cluster of size 100 nodes with 300GB per node. If we have to
Cleanup is fast and cheap and basically a no-op if you haven’t changed the ring After cassandra has transactional cluster metadata to make ring changes strongly consistent, cassandra should do this in every compaction. But until then it’s left for operators to run when they’re sure the state of
Isn't this considered a kind of *bug* in Cassandra because as we know
*cleanup* is a lengthy and unreliable operation, so relying on the *cleanup*
means higher chances of data resurrection?
Do you think we should discard the unowned token-ranges as part of the
regular compaction itself? What are
compact ion will just merge duplicate data and remove delete data in this
node .if you add or remove one node for the cluster, I think clean up is
needed. if clean up failed, I think we should come to see the reason.
Runtian Liu 于2023年5月5日周五 06:37写道：
> Hi all,
>
> Is cleanup the sole method to
Hi all,
Is cleanup the sole method to remove data that does not belong to a
specific node? In a cluster, where nodes are added or decommissioned from
time to time, failure to run cleanup may lead to data resurrection issues,
as deleted data may remain on the node that lost ownership of certain
18 matches
Mail list logo
