Re: Why nodetool cleanup should be run sequentially after node joined a cluster

[Alain RODRIGUEZ]

I confirm what Christophe said.

I always ran them in parallel without any problem, really. Historically it
was using only one compactor and impact in my clusters have always been
acceptable.

Nonetheless, newer Cassandra versions allow multiple compactor to work in
parallel during cleanup and this can be really harmful - or really
efficient if resources are available and it is not impacting the read and
write operations. If all the nodes run cleanup in parallel, then limiting
the number of threads used per node is really important.

My colleague Anthony described this option here:
http://thelastpickle.com/blog/2017/08/14/limiting-nodetool-parallel-threads.html
.

C*heers,
---
Alain Rodriguez - @arodream - al...@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com


2018-04-11 6:04 GMT+01:00 Christophe Schmitz :

> Hi Mikhail,
>
>
> Nodetool cleanup can add a fair amount of extra load (mostly IO) on your
> Cassandra nodes. Therefore it is recommended to run it during lower cluster
> usage, and one node at a time, in order to limit the impact on your
> cluster. There are no technical limitations that would prevent you to run
> it at the same time. It's just a precaution measure.
>
> Cheers,
> Christophe
>
>
> On 11 April 2018 at 14:49, Mikhail Tsaplin  wrote:
>
>> Hi,
>> In https://docs.datastax.com/en/cassandra/3.0/cassandra/oper
>> ations/opsAddNodeToCluster.html
>> there is recommendation:
>> 6) After all new nodes are running, run nodetool cleanup
>> 
>>  on each of the previously existing nodes to remove the keys that no
>> longer belong to those nodes. Wait for cleanup to complete on one node
>> before running nodetool cleanup on the next node.
>>
>> I had added a new node to the cluster, and running nodetool cleanup
>> according to this recommendation - but it takes near 10 days to complete on
>> a single node. Is it safe to start it on all nodes?
>>
>
>
>
> --
>
> *Christophe Schmitz - **VP Consulting*
>
> AU: +61 4 03751980 / FR: +33 7 82022899
>
>    
>
>
> Read our latest technical blog posts here
> . This email has been sent on behalf
> of Instaclustr Pty. Limited (Australia) and Instaclustr Inc (USA). This
> email and any attachments may contain confidential and legally
> privileged information.  If you are not the intended recipient, do not copy
> or disclose its content, but please reply to this email immediately and
> highlight the error to the sender and then immediately delete the message.
>

Re: Why nodetool cleanup should be run sequentially after node joined a cluster

[Christophe Schmitz]

Hi Mikhail,


Nodetool cleanup can add a fair amount of extra load (mostly IO) on your
Cassandra nodes. Therefore it is recommended to run it during lower cluster
usage, and one node at a time, in order to limit the impact on your
cluster. There are no technical limitations that would prevent you to run
it at the same time. It's just a precaution measure.

Cheers,
Christophe


On 11 April 2018 at 14:49, Mikhail Tsaplin  wrote:

> Hi,
> In https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/
> opsAddNodeToCluster.html
> there is recommendation:
> 6) After all new nodes are running, run nodetool cleanup
> 
>  on each of the previously existing nodes to remove the keys that no
> longer belong to those nodes. Wait for cleanup to complete on one node
> before running nodetool cleanup on the next node.
>
> I had added a new node to the cluster, and running nodetool cleanup
> according to this recommendation - but it takes near 10 days to complete on
> a single node. Is it safe to start it on all nodes?
>



-- 

*Christophe Schmitz - **VP Consulting*

AU: +61 4 03751980 / FR: +33 7 82022899

   


Read our latest technical blog posts here
. This email has been sent on behalf
of Instaclustr Pty. Limited (Australia) and Instaclustr Inc (USA). This
email and any attachments may contain confidential and legally
privileged information.  If you are not the intended recipient, do not copy
or disclose its content, but please reply to this email immediately and
highlight the error to the sender and then immediately delete the message.

Why nodetool cleanup should be run sequentially after node joined a cluster

[Mikhail Tsaplin]

Hi,
In
https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsAddNodeToCluster.html
there is recommendation:
6) After all new nodes are running, run nodetool cleanup

 on each of the previously existing nodes to remove the keys that no longer
belong to those nodes. Wait for cleanup to complete on one node before
running nodetool cleanup on the next node.

I had added a new node to the cluster, and running nodetool cleanup
according to this recommendation - but it takes near 10 days to complete on
a single node. Is it safe to start it on all nodes?