Best strategy to run repair

[Surbhi Gupta]

Hi,

We are on open source 3.11.5 .
We need to repair a production cluster .
We are using num_token as 256 .
What will be a better option to run repair ?
1. nodetool -pr  (Primary range repair on all nodes, one node at a time)
OR
2. nodetool -st -et (Subrange repair , taking the ranges for each node from
nodetool describering) and run 256 repairs on each node .

Thanks
Surbhi

Re: Best strategy to run repair

[Kane Wilson]

-pr on all nodes takes much longer as you'll do at least triple the amount
of merkle calculations I believe (with RF 3) and tends to be quite
problematic.

Subrange is the way to go, which is what cassandra-reaper will do for you
if you have it set up.

raft.so - Cassandra consulting, support, and managed services


On Tue, Mar 23, 2021 at 7:33 AM Surbhi Gupta 
wrote:

> Hi,
>
> We are on open source 3.11.5 .
> We need to repair a production cluster .
> We are using num_token as 256 .
> What will be a better option to run repair ?
> 1. nodetool -pr  (Primary range repair on all nodes, one node at a time)
> OR
> 2. nodetool -st -et (Subrange repair , taking the ranges for each node
> from nodetool describering) and run 256 repairs on each node .
>
> Thanks
> Surbhi
>
>

Re: Best strategy to run repair

[manish khandelwal]

Also try to use Cassandra reaper (as Kane also mentioned) for subrange
repair. Doing subrange repair yourself may lead to a lot of trouble as
calculating correct subranges is not an easy task.

On Tue, Mar 23, 2021 at 3:38 AM Kane Wilson  wrote:

> -pr on all nodes takes much longer as you'll do at least triple the amount
> of merkle calculations I believe (with RF 3) and tends to be quite
> problematic.
>
> Subrange is the way to go, which is what cassandra-reaper will do for you
> if you have it set up.
>
> raft.so - Cassandra consulting, support, and managed services
>
>
> On Tue, Mar 23, 2021 at 7:33 AM Surbhi Gupta 
> wrote:
>
>> Hi,
>>
>> We are on open source 3.11.5 .
>> We need to repair a production cluster .
>> We are using num_token as 256 .
>> What will be a better option to run repair ?
>> 1. nodetool -pr  (Primary range repair on all nodes, one node at a time)
>> OR
>> 2. nodetool -st -et (Subrange repair , taking the ranges for each node
>> from nodetool describering) and run 256 repairs on each node .
>>
>> Thanks
>> Surbhi
>>
>>

Re: Best strategy to run repair

[manish khandelwal]

If you can pick  ranges on your own correctly, then you can do that way. In
my opinion a ready made tested solution is available, I think it should be
used.

On Tue, Mar 23, 2021 at 10:55 AM Surbhi Gupta 
wrote:

> Does describering not give the correct sub ranges for each node ?
>
> On Mon, 22 Mar 2021 at 20:28, manish khandelwal <
> manishkhandelwa...@gmail.com> wrote:
>
>> Also try to use Cassandra reaper (as Kane also mentioned) for subrange
>> repair. Doing subrange repair yourself may lead to a lot of trouble as
>> calculating correct subranges is not an easy task.
>>
>> On Tue, Mar 23, 2021 at 3:38 AM Kane Wilson  wrote:
>>
>>> -pr on all nodes takes much longer as you'll do at least triple the
>>> amount of merkle calculations I believe (with RF 3) and tends to be quite
>>> problematic.
>>>
>>> Subrange is the way to go, which is what cassandra-reaper will do for
>>> you if you have it set up.
>>>
>>> raft.so - Cassandra consulting, support, and managed services
>>>
>>>
>>> On Tue, Mar 23, 2021 at 7:33 AM Surbhi Gupta 
>>> wrote:
>>>
 Hi,

 We are on open source 3.11.5 .
 We need to repair a production cluster .
 We are using num_token as 256 .
 What will be a better option to run repair ?
 1. nodetool -pr  (Primary range repair on all nodes, one node at a time)
 OR
 2. nodetool -st -et (Subrange repair , taking the ranges for each node
 from nodetool describering) and run 256 repairs on each node .

 Thanks
 Surbhi

Re: Best strategy to run repair

[Surbhi Gupta]

Does describering not give the correct sub ranges for each node ?

On Mon, 22 Mar 2021 at 20:28, manish khandelwal <
manishkhandelwa...@gmail.com> wrote:

> Also try to use Cassandra reaper (as Kane also mentioned) for subrange
> repair. Doing subrange repair yourself may lead to a lot of trouble as
> calculating correct subranges is not an easy task.
>
> On Tue, Mar 23, 2021 at 3:38 AM Kane Wilson  wrote:
>
>> -pr on all nodes takes much longer as you'll do at least triple the
>> amount of merkle calculations I believe (with RF 3) and tends to be quite
>> problematic.
>>
>> Subrange is the way to go, which is what cassandra-reaper will do for you
>> if you have it set up.
>>
>> raft.so - Cassandra consulting, support, and managed services
>>
>>
>> On Tue, Mar 23, 2021 at 7:33 AM Surbhi Gupta 
>> wrote:
>>
>>> Hi,
>>>
>>> We are on open source 3.11.5 .
>>> We need to repair a production cluster .
>>> We are using num_token as 256 .
>>> What will be a better option to run repair ?
>>> 1. nodetool -pr  (Primary range repair on all nodes, one node at a time)
>>> OR
>>> 2. nodetool -st -et (Subrange repair , taking the ranges for each node
>>> from nodetool describering) and run 256 repairs on each node .
>>>
>>> Thanks
>>> Surbhi
>>>
>>>

Re: Changing num_tokens and migrating to 4.0

[Lapo Luchini]

On 2021-03-22 01:27, Kane Wilson wrote:
You should be able to get repairs working fine if you use a tool such as 
cassandra-reaper to manage it for you for such a small cluster. I would 
look into that before doing major cluster topology changes, as these can 
be complex and risky.


I was looking into a migration as I'm already using Cassanrea Reaper and 
on the biggest keyspace is often taking more than 7 days to complete (I 
set the segment timeout up to 2h, and most of 138 segments take more 
than 1h, sometimes even failing the 2h limits due to, AFAICT, lengthy 
compactions).


--
Lapo Luchini
l...@lapo.it


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

RE: Changing num_tokens and migrating to 4.0

[Durity, Sean R]

I have a cluster (almost 200 nodes) with a variety of disk sizes and use 
different numbers of tokens so that the machines can use the disk they have. It 
is a very handy feature! While I agree that a node with larger disk may handle 
more requests, that may not be enough to impact CPU or memory. I rarely see 
Cassandra CPU-bound for my use cases. These are primarily write use cases with 
a low number of clients with far fewer reads. There is just a lot of data to 
keep.

Sean Durity

From: Alex Ott 
Sent: Saturday, March 20, 2021 1:01 PM
To: user 
Subject: [EXTERNAL] Re: Changing num_tokens and migrating to 4.0

if the nodes are almost the same, except the disk space, then giving them more 
may make siltation worse - they will get more requests than other nodes, and 
won't have resources to process them.
In Cassandra the disk size isn't the main "success" factor - it's a memory, 
CPU, disk type (SSD), etc.

On Sat, Mar 20, 2021 at 5:26 PM Lapo Luchini 
mailto:l...@lapo.it>> wrote:
Hi, thanks for suggestions!
I'll definitely migrate to 4.0 after all this is done, then.

Old prod DC I fear can't suffer losing a node right now (a few nodes
have the disk 70% full), but I can maybe find a third node for the new
DC right away.

BTW the new nodes have got 3× the disk space, but are not so much
different regarding CPU and RAM: does it make any sense giving them a
bit more num_tokens (maybe 20-30 instead of 16) than the rest of the old
DC hosts or "asymmetrical" clusters lead to problems?

No real need to do that anyways, moving from 6 nodes to (eventually) 8
should be enough lessen the load on the disks, and before more space is
needed I will probably have more nodes.

Lapo

On 2021-03-20 16:23, Alex Ott wrote:
> I personally maybe would go following way (need to calculate how many
> joins/decommissions will be at the end):
>
>   * Decommission one node from prod DC
>   * Form new DC from two new machines and decommissioned one.
>   * Rebuild DC from existing one, make sure that repair finished, etc.
>   * Switch traffic
>   * Remove old DC
>   * Add nodes from old DC one by one into new DC
>



-
To unsubscribe, e-mail: 
user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: 
user-h...@cassandra.apache.org


--
With best wishes,Alex Ott
http://alexott.net/ 
[alexott.net]
Twitter: alexott_en (English), alexott (Russian)



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.