Re: Question related to nodetool repair options

[Deepak Sharma]

Thanks Erick! It is clear now.

On Tue, Sep 7, 2021 at 4:07 PM Erick Ramirez 
wrote:

> No, I'm just saying that [-pr] is the same as [-pr -full], NOT the same as
> just [-full] on its own. Primary range repairs are not compatible with
> incremental repairs so by definition, -pr is a [-pr -full] repair. I think
> you're confusing the concept of a full repair vs incremental. This document
> might help you understand the concepts --
> https://docs.datastax.com/en/cassandra-oss/3.x/cassandra/operations/opsRepairNodesManualRepair.html.
> Cheers!
>
>>

Re: Question related to nodetool repair options

[Erick Ramirez]

No, I'm just saying that [-pr] is the same as [-pr -full], NOT the same as
just [-full] on its own. Primary range repairs are not compatible with
incremental repairs so by definition, -pr is a [-pr -full] repair. I think
you're confusing the concept of a full repair vs incremental. This document
might help you understand the concepts --
https://docs.datastax.com/en/cassandra-oss/3.x/cassandra/operations/opsRepairNodesManualRepair.html.
Cheers!

>

Re: Question related to nodetool repair options

[Deepak Sharma]

Thanks Erick for the response. So in option 3, -pr is not taken into
consideration which essentially means option 3 is the same as option 1
(which is the full repair).

Right, just want to be sure?

Best,
Deepak

On Tue, Sep 7, 2021 at 3:41 PM Erick Ramirez 
wrote:

>
>1. Will perform a full repair vs incremental which is the default in
>some later versions.
>2. As you said, will only repair the token range(s) on the node for
>which it is a primary owner.
>3. The -full flag with -pr is redundant -- primary range repairs are
>always done as a full repair because it is not compatible with incremental
>repairs,, i.e. -pr doesn't care that an SSTable is already marked as
>repaired.
>
>

Re: Question related to nodetool repair options

[Erick Ramirez]

   1. Will perform a full repair vs incremental which is the default in
   some later versions.
   2. As you said, will only repair the token range(s) on the node for
   which it is a primary owner.
   3. The -full flag with -pr is redundant -- primary range repairs are
   always done as a full repair because it is not compatible with incremental
   repairs,, i.e. -pr doesn't care that an SSTable is already marked as
   repaired.

Question related to nodetool repair options

[Deepak Sharma]

Hi There,

We are on Cassandra 3.0.11 and I want to understand what is the
difference between following two commands

1. nodetool repair -full
2. nodetool repair -pr
3. nodetool repair -full -pr

As per my understanding 1. will do the full repair across all keyspaces. 2.
with -pr, restricts repair to the 'primary' token ranges of the node being
repaired. With 3. I am not sure what we are trying to achieve.

Thanks,
Deepak

Re: High memory usage during nodetool repair

[Elliott Sims]

Shouldn't cause GCs.

You can usually think of heap memory separately from the rest.  It's
already allocated as far as the OS is concerned, and it doesn't know
anything about GC going on inside of that allocation.  You can set
"-XX:+AlwaysPreTouch" to make sure it's physically allocated on startup.
JVM OOMs when there's not enough memory in the heap, and a system OOM
(invocation of oom-killer) happens when there's not enough memory outside
of the heap.  The kernel will generally pretty aggressively reclaim mmap'd
memory before resorting to oom-killer.

The main con of disabling data mmap is exactly that - reduced read perf and
increased I/O load.

I think to some extent you're solving a non-problem by disabling mmap to
reduce memory utilization.  Unused memory is wasted memory, so there's not
a lot of reasons to avoid using it as a file read cache.  Especially if
you're preallocating JVM memory and not running any other services on that
host.  You probably only want to disable data mmap if your data-to-RAM
ratio is so high that it's just thrashing and not doing anything useful.

On Tue, Aug 3, 2021 at 10:18 AM Amandeep Srivastava <
amandeep.srivastava1...@gmail.com> wrote:

> Thanks. I guess some earlier thread got truncated.
>
> I already applied Erick's recommendations and that seem to have worked in
> reducing the ram consumption by around 50%.
>
> Regarding cheap memory and hardware, we are already running 96GB boxes and
> getting multiple larger ones might be a little difficult at this point.
> Hence I wanted to understand cons of disabling mmap use for data.
>
> Besides degraded read performance, wouldn't we be putting more pressure on
> heap memory, when disabling mmap, which might cause frequent GCs and OOM
> errors at some point? Since currently whatever was being served by mmap
> would be loaded over heap now and processed/stored further.
>
> Also, we've disabled the swap usage on hosts as recommended to optimize
> performance so cass won't be able to enter that too in case memory starts
> to fill up.
>
> On Tue, 3 Aug, 2021, 6:33 pm Jim Shaw,  wrote:
>
>> I think Erick posted https://community.datastax.com/questions/6947/.
>> explained very clearly.
>>
>> We hit same issue only on a huge table when upgrade, and we changed back
>> after done.
>> My understanding,  Which option to chose,  shall depend on your user
>> case. If chasing high performance on a big table, then go default one, and
>> increase capacity in memory, nowadays hardware is cheaper.
>>
>> Thanks,
>> Jim
>>
>> On Mon, Aug 2, 2021 at 7:12 PM Amandeep Srivastava <
>> amandeep.srivastava1...@gmail.com> wrote:
>>
>>> Can anyone please help with the above questions? To summarise:
>>>
>>> 1) What is the impact of using mmap only for indices besides a
>>> degradation in read performance?
>>> 2) Why does the off heap consumed during Cassandra full repair remains
>>> occupied 12+ hours after the repair completion and is there a
>>> manual/configuration driven way to clear that earlier?
>>>
>>> Thanks,
>>> Aman
>>>
>>> On Thu, 29 Jul, 2021, 6:47 pm Amandeep Srivastava, <
>>> amandeep.srivastava1...@gmail.com> wrote:
>>>
 Hi Erick,

 Limiting mmap to index only seems to have resolved the issue. The max
 ram usage remained at 60% this time. Could you please point me to the
 limitations for setting this param? - For starters, I can see read
 performance getting reduced up to 30% (CASSANDRA-8464
 )

 Also if you could please shed light on extended questions in my earlier
 email.

 Thanks a lot.

 Regards,
 Aman

 On Thu, Jul 29, 2021 at 12:52 PM Amandeep Srivastava <
 amandeep.srivastava1...@gmail.com> wrote:

> Thanks, Bowen, don't think that's an issue - but yes I can try
> upgrading to 3.11.5 and limit the merkle tree size to bring down the 
> memory
> utilization.
>
> Thanks, Erick, let me try that.
>
> Can someone please share documentation relating to internal
> functioning of full repairs - if there exists one? Wanted to understand 
> the
> role of the heap and off-heap memory separately during the process.
>
> Also, for my case, once the nodes reach the 95% memory usage, it stays
> there for almost 10-12 hours after the repair is complete, before falling
> back to 65%. Any pointers on what might be consuming off-heap for so long
> and can something be done to clear it earlier?
>
> Thanks,
> Aman
>
>
>

 --
 Regards,
 Aman

>>>

Re: High memory usage during nodetool repair

[Amandeep Srivastava]

Thanks. I guess some earlier thread got truncated.

I already applied Erick's recommendations and that seem to have worked in
reducing the ram consumption by around 50%.

Regarding cheap memory and hardware, we are already running 96GB boxes and
getting multiple larger ones might be a little difficult at this point.
Hence I wanted to understand cons of disabling mmap use for data.

Besides degraded read performance, wouldn't we be putting more pressure on
heap memory, when disabling mmap, which might cause frequent GCs and OOM
errors at some point? Since currently whatever was being served by mmap
would be loaded over heap now and processed/stored further.

Also, we've disabled the swap usage on hosts as recommended to optimize
performance so cass won't be able to enter that too in case memory starts
to fill up.

On Tue, 3 Aug, 2021, 6:33 pm Jim Shaw,  wrote:

> I think Erick posted https://community.datastax.com/questions/6947/.
> explained very clearly.
>
> We hit same issue only on a huge table when upgrade, and we changed back
> after done.
> My understanding,  Which option to chose,  shall depend on your user case.
> If chasing high performance on a big table, then go default one, and
> increase capacity in memory, nowadays hardware is cheaper.
>
> Thanks,
> Jim
>
> On Mon, Aug 2, 2021 at 7:12 PM Amandeep Srivastava <
> amandeep.srivastava1...@gmail.com> wrote:
>
>> Can anyone please help with the above questions? To summarise:
>>
>> 1) What is the impact of using mmap only for indices besides a
>> degradation in read performance?
>> 2) Why does the off heap consumed during Cassandra full repair remains
>> occupied 12+ hours after the repair completion and is there a
>> manual/configuration driven way to clear that earlier?
>>
>> Thanks,
>> Aman
>>
>> On Thu, 29 Jul, 2021, 6:47 pm Amandeep Srivastava, <
>> amandeep.srivastava1...@gmail.com> wrote:
>>
>>> Hi Erick,
>>>
>>> Limiting mmap to index only seems to have resolved the issue. The max
>>> ram usage remained at 60% this time. Could you please point me to the
>>> limitations for setting this param? - For starters, I can see read
>>> performance getting reduced up to 30% (CASSANDRA-8464
>>> )
>>>
>>> Also if you could please shed light on extended questions in my earlier
>>> email.
>>>
>>> Thanks a lot.
>>>
>>> Regards,
>>> Aman
>>>
>>> On Thu, Jul 29, 2021 at 12:52 PM Amandeep Srivastava <
>>> amandeep.srivastava1...@gmail.com> wrote:
>>>
 Thanks, Bowen, don't think that's an issue - but yes I can try
 upgrading to 3.11.5 and limit the merkle tree size to bring down the memory
 utilization.

 Thanks, Erick, let me try that.

 Can someone please share documentation relating to internal functioning
 of full repairs - if there exists one? Wanted to understand the role of the
 heap and off-heap memory separately during the process.

 Also, for my case, once the nodes reach the 95% memory usage, it stays
 there for almost 10-12 hours after the repair is complete, before falling
 back to 65%. Any pointers on what might be consuming off-heap for so long
 and can something be done to clear it earlier?

 Thanks,
 Aman



>>>
>>> --
>>> Regards,
>>> Aman
>>>
>>

Re: High memory usage during nodetool repair

[Jim Shaw]

I think Erick posted https://community.datastax.com/questions/6947/.
explained very clearly.

We hit same issue only on a huge table when upgrade, and we changed back
after done.
My understanding,  Which option to chose,  shall depend on your user case.
If chasing high performance on a big table, then go default one, and
increase capacity in memory, nowadays hardware is cheaper.

Thanks,
Jim

On Mon, Aug 2, 2021 at 7:12 PM Amandeep Srivastava <
amandeep.srivastava1...@gmail.com> wrote:

> Can anyone please help with the above questions? To summarise:
>
> 1) What is the impact of using mmap only for indices besides a degradation
> in read performance?
> 2) Why does the off heap consumed during Cassandra full repair remains
> occupied 12+ hours after the repair completion and is there a
> manual/configuration driven way to clear that earlier?
>
> Thanks,
> Aman
>
> On Thu, 29 Jul, 2021, 6:47 pm Amandeep Srivastava, <
> amandeep.srivastava1...@gmail.com> wrote:
>
>> Hi Erick,
>>
>> Limiting mmap to index only seems to have resolved the issue. The max ram
>> usage remained at 60% this time. Could you please point me to the
>> limitations for setting this param? - For starters, I can see read
>> performance getting reduced up to 30% (CASSANDRA-8464
>> )
>>
>> Also if you could please shed light on extended questions in my earlier
>> email.
>>
>> Thanks a lot.
>>
>> Regards,
>> Aman
>>
>> On Thu, Jul 29, 2021 at 12:52 PM Amandeep Srivastava <
>> amandeep.srivastava1...@gmail.com> wrote:
>>
>>> Thanks, Bowen, don't think that's an issue - but yes I can try upgrading
>>> to 3.11.5 and limit the merkle tree size to bring down the memory
>>> utilization.
>>>
>>> Thanks, Erick, let me try that.
>>>
>>> Can someone please share documentation relating to internal functioning
>>> of full repairs - if there exists one? Wanted to understand the role of the
>>> heap and off-heap memory separately during the process.
>>>
>>> Also, for my case, once the nodes reach the 95% memory usage, it stays
>>> there for almost 10-12 hours after the repair is complete, before falling
>>> back to 65%. Any pointers on what might be consuming off-heap for so long
>>> and can something be done to clear it earlier?
>>>
>>> Thanks,
>>> Aman
>>>
>>>
>>>
>>
>> --
>> Regards,
>> Aman
>>
>

Re: High memory usage during nodetool repair

[manish khandelwal]

Missed the heap part, not sure why is that happening

On Tue, Aug 3, 2021 at 8:59 AM manish khandelwal <
manishkhandelwa...@gmail.com> wrote:

> mmap is used for faster reads and as you guessed right you might see read
> performance degradation. If you are seeing high memory usage after repairs
> due to mmaped files, the only way to reduce the memory usage is to trigger
> some other process which requires memory. *mmapped* files use
> buffer/cache memory which gets released as soon as some other process
> requests memory from the kernel. Kernel does not want to waste its effort
> until some request for resource comes in.
>
>
>
> On Tue, Aug 3, 2021 at 4:42 AM Amandeep Srivastava <
> amandeep.srivastava1...@gmail.com> wrote:
>
>> Can anyone please help with the above questions? To summarise:
>>
>> 1) What is the impact of using mmap only for indices besides a
>> degradation in read performance?
>> 2) Why does the off heap consumed during Cassandra full repair remains
>> occupied 12+ hours after the repair completion and is there a
>> manual/configuration driven way to clear that earlier?
>>
>> Thanks,
>> Aman
>>
>> On Thu, 29 Jul, 2021, 6:47 pm Amandeep Srivastava, <
>> amandeep.srivastava1...@gmail.com> wrote:
>>
>>> Hi Erick,
>>>
>>> Limiting mmap to index only seems to have resolved the issue. The max
>>> ram usage remained at 60% this time. Could you please point me to the
>>> limitations for setting this param? - For starters, I can see read
>>> performance getting reduced up to 30% (CASSANDRA-8464
>>> )
>>>
>>> Also if you could please shed light on extended questions in my earlier
>>> email.
>>>
>>> Thanks a lot.
>>>
>>> Regards,
>>> Aman
>>>
>>> On Thu, Jul 29, 2021 at 12:52 PM Amandeep Srivastava <
>>> amandeep.srivastava1...@gmail.com> wrote:
>>>
 Thanks, Bowen, don't think that's an issue - but yes I can try
 upgrading to 3.11.5 and limit the merkle tree size to bring down the memory
 utilization.

 Thanks, Erick, let me try that.

 Can someone please share documentation relating to internal functioning
 of full repairs - if there exists one? Wanted to understand the role of the
 heap and off-heap memory separately during the process.

 Also, for my case, once the nodes reach the 95% memory usage, it stays
 there for almost 10-12 hours after the repair is complete, before falling
 back to 65%. Any pointers on what might be consuming off-heap for so long
 and can something be done to clear it earlier?

 Thanks,
 Aman



>>>
>>> --
>>> Regards,
>>> Aman
>>>
>>

Re: High memory usage during nodetool repair

[manish khandelwal]

mmap is used for faster reads and as you guessed right you might see read
performance degradation. If you are seeing high memory usage after repairs
due to mmaped files, the only way to reduce the memory usage is to trigger
some other process which requires memory. *mmapped* files use buffer/cache
memory which gets released as soon as some other process requests memory
from the kernel. Kernel does not want to waste its effort until some
request for resource comes in.



On Tue, Aug 3, 2021 at 4:42 AM Amandeep Srivastava <
amandeep.srivastava1...@gmail.com> wrote:

> Can anyone please help with the above questions? To summarise:
>
> 1) What is the impact of using mmap only for indices besides a degradation
> in read performance?
> 2) Why does the off heap consumed during Cassandra full repair remains
> occupied 12+ hours after the repair completion and is there a
> manual/configuration driven way to clear that earlier?
>
> Thanks,
> Aman
>
> On Thu, 29 Jul, 2021, 6:47 pm Amandeep Srivastava, <
> amandeep.srivastava1...@gmail.com> wrote:
>
>> Hi Erick,
>>
>> Limiting mmap to index only seems to have resolved the issue. The max ram
>> usage remained at 60% this time. Could you please point me to the
>> limitations for setting this param? - For starters, I can see read
>> performance getting reduced up to 30% (CASSANDRA-8464
>> )
>>
>> Also if you could please shed light on extended questions in my earlier
>> email.
>>
>> Thanks a lot.
>>
>> Regards,
>> Aman
>>
>> On Thu, Jul 29, 2021 at 12:52 PM Amandeep Srivastava <
>> amandeep.srivastava1...@gmail.com> wrote:
>>
>>> Thanks, Bowen, don't think that's an issue - but yes I can try upgrading
>>> to 3.11.5 and limit the merkle tree size to bring down the memory
>>> utilization.
>>>
>>> Thanks, Erick, let me try that.
>>>
>>> Can someone please share documentation relating to internal functioning
>>> of full repairs - if there exists one? Wanted to understand the role of the
>>> heap and off-heap memory separately during the process.
>>>
>>> Also, for my case, once the nodes reach the 95% memory usage, it stays
>>> there for almost 10-12 hours after the repair is complete, before falling
>>> back to 65%. Any pointers on what might be consuming off-heap for so long
>>> and can something be done to clear it earlier?
>>>
>>> Thanks,
>>> Aman
>>>
>>>
>>>
>>
>> --
>> Regards,
>> Aman
>>
>

Re: High memory usage during nodetool repair

[Amandeep Srivastava]

Can anyone please help with the above questions? To summarise:

1) What is the impact of using mmap only for indices besides a degradation
in read performance?
2) Why does the off heap consumed during Cassandra full repair remains
occupied 12+ hours after the repair completion and is there a
manual/configuration driven way to clear that earlier?

Thanks,
Aman

On Thu, 29 Jul, 2021, 6:47 pm Amandeep Srivastava, <
amandeep.srivastava1...@gmail.com> wrote:

> Hi Erick,
>
> Limiting mmap to index only seems to have resolved the issue. The max ram
> usage remained at 60% this time. Could you please point me to the
> limitations for setting this param? - For starters, I can see read
> performance getting reduced up to 30% (CASSANDRA-8464
> )
>
> Also if you could please shed light on extended questions in my earlier
> email.
>
> Thanks a lot.
>
> Regards,
> Aman
>
> On Thu, Jul 29, 2021 at 12:52 PM Amandeep Srivastava <
> amandeep.srivastava1...@gmail.com> wrote:
>
>> Thanks, Bowen, don't think that's an issue - but yes I can try upgrading
>> to 3.11.5 and limit the merkle tree size to bring down the memory
>> utilization.
>>
>> Thanks, Erick, let me try that.
>>
>> Can someone please share documentation relating to internal functioning
>> of full repairs - if there exists one? Wanted to understand the role of the
>> heap and off-heap memory separately during the process.
>>
>> Also, for my case, once the nodes reach the 95% memory usage, it stays
>> there for almost 10-12 hours after the repair is complete, before falling
>> back to 65%. Any pointers on what might be consuming off-heap for so long
>> and can something be done to clear it earlier?
>>
>> Thanks,
>> Aman
>>
>>
>>
>
> --
> Regards,
> Aman
>

Re: High memory usage during nodetool repair

[Amandeep Srivastava]

Hi Erick,

Limiting mmap to index only seems to have resolved the issue. The max ram
usage remained at 60% this time. Could you please point me to the
limitations for setting this param? - For starters, I can see read
performance getting reduced up to 30% (CASSANDRA-8464
)

Also if you could please shed light on extended questions in my earlier
email.

Thanks a lot.

Regards,
Aman

On Thu, Jul 29, 2021 at 12:52 PM Amandeep Srivastava <
amandeep.srivastava1...@gmail.com> wrote:

> Thanks, Bowen, don't think that's an issue - but yes I can try upgrading
> to 3.11.5 and limit the merkle tree size to bring down the memory
> utilization.
>
> Thanks, Erick, let me try that.
>
> Can someone please share documentation relating to internal functioning of
> full repairs - if there exists one? Wanted to understand the role of the
> heap and off-heap memory separately during the process.
>
> Also, for my case, once the nodes reach the 95% memory usage, it stays
> there for almost 10-12 hours after the repair is complete, before falling
> back to 65%. Any pointers on what might be consuming off-heap for so long
> and can something be done to clear it earlier?
>
> Thanks,
> Aman
>
>
>

-- 
Regards,
Aman

Re: High memory usage during nodetool repair

[Amandeep Srivastava]

Thanks, Bowen, don't think that's an issue - but yes I can try upgrading to
3.11.5 and limit the merkle tree size to bring down the memory utilization.

Thanks, Erick, let me try that.

Can someone please share documentation relating to internal functioning of
full repairs - if there exists one? Wanted to understand the role of the
heap and off-heap memory separately during the process.

Also, for my case, once the nodes reach the 95% memory usage, it stays
there for almost 10-12 hours after the repair is complete, before falling
back to 65%. Any pointers on what might be consuming off-heap for so long
and can something be done to clear it earlier?

Thanks,
Aman

Re: High memory usage during nodetool repair

[Erick Ramirez]

Based on the symptoms you described, it's most likely caused by SSTables
being mmap()ed as part of the repairs.

Set `disk_access_mode: mmap_index_only` so only index files get mapped and
not the data files. I've explained it in a bit more detail in this article
-- https://community.datastax.com/questions/6947/. Cheers!

>

Re: High memory usage during nodetool repair

[Bowen Song]

Could it be related to 
https://issues.apache.org/jira/browse/CASSANDRA-14096 ?


On 28/07/2021 13:55, Amandeep Srivastava wrote:

Hi team,

My Cluster configs: DC1 - 9 nodes, DC2 - 4 nodes
Node configs: 12 core x 96GB ram x 1 TB HDD
Repair params: -full -pr -local
Cassandra version: 3.11.4

I'm running a full repair on DC2 nodes - one node and one keyspace at 
a time. During the repair, ram usage on all 4 nodes spike up to 95%. 
Have tried setting xmx to 24GB, 31GB, 32GB, and 64GB but all show the 
same behavior. Could you please advise what might be consuming such 
high levels of off-heap memory and what can be done to cap/limit its use?


We run no other process apart from Cassandra daemon on these nodes.

Regards,
Aman

High memory usage during nodetool repair

[Amandeep Srivastava]

Hi team,

My Cluster configs: DC1 - 9 nodes, DC2 - 4 nodes
Node configs: 12 core x 96GB ram x 1 TB HDD
Repair params: -full -pr -local
Cassandra version: 3.11.4

I'm running a full repair on DC2 nodes - one node and one keyspace at a
time. During the repair, ram usage on all 4 nodes spike up to 95%. Have
tried setting xmx to 24GB, 31GB, 32GB, and 64GB but all show the same
behavior. Could you please advise what might be consuming such high levels
of off-heap memory and what can be done to cap/limit its use?

We run no other process apart from Cassandra daemon on these nodes.

Regards,
Aman

Re: How to predict time to complete for nodetool repair

[Alexander DEJANOVSKI]

Also Reaper will skip the anticompaction phase which you might be going
through with nodetool (depending on your version of Cassandra).
That'll reduce the overall time spent on repair and will remove some
compaction pressure.

But as Erick said, unless you have past repairs to rely on and a stable
data size, it is impossible to predict the time it takes for repair to
complete.

Cheers,

Alex

Le lun. 23 mars 2020 à 12:44, Oleksandr Shulgin <
oleksandr.shul...@zalando.de> a écrit :

> On Mon, Mar 23, 2020 at 5:49 AM Shishir Kumar 
> wrote:
>
>> Hi,
>>
>> Is it possible to get/predict how much time it will take for *nodetool
>> -pr *to complete on a node? Currently in one of my env (~800GB data per
>> node in 6 node cluster), it is running since last 3 days.
>>
>
> Cassandra Reaper used to provide a reasonably accurate estimate as I
> recall.  Of course, the repair has to be triggered by Reaper itself--it's
> no use if you have already started it with nodetool.
>
> Regards,
> --
> Alex
>
>

Re: How to predict time to complete for nodetool repair

[Oleksandr Shulgin]

On Mon, Mar 23, 2020 at 5:49 AM Shishir Kumar 
wrote:

> Hi,
>
> Is it possible to get/predict how much time it will take for *nodetool
> -pr *to complete on a node? Currently in one of my env (~800GB data per
> node in 6 node cluster), it is running since last 3 days.
>

Cassandra Reaper used to provide a reasonably accurate estimate as I
recall.  Of course, the repair has to be triggered by Reaper itself--it's
no use if you have already started it with nodetool.

Regards,
--
Alex

Re: How to predict time to complete for nodetool repair

[Erick Ramirez]

There's a lot of moving parts with repairs and how long it takes depends on
various factors including (but not limited to):

   - how busy the nodes are
   - how fast the CPUs are
   - how fast the disks are
   - how much network bandwidth is available
   - how much data needs to be repaired

It's more art than science trying to predict it but you get closer to
science the more successful runs you've had meaning you make an educated
guesstimate based on the previous repair runs. The very first run is always
pretty bad and consider it an outlier but use the next successive runs as
indicators and you should be able to extrapolate from there.

Finally, be careful when comparing runs during low traffic periods of the
month and primetime/peak. Repairs will run faster when the cluster is not
busy and will perform horrendously at peak times since it's competing for
the same resources as normal app traffic. Cheers!

How to predict time to complete for nodetool repair

[Shishir Kumar]

Hi,

Is it possible to get/predict how much time it will take for *nodetool -pr *to
complete on a node? Currently in one of my env (~800GB data per node in 6
node cluster), it is running since last 3 days.

Regards,
Shishir

Re: nodetool repair failing with "Validation failed in /X.X.X.X

[Rhys Campbell]

Hello Shalom,

Someone already tried a rolling restart of Cassandra. I will probably try
rebooting the OS.

Repair seems to work if you do it a keyspace at a time.

Thanks for your input.

Rhys

On Sun, May 5, 2019 at 2:14 PM shalom sagges  wrote:

> Hi Rhys,
>
> I encountered this error after adding new SSTables to a cluster and
> running nodetool refresh (v3.0.12).
> The refresh worked, but after starting repairs on the cluster, I got the
> "Validation failed in /X.X.X.X" error on the remote DC.
> A rolling restart solved the issue for me.
>
> Hope this helps!
>
>
>
> On Sat, May 4, 2019 at 3:58 PM Rhys Campbell
>  wrote:
>
>>
>> > Hello,
>> >
>> > I’m having issues running repair on an Apache Cassandra Cluster. I’m
>> getting "Failed creating a merkle tree“ errors on the replication partner
>> nodes. Anyone have any experience of this? I am running 2.2.13.
>> >
>> > Further details here…
>> https://issues.apache.org/jira/projects/CASSANDRA/issues/CASSANDRA-15109?filter=allopenissues
>> >
>> > Best,
>> >
>> > Rhys
>>
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>>

Re: nodetool repair failing with "Validation failed in /X.X.X.X

[shalom sagges]

Hi Rhys,

I encountered this error after adding new SSTables to a cluster and running
nodetool refresh (v3.0.12).
The refresh worked, but after starting repairs on the cluster, I got the
"Validation failed in /X.X.X.X" error on the remote DC.
A rolling restart solved the issue for me.

Hope this helps!



On Sat, May 4, 2019 at 3:58 PM Rhys Campbell
 wrote:

>
> > Hello,
> >
> > I’m having issues running repair on an Apache Cassandra Cluster. I’m
> getting "Failed creating a merkle tree“ errors on the replication partner
> nodes. Anyone have any experience of this? I am running 2.2.13.
> >
> > Further details here…
> https://issues.apache.org/jira/projects/CASSANDRA/issues/CASSANDRA-15109?filter=allopenissues
> >
> > Best,
> >
> > Rhys
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

nodetool repair failing with "Validation failed in /X.X.X.X

[Rhys Campbell]

> Hello,
> 
> I’m having issues running repair on an Apache Cassandra Cluster. I’m getting 
> "Failed creating a merkle tree“ errors on the replication partner nodes. 
> Anyone have any experience of this? I am running 2.2.13.
> 
> Further details here… 
> https://issues.apache.org/jira/projects/CASSANDRA/issues/CASSANDRA-15109?filter=allopenissues
> 
> Best,
> 
> Rhys


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: time tracking for down node for nodetool repair

[Kunal]

Thanks everyone for your valuable suggestion.  Really appreciate it


Regards,
Kunal Vaid

On Mon, Apr 8, 2019 at 7:41 PM Nitan Kainth  wrote:

> Valid suggestion. Stick to the plan, avoid downtime of a node more than
> hinted handoff window. OR increase window to a larger value, if you know it
> is going to take longer than current setting
>
>
> Regards,
>
> Nitan
>
> Cell: 510 449 9629
>
> On Apr 8, 2019, at 8:43 PM, Soumya Jena 
> wrote:
>
> Cassandra tracks it and no new hints will be created once the default 3
> hours window is passed  . However , cassandra will not automatically
> trigger a repair if your node is down for more than 3 hours .Default
> settings of 3 hours for hints is defined in cassandra.yaml file . Look for
> "max_hint_window_in_ms" in the cassandra.yaml file. Its configurable .
> Apart from the periodic repair you should start a repair when you bring up
> a node which has missed some writes .
>
> One more thing is  if node is down for long time and missed a lot of
> writes sometimes it may be better to add that as a new fresh node rather
> than adding it and then doing repair .
>
> On Mon, Apr 8, 2019 at 4:49 PM Stefan Miklosovic <
> stefan.mikloso...@instaclustr.com> wrote:
>
>> Ah I see it is the default for hinted handoffs. I was somehow thinking
>> its bigger figure I do not know why :)
>>
>> I would say you should run repairs continuously / periodically so you
>> would not even have to do some thinking about that and it should run
>> in the background in a scheduled manner if possible.
>>
>> Regards
>>
>> On Tue, 9 Apr 2019 at 04:19, Kunal  wrote:
>> >
>> > Hello everyone..
>> >
>> >
>> >
>> > I have a 6 node Cassandra datacenter, 3 nodes on each datacenter. If
>> one of the node goes down and remain down for more than 3 hr, I have to run
>> nodetool repair. Just wanted to ask if Cassandra  automatically tracks the
>> time when one of the Cassandra node goes down or do I need to write code to
>> track the time and run repair when node comes back online after 3 hrs.
>> >
>> >
>> > Thanks in anticipation.
>> >
>> > Regards,
>> > Kunal Vaid
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>> --



Regards,
Kunal Vaid

Re: time tracking for down node for nodetool repair

[Nitan Kainth]

Valid suggestion. Stick to the plan, avoid downtime of a node more than hinted 
handoff window. OR increase window to a larger value, if you know it is going 
to take longer than current setting


Regards,
Nitan
Cell: 510 449 9629

> On Apr 8, 2019, at 8:43 PM, Soumya Jena  wrote:
> 
> Cassandra tracks it and no new hints will be created once the default 3 hours 
> window is passed  . However , cassandra will not automatically trigger a 
> repair if your node is down for more than 3 hours .Default settings of 3 
> hours for hints is defined in cassandra.yaml file . Look for 
> "max_hint_window_in_ms" in the cassandra.yaml file. Its configurable . Apart 
> from the periodic repair you should start a repair when you bring up a node 
> which has missed some writes .   
> 
> One more thing is  if node is down for long time and missed a lot of writes 
> sometimes it may be better to add that as a new fresh node rather than adding 
> it and then doing repair .
> 
>> On Mon, Apr 8, 2019 at 4:49 PM Stefan Miklosovic 
>>  wrote:
>> Ah I see it is the default for hinted handoffs. I was somehow thinking
>> its bigger figure I do not know why :)
>> 
>> I would say you should run repairs continuously / periodically so you
>> would not even have to do some thinking about that and it should run
>> in the background in a scheduled manner if possible.
>> 
>> Regards
>> 
>> On Tue, 9 Apr 2019 at 04:19, Kunal  wrote:
>> >
>> > Hello everyone..
>> >
>> >
>> >
>> > I have a 6 node Cassandra datacenter, 3 nodes on each datacenter. If one 
>> > of the node goes down and remain down for more than 3 hr, I have to run 
>> > nodetool repair. Just wanted to ask if Cassandra  automatically tracks the 
>> > time when one of the Cassandra node goes down or do I need to write code 
>> > to track the time and run repair when node comes back online after 3 hrs.
>> >
>> >
>> > Thanks in anticipation.
>> >
>> > Regards,
>> > Kunal Vaid
>> 
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>> 

Re: time tracking for down node for nodetool repair

[Soumya Jena]

Cassandra tracks it and no new hints will be created once the default 3
hours window is passed  . However , cassandra will not automatically
trigger a repair if your node is down for more than 3 hours .Default
settings of 3 hours for hints is defined in cassandra.yaml file . Look for
"max_hint_window_in_ms" in the cassandra.yaml file. Its configurable .
Apart from the periodic repair you should start a repair when you bring up
a node which has missed some writes .

One more thing is  if node is down for long time and missed a lot of writes
sometimes it may be better to add that as a new fresh node rather than
adding it and then doing repair .

On Mon, Apr 8, 2019 at 4:49 PM Stefan Miklosovic <
stefan.mikloso...@instaclustr.com> wrote:

> Ah I see it is the default for hinted handoffs. I was somehow thinking
> its bigger figure I do not know why :)
>
> I would say you should run repairs continuously / periodically so you
> would not even have to do some thinking about that and it should run
> in the background in a scheduled manner if possible.
>
> Regards
>
> On Tue, 9 Apr 2019 at 04:19, Kunal  wrote:
> >
> > Hello everyone..
> >
> >
> >
> > I have a 6 node Cassandra datacenter, 3 nodes on each datacenter. If one
> of the node goes down and remain down for more than 3 hr, I have to run
> nodetool repair. Just wanted to ask if Cassandra  automatically tracks the
> time when one of the Cassandra node goes down or do I need to write code to
> track the time and run repair when node comes back online after 3 hrs.
> >
> >
> > Thanks in anticipation.
> >
> > Regards,
> > Kunal Vaid
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Re: time tracking for down node for nodetool repair

[Stefan Miklosovic]

Ah I see it is the default for hinted handoffs. I was somehow thinking
its bigger figure I do not know why :)

I would say you should run repairs continuously / periodically so you
would not even have to do some thinking about that and it should run
in the background in a scheduled manner if possible.

Regards

On Tue, 9 Apr 2019 at 04:19, Kunal  wrote:
>
> Hello everyone..
>
>
>
> I have a 6 node Cassandra datacenter, 3 nodes on each datacenter. If one of 
> the node goes down and remain down for more than 3 hr, I have to run nodetool 
> repair. Just wanted to ask if Cassandra  automatically tracks the time when 
> one of the Cassandra node goes down or do I need to write code to track the 
> time and run repair when node comes back online after 3 hrs.
>
>
> Thanks in anticipation.
>
> Regards,
> Kunal Vaid

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: time tracking for down node for nodetool repair

[Stefan Miklosovic]

Hi Kunal,

where do you have that "more than 3 hours" from?

Regards

On Tue, 9 Apr 2019 at 04:19, Kunal  wrote:
>
> Hello everyone..
>
>
>
> I have a 6 node Cassandra datacenter, 3 nodes on each datacenter. If one of 
> the node goes down and remain down for more than 3 hr, I have to run nodetool 
> repair. Just wanted to ask if Cassandra  automatically tracks the time when 
> one of the Cassandra node goes down or do I need to write code to track the 
> time and run repair when node comes back online after 3 hrs.
>
>
> Thanks in anticipation.
>
> Regards,
> Kunal Vaid

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

time tracking for down node for nodetool repair

[Kunal]

Hello everyone..



I have a 6 node Cassandra datacenter, 3 nodes on each datacenter. If one of
the node goes down and remain down for more than 3 hr, I have to run
nodetool repair. Just wanted to ask if Cassandra  automatically tracks the
time when one of the Cassandra node goes down or do I need to write code to
track the time and run repair when node comes back online after 3 hrs.

Thanks in anticipation.

Regards,
Kunal Vaid

Re: data consistency without using nodetool repair

[Jeff Jirsa]

If you have two replicas A and B, and you write at ONE

A acknowledged the write, B is running but drops write, the write succeeds

A fails 30 seconds later. The data is lost because it had no chance to be 
repaired to the other host via hints / repair / readrepair

10 days is a big window - repair is a smaller problem than RF=2 and writing at 
ONE
-- 
Jeff Jirsa


> On Jun 9, 2018, at 10:52 PM, onmstester onmstester  
> wrote:
> 
> Thanks Jeff,
> If i run repair every 10 days, then, would there be a chance of losing data 
> by losing one node (data inserted exactly after last repair) ? 
> 
> Sent using Zoho Mail
> 
> 
> 
>  On Sun, 10 Jun 2018 10:14:46 +0430 Jeff Jirsa  wrote 
> 
> 
> 
> 
> 
> On Jun 9, 2018, at 10:28 PM, onmstester onmstester  
> wrote:
> 
> I'm using RF=2 (i know it should be at least 3 but i'm short of resources) 
> and WCl=ONE and RCL=ONE in a cluster of 10 nodes in a insert-only scenario. 
> The problem: i dont want to use nodetool repair because it would put hige 
> load on my cluster for a long time, but also i need data consistency
> and fault tolerance in a way that:
> if one of my nodes fails:
> 1. there would be no single record data loss
> 
> This requires write > 1
> 
> 2. write/read of data would be continued with no problem
> 
> 
> This requires more replicas in the ring than the number of replicas requires 
> for reads and writes
> 
> 
> I know that current config won't satisfy No.1, so changed the Write 
> Consistensy Level to ALL and to satisfy No.2, i'm catching exceptions of
> "1 replicas needed but 2 required", write those records again with WCL = ONE 
> and put them somewhere for rewrite later with WCL=2.
> Is there anything wrong with this workaround? any better solution (Strong 
> requirements: I can't change the RF, The system should tolerate 1 node 
> failure with no data loss and no read/write failure) 
> 
> 
> Sorta works, but it’s dirty and a lot of edge cases. Depending on the budget 
> and value of data, maybe it’s ok, but I wouldn’t trust it to be good enough 
> for critical use cases
> 
> The transient replication work that Ariel is doing 
> (https://issues.apache.org/jira/browse/CASSANDRA-14404) will likely benefit 
> you here in 4.0 (but it will require you to run repair).
> 
> 

Re: data consistency without using nodetool repair

[onmstester onmstester]

Thanks Jeff,

If i run repair every 10 days, then, would there be a chance of losing data by 
losing one node (data inserted exactly after last repair) ? 


Sent using Zoho Mail






 On Sun, 10 Jun 2018 10:14:46 +0430 Jeff Jirsa jji...@gmail.com 
wrote 











On Jun 9, 2018, at 10:28 PM, onmstester onmstester onmstes...@zoho.com 
wrote:




I'm using RF=2 (i know it should be at least 3 but i'm short of resources) and 
WCl=ONE and RCL=ONE in a cluster of 10 nodes in a insert-only scenario. 

The problem: i dont want to use nodetool repair because it would put hige load 
on my cluster for a long time, but also i need data consistency

and fault tolerance in a way that:

if one of my nodes fails:

1. there would be no single record data loss





This requires write  1



2. write/read of data would be continued with no problem







This requires more replicas in the ring than the number of replicas requires 
for reads and writes





I know that current config won't satisfy No.1, so changed the Write Consistensy 
Level to ALL and to satisfy No.2, i'm catching exceptions of

"1 replicas needed but 2 required", write those records again with WCL = ONE 
and put them somewhere for rewrite later with WCL=2.



Is there anything wrong with this workaround? any better solution (Strong 
requirements: I can't change the RF, The system should tolerate 1 node failure 
with no data loss and no read/write failure) 







Sorta works, but it’s dirty and a lot of edge cases. Depending on the budget 
and value of data, maybe it’s ok, but I wouldn’t trust it to be good enough for 
critical use cases



The transient replication work that Ariel is doing 
(https://issues.apache.org/jira/browse/CASSANDRA-14404) will likely benefit you 
here in 4.0 (but it will require you to run repair).

Re: data consistency without using nodetool repair

[Jeff Jirsa]

> On Jun 9, 2018, at 10:28 PM, onmstester onmstester  
> wrote:
> 
> 
> I'm using RF=2 (i know it should be at least 3 but i'm short of resources) 
> and WCl=ONE and RCL=ONE in a cluster of 10 nodes in a insert-only scenario. 
> The problem: i dont want to use nodetool repair because it would put hige 
> load on my cluster for a long time, but also i need data consistency
> and fault tolerance in a way that:
> if one of my nodes fails:
> 1. there would be no single record data loss

This requires write > 1

> 2. write/read of data would be continued with no problem
> 

This requires more replicas in the ring than the number of replicas requires 
for reads and writes


> I know that current config won't satisfy No.1, so changed the Write 
> Consistensy Level to ALL and to satisfy No.2, i'm catching exceptions of
> "1 replicas needed but 2 required", write those records again with WCL = ONE 
> and put them somewhere for rewrite later with WCL=2.
> Is there anything wrong with this workaround? any better solution (Strong 
> requirements: I can't change the RF, The system should tolerate 1 node 
> failure with no data loss and no read/write failure) 


Sorta works, but it’s dirty and a lot of edge cases. Depending on the budget 
and value of data, maybe it’s ok, but I wouldn’t trust it to be good enough for 
critical use cases

The transient replication work that Ariel is doing 
(https://issues.apache.org/jira/browse/CASSANDRA-14404) will likely benefit you 
here in 4.0 (but it will require you to run repair).

data consistency without using nodetool repair

[onmstester onmstester]

I'm using RF=2 (i know it should be at least 3 but i'm short of resources) and 
WCl=ONE and RCL=ONE in a cluster of 10 nodes in a insert-only scenario. 

The problem: i dont want to use nodetool repair because it would put hige load 
on my cluster for a long time, but also i need data consistency

and fault tolerance in a way that:

if one of my nodes fails:

1. there would be no single record data loss

2. write/read of data would be continued with no problem



I know that current config won't satisfy No.1, so changed the Write Consistensy 
Level to ALL and to satisfy No.2, i'm catching exceptions of

"1 replicas needed but 2 required", write those records again with WCL = ONE 
and put them somewhere for rewrite later with WCL=2.

Is there anything wrong with this workaround? any better solution (Strong 
requirements: I can't change the RF, The system should tolerate 1 node failure 
with no data loss and no read/write failure) ?

Sent using Zoho Mail

Re: nodetool repair -pr

[Arvinder Dhillon]

It depends on your data model. -pr only repair primary range. So if there
is a keyspace with replication 'DC2:3', and you run repair -pr only on all
nodes of DC1, it is not going to repair token ranges corsponding to DC2. So
you will have to run on each node.

-Arvinder

On Fri, Jun 8, 2018, 8:42 PM Igor Zubchenok  wrote:

> According docs at
> http://cassandra.apache.org/doc/latest/operating/repair.html?highlight=single
>
>
> *The -pr flag will only repair the “primary” ranges on a node, so you can
> repair your entire cluster by running nodetool repair -pr on each node in
> a single datacenter.*
> But I saw many places, where it is noted that I should run it at ALL data
> centers.
>
> Looking for a qualified answer.
>
>
> On Fri, 8 Jun 2018 at 18:08 Igor Zubchenok  wrote:
>
>> I want to repair all nodes at all data centers.
>>
>> Example:
>> DC1
>>  nodeA
>>  nodeB
>>  nodeC
>> DC2
>>  node D
>>  node E
>>  node F
>>
>> If I run `nodetool repair -pr` at nodeA nodeB and nodeC, will all ranges
>> be repaired?
>>
>>
>> On Fri, 8 Jun 2018 at 17:57 Rahul Singh 
>> wrote:
>>
>>> From DS dox : "Do not use -pr with this option to repair only a local
>>> data center."
>>> On Jun 8, 2018, 10:42 AM -0400, user@cassandra.apache.org, wrote:
>>>
>>>
>>> *nodetool repair -pr*
>>>
>>>

Re: nodetool repair -pr

[Igor Zubchenok]

According docs at
http://cassandra.apache.org/doc/latest/operating/repair.html?highlight=single


*The -pr flag will only repair the “primary” ranges on a node, so you can
repair your entire cluster by running nodetool repair -pr on each node in
a single datacenter.*
But I saw many places, where it is noted that I should run it at ALL data
centers.

Looking for a qualified answer.


On Fri, 8 Jun 2018 at 18:08 Igor Zubchenok  wrote:

> I want to repair all nodes at all data centers.
>
> Example:
> DC1
>  nodeA
>  nodeB
>  nodeC
> DC2
>  node D
>  node E
>  node F
>
> If I run `nodetool repair -pr` at nodeA nodeB and nodeC, will all ranges
> be repaired?
>
>
> On Fri, 8 Jun 2018 at 17:57 Rahul Singh 
> wrote:
>
>> From DS dox : "Do not use -pr with this option to repair only a local
>> data center."
>> On Jun 8, 2018, 10:42 AM -0400, user@cassandra.apache.org, wrote:
>>
>>
>> *nodetool repair -pr*
>>
>>

Re: nodetool repair -pr

[Igor Zubchenok]

I want to repair all nodes at all data centers.

Example:
DC1
 nodeA
 nodeB
 nodeC
DC2
 node D
 node E
 node F

If I run `nodetool repair -pr` at nodeA nodeB and nodeC, will all ranges be
repaired?


On Fri, 8 Jun 2018 at 17:57 Rahul Singh 
wrote:

> From DS dox : "Do not use -pr with this option to repair only a local
> data center."
> On Jun 8, 2018, 10:42 AM -0400, user@cassandra.apache.org, wrote:
>
>
> *nodetool repair -pr*
>
> --
Regards,
Igor Zubchenok

CTO at Multi Brains LLC
Founder of taxistartup.com saytaxi.com chauffy.com
Skype: igor.zubchenok

Re: nodetool repair -pr

[Rahul Singh]

>From DS dox : "Do not use -pr with this option to repair only a local data 
>center."
On Jun 8, 2018, 10:42 AM -0400, user@cassandra.apache.org, wrote:
>
> nodetool repair -pr

nodetool repair -pr

[Igor Zubchenok]

Hi!

I want to repair all nodes in all datacenters.
Should I run *nodetool repair -pr* at all nodes of a SINGLE datacenter or
at all nodes of ALL datacenters?
-- 
Regards,
Igor Zubchenok

CTO at Multi Brains LLC
Founder of taxistartup.com saytaxi.com chauffy.com
Skype: igor.zubchenok

Re: Nodetool repair multiple dc

[Abdul Patel]

One quick question on reaper ..what data is stored in reaper_db keyspace ?
And how much does it grow?
Do we have to cleanup that frequently or reaper has mechnism to slef clean ?

On Friday, April 13, 2018, Alexander Dejanovski <a...@thelastpickle.com>
wrote:

> Hi Abdul,
>
> Reaper has been used in production for several years now, by many
> companies.
> I've seen it handling 100s of clusters and 1000s of nodes with a single
> Reaper process.
> Check the docs on cassandra-reaper.io to see which architecture matches
> your cluster : http://cassandra-reaper.io/docs/usage/multi_dc/
>
> Cheers,
>
> On Fri, Apr 13, 2018 at 4:38 PM Rahul Singh <rahul.xavier.si...@gmail.com>
> wrote:
>
>> Makes sense it takes a long time since it has to reconcile against
>> replicas in all DCs. I leverage commercial tools for production clusters,
>> but I’m pretty sure Reaper is the best open source option. Otherwise you’ll
>> waste a lot of time trying to figure it out own your own. No need to
>> reinvent the wheel.
>>
>> On Apr 12, 2018, 11:02 PM -0400, Abdul Patel <abd786...@gmail.com>,
>> wrote:
>>
>> Hi All,
>>
>> I have 18 node cluster across 3 dc , if i tey to run incremental repair
>> on singke node it takes forever sometome 45 to 1hr and sometime times out
>> ..so i started running "nodetool repair -dc dc1" for each dc one by one
>> ..which works fine ..do we have an better way to handle this?
>> I am thinking abouy exploring cassandra reaper ..does anyone has used
>> that in prod?
>>
>> --
> -
> Alexander Dejanovski
> France
> @alexanderdeja
>
> Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>

Re: Nodetool repair multiple dc

[Alexander Dejanovski]

Hi Abdul,

Reaper has been used in production for several years now, by many companies.
I've seen it handling 100s of clusters and 1000s of nodes with a single
Reaper process.
Check the docs on cassandra-reaper.io to see which architecture matches
your cluster : http://cassandra-reaper.io/docs/usage/multi_dc/

Cheers,

On Fri, Apr 13, 2018 at 4:38 PM Rahul Singh <rahul.xavier.si...@gmail.com>
wrote:

> Makes sense it takes a long time since it has to reconcile against
> replicas in all DCs. I leverage commercial tools for production clusters,
> but I’m pretty sure Reaper is the best open source option. Otherwise you’ll
> waste a lot of time trying to figure it out own your own. No need to
> reinvent the wheel.
>
> On Apr 12, 2018, 11:02 PM -0400, Abdul Patel <abd786...@gmail.com>, wrote:
>
> Hi All,
>
> I have 18 node cluster across 3 dc , if i tey to run incremental repair on
> singke node it takes forever sometome 45 to 1hr and sometime times out ..so
> i started running "nodetool repair -dc dc1" for each dc one by one ..which
> works fine ..do we have an better way to handle this?
> I am thinking abouy exploring cassandra reaper ..does anyone has used that
> in prod?
>
> --
-
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

Re: Nodetool repair multiple dc

[Rahul Singh]

Makes sense it takes a long time since it has to reconcile against replicas in 
all DCs. I leverage commercial tools for production clusters, but I’m pretty 
sure Reaper is the best open source option. Otherwise you’ll waste a lot of 
time trying to figure it out own your own. No need to reinvent the wheel.

On Apr 12, 2018, 11:02 PM -0400, Abdul Patel <abd786...@gmail.com>, wrote:
> Hi All,
>
> I have 18 node cluster across 3 dc , if i tey to run incremental repair on 
> singke node it takes forever sometome 45 to 1hr and sometime times out ..so i 
> started running "nodetool repair -dc dc1" for each dc one by one ..which 
> works fine ..do we have an better way to handle this?
> I am thinking abouy exploring cassandra reaper ..does anyone has used that in 
> prod?

Nodetool repair multiple dc

[Abdul Patel]

Hi All,

I have 18 node cluster across 3 dc , if i tey to run incremental repair on
singke node it takes forever sometome 45 to 1hr and sometime times out ..so
i started running "nodetool repair -dc dc1" for each dc one by one ..which
works fine ..do we have an better way to handle this?
I am thinking abouy exploring cassandra reaper ..does anyone has used that
in prod?

答复: nodetool repair and compact

[Xiangfei Ni]

Hi James and Jon,
Thank you very much for  your advice, I will go through the article.
By the way,is there any official documentation for instructing the detail of 
nodetool command,the web pages in datastax are just brief instruction.

Best Regards,

倪项菲/ David Ni
中移德电网络科技有限公司
Virtue Intelligent Network Ltd, co.
Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
Mob: +86 13797007811|Tel: + 86 27 5024 2516

发件人: James Shaw <jxys...@gmail.com>
发送时间: 2018年4月2日 21:56
收件人: user@cassandra.apache.org
主题: Re: nodetool repair and compact

you may use:  nodetool upgradesstables -a keyspace_name table_name
it will re-write this table's sstable files to current version, while 
re-writing, will evit droppable tombstones (expired +  gc_grace_seconds 
(default 10 days) ), if partition cross different files, they will still be 
kept, but most droppable tombstones gone and size reduced.
It works well for ours.



On Mon, Apr 2, 2018 at 12:45 AM, Jon Haddad 
<j...@jonhaddad.com<mailto:j...@jonhaddad.com>> wrote:
You’ll find the answers to your questions (and quite a bit more) in this blog 
post from my coworker: 
http://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html

Repair doesn’t clean up tombstones, they’re only removed through compaction.  I 
advise taking care with nodetool compact, most of the time it’s not a great 
idea for a variety of reasons.  Check out the above post, if you still have 
questions, ask away.



On Apr 1, 2018, at 9:41 PM, Xiangfei Ni 
<xiangfei...@cm-dt.com<mailto:xiangfei...@cm-dt.com>> wrote:

Hi All,
  I want to delete the expired tombstone, someone uses nodetool repair ,but 
someone uses compact,so I want to know which one is the correct way,
  I have read the below pages from Datastax,but the page just tells us how to 
use the command,but doesn’t tell us what it is exactly dose,
  https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsRepair.html
   could anybody tell me how to clean the tombstone and give me some materials 
include the detailed instruction about the nodetool command and options?Web 
link is also ok.
  Thanks very much
Best Regards,

倪项菲/ David Ni
中移德电网络科技有限公司
Virtue Intelligent Network Ltd, co.
Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
Mob: +86 13797007811<tel:+86%20137%209700%207811>|Tel: + 86 27 5024 
2516<tel:+86%2027%205024%202516>

Re: nodetool repair and compact

[Alain RODRIGUEZ]

I have just this been told that my first statement is inaccurate:

If  'upgradesstable' is run as a routine operation, you might forget about
> it and suffer consequences. 'upgradesstable' is not only doing the
> compaction.


I should probably have checked upgradesstable closely before making this
statement and I definitely will.

Yet, I believe the second point still holds though: 'With UDC, you can
trigger the compaction of the sstables you want to remove the tombstones
from, instead of compacting *all* the sstables for a given table.'

C*heers,

2018-04-02 16:39 GMT+01:00 Alain RODRIGUEZ <arodr...@gmail.com>:

> Hi,
>
> it will re-write this table's sstable files to current version, while
>> re-writing, will evit droppable tombstones (expired +  gc_grace_seconds
>> (default 10 days) ), if partition cross different files, they will still
>> be kept, but most droppable tombstones gone and size reduced.
>>
>
> Nice tip James, I never thought about doing this, it could have been handy
> :).
>
> Now, these compactions can be automatically done using the proper
> tombstone compaction settings in most cases. Generally, tombstone
> compaction is enabled, but if tombstone eviction is still an issue, you
> might want to give a try enabling 'unchecked_tombstone_compaction' in the
> table options. This might claim quite a lot of disk space (depending on the
> sstable overlapping levels).
>
> In case manual action is really needed (even more if it is run
> automatically), I would recommend using 'User Defined Compactions' - UDC
> (accessible through JMX at least) instead of 'uprade sstable':
>
> - It will remove the tombstones the same way, but with no side effect if
> you are currently upgrading for example. If  'upgradesstable' is run as a
> routine operation, you might forget about it and suffer consequences.
> 'upgradesstable' is not only doing the compaction.
> - With UDC, you can trigger the compaction of the sstables you want to
> remove the tombstones from, instead of compacting *all* the sstables for a
> given table.
>
> This last point can prevent harming the cluster with useless compaction,
> and even allow the operator to do things like: 'Compact the 10% biggest
> sstables, that have an estimated tombstone ratio above 0.5, every day' or
> 'compact any sstable having more than 75% of tombstones' as you see fit,
> and using information such as the sstables sizes and sstablemetadata to get
> the tombstone ratio.
>
> C*heers,
> ---
> Alain Rodriguez - @arodream - al...@thelastpickle.com
> France / Spain
>
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> 2018-04-02 14:55 GMT+01:00 James Shaw <jxys...@gmail.com>:
>
>> you may use:  nodetool upgradesstables -a keyspace_name table_name
>> it will re-write this table's sstable files to current version, while
>> re-writing, will evit droppable tombstones (expired +  gc_grace_seconds
>> (default 10 days) ), if partition cross different files, they will still
>> be kept, but most droppable tombstones gone and size reduced.
>> It works well for ours.
>>
>>
>>
>> On Mon, Apr 2, 2018 at 12:45 AM, Jon Haddad <j...@jonhaddad.com> wrote:
>>
>>> You’ll find the answers to your questions (and quite a bit more) in this
>>> blog post from my coworker: http://thelastpickle
>>> .com/blog/2016/07/27/about-deletes-and-tombstones.html
>>>
>>> Repair doesn’t clean up tombstones, they’re only removed through
>>> compaction.  I advise taking care with nodetool compact, most of the time
>>> it’s not a great idea for a variety of reasons.  Check out the above post,
>>> if you still have questions, ask away.
>>>
>>>
>>> On Apr 1, 2018, at 9:41 PM, Xiangfei Ni <xiangfei...@cm-dt.com> wrote:
>>>
>>> Hi All,
>>>   I want to delete the expired tombstone, someone uses nodetool repair
>>> ,but someone uses compact,so I want to know which one is the correct way,
>>>   I have read the below pages from Datastax,but the page just tells us
>>> how to use the command,but doesn’t tell us what it is exactly dose,
>>>   https://docs.datastax.com/en/cassandra/3.0/cassandra/tools
>>> /toolsRepair.html
>>>could anybody tell me how to clean the tombstone and give me some
>>> materials include the detailed instruction about the nodetool command and
>>> options?Web link is also ok.
>>>   Thanks very much
>>> Best Regards,
>>>
>>> 倪项菲*/ **David Ni*
>>> 中移德电网络科技有限公司
>>>
>>> Virtue Intelligent Network Ltd, co.
>>> Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
>>> Mob: +86 13797007811 <+86%20137%209700%207811>|Tel: + 86 27 5024 2516
>>> <+86%2027%205024%202516>
>>>
>>>
>>>
>>
>

Re: nodetool repair and compact

[Alain RODRIGUEZ]

Hi,

it will re-write this table's sstable files to current version, while
> re-writing, will evit droppable tombstones (expired +  gc_grace_seconds
> (default 10 days) ), if partition cross different files, they will still
> be kept, but most droppable tombstones gone and size reduced.
>

Nice tip James, I never thought about doing this, it could have been handy
:).

Now, these compactions can be automatically done using the proper tombstone
compaction settings in most cases. Generally, tombstone compaction is
enabled, but if tombstone eviction is still an issue, you might want to
give a try enabling 'unchecked_tombstone_compaction' in the table options.
This might claim quite a lot of disk space (depending on the sstable
overlapping levels).

In case manual action is really needed (even more if it is run
automatically), I would recommend using 'User Defined Compactions' - UDC
(accessible through JMX at least) instead of 'uprade sstable':

- It will remove the tombstones the same way, but with no side effect if
you are currently upgrading for example. If  'upgradesstable' is run as a
routine operation, you might forget about it and suffer consequences.
'upgradesstable' is not only doing the compaction.
- With UDC, you can trigger the compaction of the sstables you want to
remove the tombstones from, instead of compacting *all* the sstables for a
given table.

This last point can prevent harming the cluster with useless compaction,
and even allow the operator to do things like: 'Compact the 10% biggest
sstables, that have an estimated tombstone ratio above 0.5, every day' or
'compact any sstable having more than 75% of tombstones' as you see fit,
and using information such as the sstables sizes and sstablemetadata to get
the tombstone ratio.

C*heers,
---
Alain Rodriguez - @arodream - al...@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2018-04-02 14:55 GMT+01:00 James Shaw <jxys...@gmail.com>:

> you may use:  nodetool upgradesstables -a keyspace_name table_name
> it will re-write this table's sstable files to current version, while
> re-writing, will evit droppable tombstones (expired +  gc_grace_seconds
> (default 10 days) ), if partition cross different files, they will still
> be kept, but most droppable tombstones gone and size reduced.
> It works well for ours.
>
>
>
> On Mon, Apr 2, 2018 at 12:45 AM, Jon Haddad <j...@jonhaddad.com> wrote:
>
>> You’ll find the answers to your questions (and quite a bit more) in this
>> blog post from my coworker: http://thelastpickle
>> .com/blog/2016/07/27/about-deletes-and-tombstones.html
>>
>> Repair doesn’t clean up tombstones, they’re only removed through
>> compaction.  I advise taking care with nodetool compact, most of the time
>> it’s not a great idea for a variety of reasons.  Check out the above post,
>> if you still have questions, ask away.
>>
>>
>> On Apr 1, 2018, at 9:41 PM, Xiangfei Ni <xiangfei...@cm-dt.com> wrote:
>>
>> Hi All,
>>   I want to delete the expired tombstone, someone uses nodetool repair
>> ,but someone uses compact,so I want to know which one is the correct way,
>>   I have read the below pages from Datastax,but the page just tells us
>> how to use the command,but doesn’t tell us what it is exactly dose,
>>   https://docs.datastax.com/en/cassandra/3.0/cassandra/tools
>> /toolsRepair.html
>>could anybody tell me how to clean the tombstone and give me some
>> materials include the detailed instruction about the nodetool command and
>> options?Web link is also ok.
>>   Thanks very much
>> Best Regards,
>>
>> 倪项菲*/ **David Ni*
>> 中移德电网络科技有限公司
>>
>> Virtue Intelligent Network Ltd, co.
>> Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
>> Mob: +86 13797007811 <+86%20137%209700%207811>|Tel: + 86 27 5024 2516
>> <+86%2027%205024%202516>
>>
>>
>>
>

Re: nodetool repair and compact

[James Shaw]

you may use:  nodetool upgradesstables -a keyspace_name table_name
it will re-write this table's sstable files to current version, while
re-writing, will evit droppable tombstones (expired +  gc_grace_seconds
(default 10 days) ), if partition cross different files, they will still be
kept, but most droppable tombstones gone and size reduced.
It works well for ours.



On Mon, Apr 2, 2018 at 12:45 AM, Jon Haddad <j...@jonhaddad.com> wrote:

> You’ll find the answers to your questions (and quite a bit more) in this
> blog post from my coworker: http://thelastpickle.com/blog/2016/
> 07/27/about-deletes-and-tombstones.html
>
> Repair doesn’t clean up tombstones, they’re only removed through
> compaction.  I advise taking care with nodetool compact, most of the time
> it’s not a great idea for a variety of reasons.  Check out the above post,
> if you still have questions, ask away.
>
>
> On Apr 1, 2018, at 9:41 PM, Xiangfei Ni <xiangfei...@cm-dt.com> wrote:
>
> Hi All,
>   I want to delete the expired tombstone, someone uses nodetool repair
> ,but someone uses compact,so I want to know which one is the correct way,
>   I have read the below pages from Datastax,but the page just tells us how
> to use the command,but doesn’t tell us what it is exactly dose,
>   https://docs.datastax.com/en/cassandra/3.0/cassandra/
> tools/toolsRepair.html
>could anybody tell me how to clean the tombstone and give me some
> materials include the detailed instruction about the nodetool command and
> options?Web link is also ok.
>   Thanks very much
> Best Regards,
>
> 倪项菲*/ **David Ni*
> 中移德电网络科技有限公司
>
> Virtue Intelligent Network Ltd, co.
> Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
> Mob: +86 13797007811 <+86%20137%209700%207811>|Tel: + 86 27 5024 2516
> <+86%2027%205024%202516>
>
>
>

Re: nodetool repair and compact

[Jon Haddad]

You’ll find the answers to your questions (and quite a bit more) in this blog 
post from my coworker: 
http://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html 
<http://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html>

Repair doesn’t clean up tombstones, they’re only removed through compaction.  I 
advise taking care with nodetool compact, most of the time it’s not a great 
idea for a variety of reasons.  Check out the above post, if you still have 
questions, ask away.  


> On Apr 1, 2018, at 9:41 PM, Xiangfei Ni <xiangfei...@cm-dt.com> wrote:
> 
> Hi All,
>   I want to delete the expired tombstone, someone uses nodetool repair ,but 
> someone uses compact,so I want to know which one is the correct way,
>   I have read the below pages from Datastax,but the page just tells us how to 
> use the command,but doesn’t tell us what it is exactly dose,
>   https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsRepair.html 
> <https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsRepair.html>
>could anybody tell me how to clean the tombstone and give me some 
> materials include the detailed instruction about the nodetool command and 
> options?Web link is also ok.
>   Thanks very much
> Best Regards,
>  
> 倪项菲/ David Ni
> 中移德电网络科技有限公司
> Virtue Intelligent Network Ltd, co.
> 
> Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
> Mob: +86 13797007811|Tel: + 86 27 5024 2516

nodetool repair and compact

[Xiangfei Ni]

Hi All,
  I want to delete the expired tombstone, someone uses nodetool repair ,but 
someone uses compact,so I want to know which one is the correct way,
  I have read the below pages from Datastax,but the page just tells us how to 
use the command,but doesn’t tell us what it is exactly dose,
   https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsRepair.html
   could anybody tell me how to clean the tombstone and give me some materials 
include the detailed instruction about the nodetool command and options?Web 
link is also ok.
  Thanks very much
Best Regards,

倪项菲/ David Ni
中移德电网络科技有限公司
Virtue Intelligent Network Ltd, co.
Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
Mob: +86 13797007811|Tel: + 86 27 5024 2516

Re: Nodetool Repair --full

[kurt greaves]

Worth noting that if you have racks == RF you only need to repair one rack
to repair all the data in the cluster if you *don't* use -pr. Also note
that full repairs on >=3.0 case anti-compactions and will mark things as
repaired, so once you start repairs you need to keep repairing to ensure
you don't have any zombie data or other problems.

On 17 March 2018 at 15:52, Hannu Kröger <hkro...@gmail.com> wrote:

> Hi Jonathan,
>
> If you want to repair just one node (for example if it has been down for
> more than 3h), run “nodetool repair -full” on that node. This will bring
> all data on that node up to date.
>
> If you want to repair all data on the cluster, run “nodetool repair -full
> -pr” on each node. This will run full repair on all nodes but it will do it
> so only the primary range for each node is fixed. If you do it on all
> nodes, effectively the whole token range is repaired. You can run the same
> without -pr to get the same effect but it’s not efficient because then you
> are doing the repair RF times on all data instead of just repairing the
> whole data once.
>
> I hope this clarifies,
> Hannu
>
> On 17 Mar 2018, at 17:20, Jonathan Baynes <jonathan.bay...@tradeweb.com>
> wrote:
>
> Hi Community,
>
> Can someone confirm, as the documentation out on the web is so
> contradictory and vague.
>
> Nodetool repair –full if I call this, do I need to run this on ALL my
> nodes or is just the once sufficient?
>
> Thanks
> J
>
> *Jonathan Baynes*
> DBA
> Tradeweb Europe Limited
> Moor Place  •  1 Fore Street Avenue
> <https://maps.google.com/?q=1+Fore+Street+Avenue+%E2%80%A2+London+EC2Y+9DT=gmail=g>
>   •
> <https://maps.google.com/?q=1+Fore+Street+Avenue+%E2%80%A2+London+EC2Y+9DT=gmail=g>
>   London EC2Y 9DT
> <https://maps.google.com/?q=1+Fore+Street+Avenue+%E2%80%A2+London+EC2Y+9DT=gmail=g>
> P +44 (0)20 77760988 <+44%2020%207776%200988>  •  F +44 (0)20 7776 3201
> <+44%2020%207776%203201>  •  M +44 (0)7884111546 <+44%207884%20111546>
> jonathan.bay...@tradeweb.com
>
>  <http://www.tradeweb.com/>   follow us:  **
> <https://www.linkedin.com/company/tradeweb?trk=top_nav_home>   <
> image003.jpg> <http://www.twitter.com/Tradeweb>
> —
> A leading marketplace <http://www.tradeweb.com/About-Us/Awards/> for
> electronic fixed income, derivatives and ETF trading
>
>
> 
>
> This e-mail may contain confidential and/or privileged information. If you
> are not the intended recipient (or have received this e-mail in error)
> please notify the sender immediately and destroy it. Any unauthorized
> copying, disclosure or distribution of the material in this e-mail is
> strictly forbidden. Tradeweb reserves the right to monitor all e-mail
> communications through its networks. If you do not wish to receive
> marketing emails about our products / services, please let us know by
> contacting us, either by email at contac...@tradeweb.com or by writing to
> us at the registered office of Tradeweb in the UK, which is: Tradeweb
> Europe Limited (company number 3912826), 1 Fore Street Avenue London EC2Y
> 9DT
> <https://maps.google.com/?q=1+Fore+Street+Avenue+London+EC2Y+9DT=gmail=g>.
> To see our privacy policy, visit our website @ www.tradeweb.com.
>
>
>

Re: Nodetool Repair --full

[Hannu Kröger]

Hi Jonathan,

If you want to repair just one node (for example if it has been down for more 
than 3h), run “nodetool repair -full” on that node. This will bring all data on 
that node up to date.

If you want to repair all data on the cluster, run “nodetool repair -full -pr” 
on each node. This will run full repair on all nodes but it will do it so only 
the primary range for each node is fixed. If you do it on all nodes, 
effectively the whole token range is repaired. You can run the same without -pr 
to get the same effect but it’s not efficient because then you are doing the 
repair RF times on all data instead of just repairing the whole data once.

I hope this clarifies,
Hannu

> On 17 Mar 2018, at 17:20, Jonathan Baynes <jonathan.bay...@tradeweb.com> 
> wrote:
> 
> Hi Community,
>  
> Can someone confirm, as the documentation out on the web is so contradictory 
> and vague.
>  
> Nodetool repair –full if I call this, do I need to run this on ALL my nodes 
> or is just the once sufficient?
>  
> Thanks
> J
>  
> Jonathan Baynes
> DBA
> Tradeweb Europe Limited
> Moor Place  •  1 Fore Street Avenue  •  London EC2Y 9DT
> P +44 (0)20 77760988  •  F +44 (0)20 7776 3201  •  M +44 (0)7884111546
> jonathan.bay...@tradeweb.com <mailto:jonathan.bay...@tradeweb.com>
>  
>  <http://www.tradeweb.com/>   follow us:   
> <https://www.linkedin.com/company/tradeweb?trk=top_nav_home>
> <http://www.twitter.com/Tradeweb>
> —
> A leading marketplace <http://www.tradeweb.com/About-Us/Awards/> for 
> electronic fixed income, derivatives and ETF trading
>  
> 
> 
> This e-mail may contain confidential and/or privileged information. If you 
> are not the intended recipient (or have received this e-mail in error) please 
> notify the sender immediately and destroy it. Any unauthorized copying, 
> disclosure or distribution of the material in this e-mail is strictly 
> forbidden. Tradeweb reserves the right to monitor all e-mail communications 
> through its networks. If you do not wish to receive marketing emails about 
> our products / services, please let us know by contacting us, either by email 
> at contac...@tradeweb.com <mailto:contac...@tradeweb.com> or by writing to us 
> at the registered office of Tradeweb in the UK, which is: Tradeweb Europe 
> Limited (company number 3912826), 1 Fore Street Avenue London EC2Y 9DT. To 
> see our privacy policy, visit our website @ www.tradeweb.com 
> <http://www.tradeweb.com/>.
> 

Nodetool Repair --full

[Jonathan Baynes]

Hi Community,

Can someone confirm, as the documentation out on the web is so contradictory 
and vague.

Nodetool repair -full if I call this, do I need to run this on ALL my nodes or 
is just the once sufficient?

Thanks
J

Jonathan Baynes
DBA
Tradeweb Europe Limited
Moor Place  *  1 Fore Street Avenue  *  London EC2Y 9DT
P +44 (0)20 77760988  *  F +44 (0)20 7776 3201  *  M +44 (0)7884111546
jonathan.bay...@tradeweb.com

[cid:image001.jpg@01CD26AD.4165F110]<http://www.tradeweb.com/>   follow us:  
[cid:image002.jpg@01CD26AD.4165F110] 
<https://www.linkedin.com/company/tradeweb?trk=top_nav_home>
[cid:image003.jpg@01CD26AD.4165F110] <http://www.twitter.com/Tradeweb>
-
A leading marketplace<http://www.tradeweb.com/About-Us/Awards/> for electronic 
fixed income, derivatives and ETF trading




This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves the right to monitor all e-mail communications 
through its networks. If you do not wish to receive marketing emails about our 
products / services, please let us know by contacting us, either by email at 
contac...@tradeweb.com or by writing to us at the registered office of Tradeweb 
in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore 
Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ 
www.tradeweb.com.

Re: Cassandra 2.1.18 - Concurrent nodetool repair resulting in > 30K SSTables for a single small (GBytes) CF

[kurt greaves]

>
>  What we did have was some sort of overlapping between our daily repair
> cronjob and the newly added node still in progress joining. Don’t know if
> this sort of combination might causing troubles.

I wouldn't be surprised if this caused problems. Probably want to avoid
that.

with waiting a few minutes after each finished execution and every time I
> see “… out of sync …” log messages in context of the repair, so it looks
> like, that each repair execution is detecting inconsistencies. Does this
> make sense?

Well it doesn't, but there have been issues in the past that caused exactly
this problem. I was under the impression they were all fixed by 2.1.18
though.

Additionally, we are writing at CL ANY, reading at ONE and repair chance
> for the 2 CFs in question is default 0.1

Have you considered writing at least at CL [LOCAL_]ONE? At the very least
it would rule out if there's a problem with hints.
​

RE: Cassandra 2.1.18 - Concurrent nodetool repair resulting in > 30K SSTables for a single small (GBytes) CF

[Steinmaurer, Thomas]

Hi Kurt,

our provisioning layer allows extending a cluster only one-by-one, thus we 
didn’t add multiple nodes at the same time.

What we did have was some sort of overlapping between our daily repair cronjob 
and the newly added node still in progress joining. Don’t know if this sort of 
combination might causing troubles.

I did some further testing and run on the same node the following repair call.

nodetool repair -pr ks cf1 cf2

with waiting a few minutes after each finished execution and every time I see 
“… out of sync …” log messages in context of the repair, so it looks like, that 
each repair execution is detecting inconsistencies. Does this make sense?

As said: We are using vnodes (256), RF=3. Additionally, we are writing at CL 
ANY, reading at ONE and repair chance for the 2 CFs in question is default 0.1

Currently testing a few consecutives executions without -pr on the same node.

Thanks,
Thomas

From: kurt greaves [mailto:k...@instaclustr.com]
Sent: Montag, 05. März 2018 01:10
To: User <user@cassandra.apache.org>
Subject: Re: Cassandra 2.1.18 - Concurrent nodetool repair resulting in > 30K 
SSTables for a single small (GBytes) CF

Repairs with vnodes is likely to cause a lot of small SSTables if you have 
inconsistencies (at least 1 per vnode). Did you have any issues when adding 
nodes, or did you add multiple nodes at a time? Anything that could have lead 
to a bit of inconsistency could have been the cause.

I'd probably avoid running the repairs across all the nodes simultaneously and 
instead spread them out over a week. That likely made it worse. Also worth 
noting that in versions 3.0+ you won't be able to run nodetool repair in such a 
way because anti-compaction will be triggered which will fail if multiple 
anti-compactions are attempted simultaneously (if you run multiple repairs 
simultaneously).

Have a look at orchestrating your repairs with TLP's fork of 
cassandra-reaper<http://cassandra-reaper.io/>.
​
The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 313

Re: Cassandra 2.1.18 - Concurrent nodetool repair resulting in > 30K SSTables for a single small (GBytes) CF

[kurt greaves]

Repairs with vnodes is likely to cause a lot of small SSTables if you have
inconsistencies (at least 1 per vnode). Did you have any issues when adding
nodes, or did you add multiple nodes at a time? Anything that could have
lead to a bit of inconsistency could have been the cause.

I'd probably avoid running the repairs across all the nodes simultaneously
and instead spread them out over a week. That likely made it worse. Also
worth noting that in versions 3.0+ you won't be able to run nodetool repair
in such a way because anti-compaction will be triggered which will fail if
multiple anti-compactions are attempted simultaneously (if you run multiple
repairs simultaneously).

Have a look at orchestrating your repairs with TLP's fork of
cassandra-reaper <http://cassandra-reaper.io/>.
​

Cassandra 2.1.18 - Concurrent nodetool repair resulting in > 30K SSTables for a single small (GBytes) CF

[Steinmaurer, Thomas]

Hello,

Production, 9 node cluster with Cassandra 2.1.18, vnodes, default 256 tokens, 
RF=3, compaction throttling = 16, concurrent compactors = 4, running in AWS 
using m4.xlarge at ~ 35% CPU AVG

We have a nightly cronjob starting a "nodetool repair -pr ks cf1 cf2" 
concurrently on all nodes, where data volume for cf1 and cf2 is ~ 1-5GB in 
size, so pretty small.

After extending the cluster from 6 to the current 9 nodes and "nodetool 
cleanup" being finished, the above repair is resulting in > 30K SSTables for 
these two CFs on several nodes with very, very tiny files < 1Kbytes , but not 
on all nodes. Obviously, this affects read latency + disk IO + CPU a lot and it 
needs several hours until the situation relaxes. We have other clusters with 
the same spec which also have been extended from 6 to 9 nodes in the past, 
where we don't see this issue. For now, we have disabled the nightly cron job.

Any input on how to trouble-shoot this issue about the root cause?

Thanks,
Thomas

The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freist?dterstra?e 313

Re: Cluster Repairs 'nodetool repair -pr' Cause Severe Increase in Read Latency After Shrinking Cluster

[Fred Habash]

 .

On Feb 21, 2018 1:29 PM, "Fred Habash" <fmhab...@gmail.com> wrote:

> One node at a time
>
> On Feb 21, 2018 10:23 AM, "Carl Mueller" <carl.muel...@smartthings.com>
> wrote:
>
>> What is your replication factor?
>> Single datacenter, three availability zones, is that right?
>> You removed one node at a time or three at once?
>>
>> On Wed, Feb 21, 2018 at 10:20 AM, Fd Habash <fmhab...@gmail.com> wrote:
>>
>>> We have had a 15 node cluster across three zones and cluster repairs
>>> using ‘nodetool repair -pr’ took about 3 hours to finish. Lately, we shrunk
>>> the cluster to 12. Since then, same repair job has taken up to 12 hours to
>>> finish and most times, it never does.
>>>
>>>
>>>
>>> More importantly, at some point during the repair cycle, we see read
>>> latencies jumping to 1-2 seconds and applications immediately notice the
>>> impact.
>>>
>>>
>>>
>>> stream_throughput_outbound_megabits_per_sec is set at 200 and
>>> compaction_throughput_mb_per_sec at 64. The /data dir on the nodes is
>>> around ~500GB at 44% usage.
>>>
>>>
>>>
>>> When shrinking the cluster, the ‘nodetool decommision’ was eventless. It
>>> completed successfully with no issues.
>>>
>>>
>>>
>>> What could possibly cause repairs to cause this impact following cluster
>>> downsizing? Taking three nodes out does not seem compatible with such a
>>> drastic effect on repair and read latency.
>>>
>>>
>>>
>>> Any expert insights will be appreciated.
>>>
>>> 
>>> Thank you
>>>
>>>
>>>
>>
>>

Re: Cluster Repairs 'nodetool repair -pr' Cause Severe IncreaseinRead Latency After Shrinking Cluster

[Carl Mueller]

Your partition sizes aren't ridiculous... kinda big cells if there are 4
cells and 12 MB partitions, but still I don't think that is ludicrous.

Whelp, I'm out of ideas from my "pay grade". Honestly, with AZ/racks you
should have theoretically might have been able to take the nodes off
simultaneously, but (Disclaimer) I've never done that.

?Rolling Restart? <-- definitely indicates I have no ideas :-)

On Thu, Feb 22, 2018 at 8:15 AM, Fd Habash <fmhab...@gmail.com> wrote:

> One more observation …
>
>
>
> When we compare read latencies between non-prod (where nodes were removed)
> to prod clusters, even though the node load as measure by size of /data dir
> is similar, yet the read latencies are 5 times slower in the downsized
> non-prod cluster.
>
>
>
> The only difference we see is that prod reads from 4 sstables whereas
> non-prod reads from 5 as cfhistograms.
>
>
>
> Non-prod /data size
>
> -
>
> Filesystem  Size  Used Avail Use% Mounted on
>
> /dev/nvme0n1885G  454G  432G  52% /data
>
> Filesystem  Size  Used Avail Use% Mounted on
>
> /dev/nvme0n1885G  439G  446G  50% /data
>
> Filesystem  Size  Used Avail Use% Mounted on
>
> /dev/nvme0n1885G  368G  518G  42% /data
>
> Filesystem  Size  Used Avail Use% Mounted on
>
> /dev/nvme0n1885G  431G  455G  49% /data
>
> Filesystem  Size  Used Avail Use% Mounted on
>
> /dev/nvme0n1885G  463G  423G  53% /data
>
> Filesystem  Size  Used Avail Use% Mounted on
>
> /dev/nvme0n1885G  406G  479G  46% /data
>
> Filesystem  Size  Used Avail Use% Mounted on
>
> /dev/nvme0n1885G  419G  466G  48% /data
>
> Filesystem  Size  Used Avail Use% Mounted on
>
>
>
> Prod /data size
>
> 
>
> Filesystem  Size  Used Avail Use% Mounted on
>
> /dev/nvme0n1885G  352G  534G  40% /data
>
> Filesystem  Size  Used Avail Use% Mounted on
>
> /dev/nvme0n1885G  423G  462G  48% /data
>
> Filesystem  Size  Used Avail Use% Mounted on
>
> /dev/nvme0n1885G  431G  454G  49% /data
>
> Filesystem  Size  Used Avail Use% Mounted on
>
> /dev/nvme0n1885G  442G  443G  50% /data
>
> Filesystem  Size  Used Avail Use% Mounted on
>
> /dev/nvme0n1885G  454G  431G  52% /data
>
>
>
>
>
> Cfhistograms: comparing prod to non-prod
>
> -
>
>
>
> Non-prod
>
> --
>
> 08:21:38Percentile  SSTables Write Latency  Read
> LatencyPartition SizeCell Count
>
> 08:21:38  (micros)
> (micros)   (bytes)
>
> 08:21:3850% 1.00 24.60
> 4055.27 11864 4
>
> 08:21:3875% 2.00 35.43
> 14530.76 17084 4
>
> 08:21:3895% 4.00126.93
> 89970.66 35425 4
>
> 08:21:3898% 5.00219.34
> 155469.30 73457 4
>
> 08:21:3899% 5.00219.34
>186563.16105778 4
>
> 08:21:38Min 0.00  5.72
> 17.0987 3
>
> 08:21:38Max 7.00  20924.30
> 1386179.89  14530764 4
>
>
>
> Prod
>
> ---
>
> 07:41:42Percentile  SSTables Write Latency  Read
> LatencyPartition SizeCell Count
>
> 07:41:42  (micros)
> (micros)   (bytes)
>
> 07:41:4250% 1.00 24.60
> 2346.80 11864 4
>
> 07:41:4275% 2.00 29.52
> 4866.32 17084 4
>
> 07:41:4295% 3.00 73.46
> 14530.76 29521 4
>
> 07:41:4298% 4.00182.79
> 25109.16 61214 4
>
> 07:41:4299% 4.00182.79
> 36157.19     88148 4
>
> 07:41:42Min 0.00  9.89
> 20.5087 0
>
> 07:41:42Max 5.00219.34
> 155469.30  12108970 4
>
>
>
>
>
> 
> Thank you
>
>
>
> *From: *Fd Habash <fmhab...@gmail.com>
> *Sent: *Thursday, February 22, 2018 9:00 AM
> *To: *user@cassandra.apache.org
>

RE: Cluster Repairs 'nodetool repair -pr' Cause Severe IncreaseinRead Latency After Shrinking Cluster

[Fd Habash]

One more observation …

When we compare read latencies between non-prod (where nodes were removed) to 
prod clusters, even though the node load as measure by size of /data dir is 
similar, yet the read latencies are 5 times slower in the downsized non-prod 
cluster.

The only difference we see is that prod reads from 4 sstables whereas non-prod 
reads from 5 as cfhistograms. 

Non-prod /data size
-
Filesystem  Size  Used Avail Use% Mounted on
/dev/nvme0n1885G  454G  432G  52% /data
Filesystem  Size  Used Avail Use% Mounted on
/dev/nvme0n1885G  439G  446G  50% /data
Filesystem  Size  Used Avail Use% Mounted on
/dev/nvme0n1885G  368G  518G  42% /data
Filesystem  Size  Used Avail Use% Mounted on
/dev/nvme0n1885G  431G  455G  49% /data
Filesystem  Size  Used Avail Use% Mounted on
/dev/nvme0n1885G  463G  423G  53% /data
Filesystem  Size  Used Avail Use% Mounted on
/dev/nvme0n1885G  406G  479G  46% /data
Filesystem  Size  Used Avail Use% Mounted on
/dev/nvme0n1885G  419G  466G  48% /data
Filesystem  Size  Used Avail Use% Mounted on

Prod /data size

Filesystem  Size  Used Avail Use% Mounted on
/dev/nvme0n1885G  352G  534G  40% /data
Filesystem  Size  Used Avail Use% Mounted on
/dev/nvme0n1885G  423G  462G  48% /data
Filesystem  Size  Used Avail Use% Mounted on
/dev/nvme0n1885G  431G  454G  49% /data
Filesystem  Size  Used Avail Use% Mounted on
/dev/nvme0n1885G  442G  443G  50% /data
Filesystem  Size  Used Avail Use% Mounted on
/dev/nvme0n1885G  454G  431G  52% /data


Cfhistograms: comparing prod to non-prod
-

Non-prod
--
08:21:38Percentile  SSTables Write Latency  Read 
LatencyPartition SizeCell Count
08:21:38  (micros)  
(micros)   (bytes)  
08:21:3850% 1.00 24.60   
4055.27 11864 4
08:21:3875% 2.00 35.43  
14530.76 17084 4
08:21:3895% 4.00126.93  
89970.66 35425 4
08:21:3898% 5.00219.34 
155469.30 73457 4
08:21:3899% 5.00219.34 
186563.16105778 4
08:21:38Min 0.00  5.72 
17.0987 3
08:21:38Max 7.00  20924.30
1386179.89  14530764 4

Prod
--- 
07:41:42Percentile  SSTables Write Latency  Read 
LatencyPartition SizeCell Count
07:41:42  (micros)  
(micros)   (bytes)  
07:41:4250% 1.00 24.60   
2346.80 11864 4
07:41:4275% 2.00 29.52   
4866.32 17084 4
07:41:4295% 3.00 73.46  
14530.76 29521 4
07:41:4298% 4.00182.79  
25109.16 61214 4
07:41:4299% 4.00182.79  
36157.19 88148 4
07:41:42Min 0.00  9.89 
20.5087 0
07:41:42Max 5.00219.34 
155469.30  12108970 4



Thank you

From: Fd Habash
Sent: Thursday, February 22, 2018 9:00 AM
To: user@cassandra.apache.org
Subject: RE: Cluster Repairs 'nodetool repair -pr' Cause Severe IncreaseinRead 
Latency After Shrinking Cluster


“ data was allowed to fully rebalance/repair/drain before the next node was 
taken off?”
--
Judging by the messages, the decomm was healthy. As an example 

  StorageService.java:3425 - Announcing that I have left the ring for 3ms   
…
INFO  [RMI TCP Connection(4)-127.0.0.1] 2016-01-07 06:00:52,662 
StorageService.java:1191 – DECOMMISSIONED

I do not believe repairs were run after each node removal. I’ll double-check. 

I’m not sure what you mean by ‘rebalance’? How do you check if a node is 
balanced? Load/size of data dir? 

As for the drain, there was no need to drain and I believe it is not something 
you do as part of decomm’ing a node. 

did you take 1 off per rack/AZ?
--
We removed 3 nodes, one from each AZ in sequence

These are some

RE: Cluster Repairs 'nodetool repair -pr' Cause Severe Increase inRead Latency After Shrinking Cluster

[Fd Habash]

“ data was allowed to fully rebalance/repair/drain before the next node was 
taken off?”
--
Judging by the messages, the decomm was healthy. As an example 

  StorageService.java:3425 - Announcing that I have left the ring for 3ms   
 
…
INFO  [RMI TCP Connection(4)-127.0.0.1] 2016-01-07 06:00:52,662 
StorageService.java:1191 – DECOMMISSIONED

I do not believe repairs were run after each node removal. I’ll double-check. 

I’m not sure what you mean by ‘rebalance’? How do you check if a node is 
balanced? Load/size of data dir? 

As for the drain, there was no need to drain and I believe it is not something 
you do as part of decomm’ing a node. 

did you take 1 off per rack/AZ?
--
We removed 3 nodes, one from each AZ in sequence

These are some of the cfhistogram metrics. Read latencies are high after the 
removal of the nodes
--
You can see reads of 186ms are at the 99th% from 5 sstables. There are awfully 
high numbers given that these metrics measure C* storage layer read 
performance. 

Does this mean removing the nodes undersized the cluster? 

key_space_01/cf_01 histograms
Percentile  SSTables Write Latency  Read LatencyPartition Size  
  Cell Count
  (micros)  (micros)   (bytes)  

50% 1.00 24.60   4055.27 11864  
   4
75% 2.00 35.43  14530.76 17084  
   4
95% 4.00126.93  89970.66 35425  
   4
98% 5.00219.34 155469.30 73457  
   4
99% 5.00219.34 186563.16105778  
   4
Min 0.00  5.72 17.0987  
   3
Max 7.00  20924.301386179.89  14530764  
   4

key_space_01/cf_01 histograms
Percentile  SSTables Write Latency  Read LatencyPartition Size  
  Cell Count
  (micros)  (micros)   (bytes)  

50% 1.00 29.52   4055.27 11864  
   4
75% 2.00 42.51  10090.81 17084  
   4
95% 4.00152.32  52066.35 35425  
   4
98% 4.00219.34  89970.66 73457  
   4
99% 5.00219.34 155469.30 88148  
   4
Min 0.00  9.89 24.6087  
   0
Max 6.00   1955.67 557074.61  14530764  
   4


Thank you

From: Carl Mueller
Sent: Wednesday, February 21, 2018 4:33 PM
To: user@cassandra.apache.org
Subject: Re: Cluster Repairs 'nodetool repair -pr' Cause Severe Increase inRead 
Latency After Shrinking Cluster

Hm nodetool decommision performs the streamout of the replicated data, and you 
said that was apparently without error...

But if you dropped three nodes in one AZ/rack on a five node with RF3, then we 
have a missing RF factor unless NetworkTopologyStrategy fails over to another 
AZ. But that would also entail cross-az streaming and queries and repair.

On Wed, Feb 21, 2018 at 3:30 PM, Carl Mueller <carl.muel...@smartthings.com> 
wrote:
sorry for the idiot questions... 

data was allowed to fully rebalance/repair/drain before the next node was taken 
off?

did you take 1 off per rack/AZ?


On Wed, Feb 21, 2018 at 12:29 PM, Fred Habash <fmhab...@gmail.com> wrote:
One node at a time 

On Feb 21, 2018 10:23 AM, "Carl Mueller" <carl.muel...@smartthings.com> wrote:
What is your replication factor? 
Single datacenter, three availability zones, is that right?
You removed one node at a time or three at once?

On Wed, Feb 21, 2018 at 10:20 AM, Fd Habash <fmhab...@gmail.com> wrote:
We have had a 15 node cluster across three zones and cluster repairs using 
‘nodetool repair -pr’ took about 3 hours to finish. Lately, we shrunk the 
cluster to 12. Since then, same repair job has taken up to 12 hours to finish 
and most times, it never does. 
 
More importantly, at some point during the repair cycle, we see read latencies 
jumping to 1-2 seconds and applications immediately notice the impact.
 
stream_throughput_outbound_megabits_per_sec is set at 200 and 
compaction_throughput_mb_per_sec at 64. The /data dir on the nodes is around 
~500GB at 44% usage. 
 
When shrinking the cluster, the ‘nodetool decommision’ was eventless. It 
completed successfully with no issues.
 
What could possibly cause repairs to c

Re: Cluster Repairs 'nodetool repair -pr' Cause Severe Increase in Read Latency After Shrinking Cluster

[Carl Mueller]

Hm nodetool decommision performs the streamout of the replicated data, and
you said that was apparently without error...

But if you dropped three nodes in one AZ/rack on a five node with RF3, then
we have a missing RF factor unless NetworkTopologyStrategy fails over to
another AZ. But that would also entail cross-az streaming and queries and
repair.

On Wed, Feb 21, 2018 at 3:30 PM, Carl Mueller <carl.muel...@smartthings.com>
wrote:

> sorry for the idiot questions...
>
> data was allowed to fully rebalance/repair/drain before the next node was
> taken off?
>
> did you take 1 off per rack/AZ?
>
>
> On Wed, Feb 21, 2018 at 12:29 PM, Fred Habash <fmhab...@gmail.com> wrote:
>
>> One node at a time
>>
>> On Feb 21, 2018 10:23 AM, "Carl Mueller" <carl.muel...@smartthings.com>
>> wrote:
>>
>>> What is your replication factor?
>>> Single datacenter, three availability zones, is that right?
>>> You removed one node at a time or three at once?
>>>
>>> On Wed, Feb 21, 2018 at 10:20 AM, Fd Habash <fmhab...@gmail.com> wrote:
>>>
>>>> We have had a 15 node cluster across three zones and cluster repairs
>>>> using ‘nodetool repair -pr’ took about 3 hours to finish. Lately, we shrunk
>>>> the cluster to 12. Since then, same repair job has taken up to 12 hours to
>>>> finish and most times, it never does.
>>>>
>>>>
>>>>
>>>> More importantly, at some point during the repair cycle, we see read
>>>> latencies jumping to 1-2 seconds and applications immediately notice the
>>>> impact.
>>>>
>>>>
>>>>
>>>> stream_throughput_outbound_megabits_per_sec is set at 200 and
>>>> compaction_throughput_mb_per_sec at 64. The /data dir on the nodes is
>>>> around ~500GB at 44% usage.
>>>>
>>>>
>>>>
>>>> When shrinking the cluster, the ‘nodetool decommision’ was eventless.
>>>> It completed successfully with no issues.
>>>>
>>>>
>>>>
>>>> What could possibly cause repairs to cause this impact following
>>>> cluster downsizing? Taking three nodes out does not seem compatible with
>>>> such a drastic effect on repair and read latency.
>>>>
>>>>
>>>>
>>>> Any expert insights will be appreciated.
>>>>
>>>> 
>>>> Thank you
>>>>
>>>>
>>>>
>>>
>>>
>

Re: Cluster Repairs 'nodetool repair -pr' Cause Severe Increase in Read Latency After Shrinking Cluster

[Carl Mueller]

sorry for the idiot questions...

data was allowed to fully rebalance/repair/drain before the next node was
taken off?

did you take 1 off per rack/AZ?


On Wed, Feb 21, 2018 at 12:29 PM, Fred Habash <fmhab...@gmail.com> wrote:

> One node at a time
>
> On Feb 21, 2018 10:23 AM, "Carl Mueller" <carl.muel...@smartthings.com>
> wrote:
>
>> What is your replication factor?
>> Single datacenter, three availability zones, is that right?
>> You removed one node at a time or three at once?
>>
>> On Wed, Feb 21, 2018 at 10:20 AM, Fd Habash <fmhab...@gmail.com> wrote:
>>
>>> We have had a 15 node cluster across three zones and cluster repairs
>>> using ‘nodetool repair -pr’ took about 3 hours to finish. Lately, we shrunk
>>> the cluster to 12. Since then, same repair job has taken up to 12 hours to
>>> finish and most times, it never does.
>>>
>>>
>>>
>>> More importantly, at some point during the repair cycle, we see read
>>> latencies jumping to 1-2 seconds and applications immediately notice the
>>> impact.
>>>
>>>
>>>
>>> stream_throughput_outbound_megabits_per_sec is set at 200 and
>>> compaction_throughput_mb_per_sec at 64. The /data dir on the nodes is
>>> around ~500GB at 44% usage.
>>>
>>>
>>>
>>> When shrinking the cluster, the ‘nodetool decommision’ was eventless. It
>>> completed successfully with no issues.
>>>
>>>
>>>
>>> What could possibly cause repairs to cause this impact following cluster
>>> downsizing? Taking three nodes out does not seem compatible with such a
>>> drastic effect on repair and read latency.
>>>
>>>
>>>
>>> Any expert insights will be appreciated.
>>>
>>> 
>>> Thank you
>>>
>>>
>>>
>>
>>

Re: Cluster Repairs 'nodetool repair -pr' Cause Severe Increase in Read Latency After Shrinking Cluster

[Fred Habash]

RF of 3 with three racs AZ's in a single region.

On Feb 21, 2018 10:23 AM, "Carl Mueller" <carl.muel...@smartthings.com>
wrote:

> What is your replication factor?
> Single datacenter, three availability zones, is that right?
> You removed one node at a time or three at once?
>
> On Wed, Feb 21, 2018 at 10:20 AM, Fd Habash <fmhab...@gmail.com> wrote:
>
>> We have had a 15 node cluster across three zones and cluster repairs
>> using ‘nodetool repair -pr’ took about 3 hours to finish. Lately, we shrunk
>> the cluster to 12. Since then, same repair job has taken up to 12 hours to
>> finish and most times, it never does.
>>
>>
>>
>> More importantly, at some point during the repair cycle, we see read
>> latencies jumping to 1-2 seconds and applications immediately notice the
>> impact.
>>
>>
>>
>> stream_throughput_outbound_megabits_per_sec is set at 200 and
>> compaction_throughput_mb_per_sec at 64. The /data dir on the nodes is
>> around ~500GB at 44% usage.
>>
>>
>>
>> When shrinking the cluster, the ‘nodetool decommision’ was eventless. It
>> completed successfully with no issues.
>>
>>
>>
>> What could possibly cause repairs to cause this impact following cluster
>> downsizing? Taking three nodes out does not seem compatible with such a
>> drastic effect on repair and read latency.
>>
>>
>>
>> Any expert insights will be appreciated.
>>
>> 
>> Thank you
>>
>>
>>
>
>

Re: Cluster Repairs 'nodetool repair -pr' Cause Severe Increase in Read Latency After Shrinking Cluster

[Fred Habash]

One node at a time

On Feb 21, 2018 10:23 AM, "Carl Mueller" <carl.muel...@smartthings.com>
wrote:

> What is your replication factor?
> Single datacenter, three availability zones, is that right?
> You removed one node at a time or three at once?
>
> On Wed, Feb 21, 2018 at 10:20 AM, Fd Habash <fmhab...@gmail.com> wrote:
>
>> We have had a 15 node cluster across three zones and cluster repairs
>> using ‘nodetool repair -pr’ took about 3 hours to finish. Lately, we shrunk
>> the cluster to 12. Since then, same repair job has taken up to 12 hours to
>> finish and most times, it never does.
>>
>>
>>
>> More importantly, at some point during the repair cycle, we see read
>> latencies jumping to 1-2 seconds and applications immediately notice the
>> impact.
>>
>>
>>
>> stream_throughput_outbound_megabits_per_sec is set at 200 and
>> compaction_throughput_mb_per_sec at 64. The /data dir on the nodes is
>> around ~500GB at 44% usage.
>>
>>
>>
>> When shrinking the cluster, the ‘nodetool decommision’ was eventless. It
>> completed successfully with no issues.
>>
>>
>>
>> What could possibly cause repairs to cause this impact following cluster
>> downsizing? Taking three nodes out does not seem compatible with such a
>> drastic effect on repair and read latency.
>>
>>
>>
>> Any expert insights will be appreciated.
>>
>> 
>> Thank you
>>
>>
>>
>
>

Re: Cluster Repairs 'nodetool repair -pr' Cause Severe Increase in Read Latency After Shrinking Cluster

[Jeff Jirsa]

nodetool cfhistograms, nodetool compactionstats would be helpful

Compaction is probably behind from streaming, and reads are touching many 
sstables.

-- 
Jeff Jirsa


> On Feb 21, 2018, at 8:20 AM, Fd Habash <fmhab...@gmail.com> wrote:
> 
> We have had a 15 node cluster across three zones and cluster repairs using 
> ‘nodetool repair -pr’ took about 3 hours to finish. Lately, we shrunk the 
> cluster to 12. Since then, same repair job has taken up to 12 hours to finish 
> and most times, it never does.
>  
> More importantly, at some point during the repair cycle, we see read 
> latencies jumping to 1-2 seconds and applications immediately notice the 
> impact.
>  
> stream_throughput_outbound_megabits_per_sec is set at 200 and 
> compaction_throughput_mb_per_sec at 64. The /data dir on the nodes is around 
> ~500GB at 44% usage.
>  
> When shrinking the cluster, the ‘nodetool decommision’ was eventless. It 
> completed successfully with no issues.
>  
> What could possibly cause repairs to cause this impact following cluster 
> downsizing? Taking three nodes out does not seem compatible with such a 
> drastic effect on repair and read latency.
>  
> Any expert insights will be appreciated.
> 
> Thank you
>  

Re: Cluster Repairs 'nodetool repair -pr' Cause Severe Increase in Read Latency After Shrinking Cluster

[Carl Mueller]

What is your replication factor?
Single datacenter, three availability zones, is that right?
You removed one node at a time or three at once?

On Wed, Feb 21, 2018 at 10:20 AM, Fd Habash <fmhab...@gmail.com> wrote:

> We have had a 15 node cluster across three zones and cluster repairs using
> ‘nodetool repair -pr’ took about 3 hours to finish. Lately, we shrunk the
> cluster to 12. Since then, same repair job has taken up to 12 hours to
> finish and most times, it never does.
>
>
>
> More importantly, at some point during the repair cycle, we see read
> latencies jumping to 1-2 seconds and applications immediately notice the
> impact.
>
>
>
> stream_throughput_outbound_megabits_per_sec is set at 200 and
> compaction_throughput_mb_per_sec at 64. The /data dir on the nodes is
> around ~500GB at 44% usage.
>
>
>
> When shrinking the cluster, the ‘nodetool decommision’ was eventless. It
> completed successfully with no issues.
>
>
>
> What could possibly cause repairs to cause this impact following cluster
> downsizing? Taking three nodes out does not seem compatible with such a
> drastic effect on repair and read latency.
>
>
>
> Any expert insights will be appreciated.
>
> 
> Thank you
>
>
>

Cluster Repairs 'nodetool repair -pr' Cause Severe Increase in Read Latency After Shrinking Cluster

[Fd Habash]

We have had a 15 node cluster across three zones and cluster repairs using 
‘nodetool repair -pr’ took about 3 hours to finish. Lately, we shrunk the 
cluster to 12. Since then, same repair job has taken up to 12 hours to finish 
and most times, it never does. 

More importantly, at some point during the repair cycle, we see read latencies 
jumping to 1-2 seconds and applications immediately notice the impact.

stream_throughput_outbound_megabits_per_sec is set at 200 and 
compaction_throughput_mb_per_sec at 64. The /data dir on the nodes is around 
~500GB at 44% usage. 

When shrinking the cluster, the ‘nodetool decommision’ was eventless. It 
completed successfully with no issues.

What could possibly cause repairs to cause this impact following cluster 
downsizing? Taking three nodes out does not seem compatible with such a drastic 
effect on repair and read latency. 

Any expert insights will be appreciated. 

Thank you

Re: Nodetool repair on read only cluster

[Jeff Jirsa]

Over time the various nodes likely got slightly out of sync - dropped mutations 
primarily, during Long GC pauses or maybe network failures

In that case, repair will make all of the data match - how long it takes 
depends on size of data (more data takes longer to validate), size of your 
partitions (big partitions are more work to repair), and how you invoke repair



-- 
Jeff Jirsa


> On Nov 29, 2017, at 5:42 PM, Roger Warner  wrote:
> 
>  
> What would running a repair on a cluster do when there are no deletes nor 
> have there ever been?I have no deletes yet on my data.Yet running a 
> repair took over 9 hours on a 5 node cluster?
>  
> Roger?

Re: Nodetool repair on read only cluster

[@Nandan@]

Hi Roger,
As you provide incomplete information which is so tough to analyse .
But if you like to refer then please check below JIRA link to check out is
it useful or not. ?
https://issues.apache.org/jira/browse/CASSANDRA-6616

Thanks.

On Thu, Nov 30, 2017 at 9:42 AM, Roger Warner  wrote:

>
>
> What would running a repair on a cluster do when there are no deletes nor
> have there ever been?I have no deletes yet on my data.Yet running a
> repair took over 9 hours on a 5 node cluster?
>
>
>
> Roger?
>

Nodetool repair on read only cluster

[Roger Warner]

What would running a repair on a cluster do when there are no deletes nor have 
there ever been?I have no deletes yet on my data.Yet running a repair 
took over 9 hours on a 5 node cluster?

Roger?

RE: Not serving read requests while running nodetool repair

[Steinmaurer, Thomas]

Hi Jeff,

thanks for the info. Hoped that nodetool provides an option for that. We will 
go with the temporary QUORUM approach for now.

Thanks again.

Thomas

From: Jeff Jirsa [mailto:jji...@gmail.com]
Sent: Mittwoch, 18. Oktober 2017 15:46
To: user@cassandra.apache.org
Subject: Re: Not serving read requests while running nodetool repair

You can accomplish this by manually tweaking the values in the dynamic snitch 
mbean so other nodes won’t select it for reads
--
Jeff Jirsa


On Oct 18, 2017, at 3:24 AM, Steinmaurer, Thomas 
<thomas.steinmau...@dynatrace.com<mailto:thomas.steinmau...@dynatrace.com>> 
wrote:
Hi Nicolas,

newly bootstrapping is not really an option, I think.

I hoped there might be a mode (perhaps there is, but haven’t found it yet) 
where a node might not actively participate in read requests but still writing 
newly arriving data during a repair or whatever maintenance task one thinks 
that some sort of write-only is appropriate.

Thanks,
Thomas

From: Nicolas Guyomar [mailto:nicolas.guyo...@gmail.com]
Sent: Mittwoch, 18. Oktober 2017 09:58
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Re: Not serving read requests while running nodetool repair

Hi Thomas,

AFAIK  temporarily reading at LOCAL_QUORUM/QUORUM until nodetool repair is 
finished is the way to go. You can still disable binary/thrift on the node to 
"protect" it from acting as a coordinator, and complete its repair quietly, but 
I'm not sure that would make such a huge difference in recovery time.

If you disable gossip the node will be see as down, thus disabling repair if 
I'm correct

If repair is taking too much time, or switching to Read Quorum not feasible 
within your application, is it OK for you to shutdown the node, wipe its data 
and launch a repair on it at an appropriate time windows ?



On 18 October 2017 at 08:04, Steinmaurer, Thomas 
<thomas.steinmau...@dynatrace.com<mailto:thomas.steinmau...@dynatrace.com>> 
wrote:
Hello,

due to performance/latency reasons, we are currently reading and writing time 
series data at consistency level  ONE/ANY.

In case of a node being down and recovering after the default hinted handoff 
window of 3 hrs, we may potentially read stale data from the recovering node. 
Of course, from an operational POV, we will trigger a nodetool repair after the 
recovered node has start up, but to my understanding, this still may cause 
reading stale data from this particular node until nodetool repair is finished, 
which may take several hours. Is this correct?

Is there a way (e.g. in newer releases 3.0/3.11) to tell a node not being part 
of read requests? Something like a counterpart of nodetool drain for writes?

Other options:


-  Disabling binary/thrift: Although the node won’t act as coordinator 
then for client requests, as it won’t be available from a client perspective, 
inter-node, the recovering node is being contacted by other nodes for serving 
requests, right?

-  Disabling gossip: Basically the node is marked as DOWN, so treated 
like it is not available, thus not an option?

-  Temporarily reading at LOCAL_QUORUM/QUORUM until nodetool repair is 
finished?

Did I miss something?

Thanks,
Thomas

The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freist<https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313=gmail=g>ädterstra<https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313=gmail=g>ße
 
313<https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313=gmail=g>

The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 313
The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 313

Re: Not serving read requests while running nodetool repair

[Jeff Jirsa]

You can accomplish this by manually tweaking the values in the dynamic snitch 
mbean so other nodes won’t select it for reads

-- 
Jeff Jirsa


> On Oct 18, 2017, at 3:24 AM, Steinmaurer, Thomas 
> <thomas.steinmau...@dynatrace.com> wrote:
> 
> Hi Nicolas,
>  
> newly bootstrapping is not really an option, I think.
>  
> I hoped there might be a mode (perhaps there is, but haven’t found it yet) 
> where a node might not actively participate in read requests but still 
> writing newly arriving data during a repair or whatever maintenance task one 
> thinks that some sort of write-only is appropriate.
>  
> Thanks,
> Thomas
>  
> From: Nicolas Guyomar [mailto:nicolas.guyo...@gmail.com] 
> Sent: Mittwoch, 18. Oktober 2017 09:58
> To: user@cassandra.apache.org
> Subject: Re: Not serving read requests while running nodetool repair
>  
> Hi Thomas,
>  
> AFAIK  temporarily reading at LOCAL_QUORUM/QUORUM until nodetool repair is 
> finished is the way to go. You can still disable binary/thrift on the node to 
> "protect" it from acting as a coordinator, and complete its repair quietly, 
> but I'm not sure that would make such a huge difference in recovery time.
>  
> If you disable gossip the node will be see as down, thus disabling repair if 
> I'm correct
>  
> If repair is taking too much time, or switching to Read Quorum not feasible 
> within your application, is it OK for you to shutdown the node, wipe its data 
> and launch a repair on it at an appropriate time windows ? 
>  
>  
>  
> On 18 October 2017 at 08:04, Steinmaurer, Thomas 
> <thomas.steinmau...@dynatrace.com> wrote:
> Hello,
>  
> due to performance/latency reasons, we are currently reading and writing time 
> series data at consistency level  ONE/ANY.
>  
> In case of a node being down and recovering after the default hinted handoff 
> window of 3 hrs, we may potentially read stale data from the recovering node. 
> Of course, from an operational POV, we will trigger a nodetool repair after 
> the recovered node has start up, but to my understanding, this still may 
> cause reading stale data from this particular node until nodetool repair is 
> finished, which may take several hours. Is this correct?
>  
> Is there a way (e.g. in newer releases 3.0/3.11) to tell a node not being 
> part of read requests? Something like a counterpart of nodetool drain for 
> writes?
>  
> Other options:
>  
> -  Disabling binary/thrift: Although the node won’t act as 
> coordinator then for client requests, as it won’t be available from a client 
> perspective, inter-node, the recovering node is being contacted by other 
> nodes for serving requests, right?
> 
> -  Disabling gossip: Basically the node is marked as DOWN, so treated 
> like it is not available, thus not an option?
> 
> -  Temporarily reading at LOCAL_QUORUM/QUORUM until nodetool repair 
> is finished?
> 
>  
> Did I miss something?
>  
> Thanks,
> Thomas
>  
> The contents of this e-mail are intended for the named addressee only. It 
> contains information that may be confidential. Unless you are the named 
> addressee or an authorized designee, you may not copy or use it, or disclose 
> it to anyone else. If you received it in error please notify us immediately 
> and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) 
> is a company registered in Linz whose registered office is at 4040 Linz, 
> Austria, Freistädterstraße 313
>  
> The contents of this e-mail are intended for the named addressee only. It 
> contains information that may be confidential. Unless you are the named 
> addressee or an authorized designee, you may not copy or use it, or disclose 
> it to anyone else. If you received it in error please notify us immediately 
> and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) 
> is a company registered in Linz whose registered office is at 4040 Linz, 
> Austria, Freistädterstraße 313

RE: Not serving read requests while running nodetool repair

[Steinmaurer, Thomas]

Hi Nicolas,

newly bootstrapping is not really an option, I think.

I hoped there might be a mode (perhaps there is, but haven’t found it yet) 
where a node might not actively participate in read requests but still writing 
newly arriving data during a repair or whatever maintenance task one thinks 
that some sort of write-only is appropriate.

Thanks,
Thomas

From: Nicolas Guyomar [mailto:nicolas.guyo...@gmail.com]
Sent: Mittwoch, 18. Oktober 2017 09:58
To: user@cassandra.apache.org
Subject: Re: Not serving read requests while running nodetool repair

Hi Thomas,

AFAIK  temporarily reading at LOCAL_QUORUM/QUORUM until nodetool repair is 
finished is the way to go. You can still disable binary/thrift on the node to 
"protect" it from acting as a coordinator, and complete its repair quietly, but 
I'm not sure that would make such a huge difference in recovery time.

If you disable gossip the node will be see as down, thus disabling repair if 
I'm correct

If repair is taking too much time, or switching to Read Quorum not feasible 
within your application, is it OK for you to shutdown the node, wipe its data 
and launch a repair on it at an appropriate time windows ?



On 18 October 2017 at 08:04, Steinmaurer, Thomas 
<thomas.steinmau...@dynatrace.com<mailto:thomas.steinmau...@dynatrace.com>> 
wrote:
Hello,

due to performance/latency reasons, we are currently reading and writing time 
series data at consistency level  ONE/ANY.

In case of a node being down and recovering after the default hinted handoff 
window of 3 hrs, we may potentially read stale data from the recovering node. 
Of course, from an operational POV, we will trigger a nodetool repair after the 
recovered node has start up, but to my understanding, this still may cause 
reading stale data from this particular node until nodetool repair is finished, 
which may take several hours. Is this correct?

Is there a way (e.g. in newer releases 3.0/3.11) to tell a node not being part 
of read requests? Something like a counterpart of nodetool drain for writes?

Other options:


-  Disabling binary/thrift: Although the node won’t act as coordinator 
then for client requests, as it won’t be available from a client perspective, 
inter-node, the recovering node is being contacted by other nodes for serving 
requests, right?

-  Disabling gossip: Basically the node is marked as DOWN, so treated 
like it is not available, thus not an option?

-  Temporarily reading at LOCAL_QUORUM/QUORUM until nodetool repair is 
finished?

Did I miss something?

Thanks,
Thomas

The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freist<https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313=gmail=g>ädterstra<https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313=gmail=g>ße
 
313<https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313=gmail=g>

The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 313

Re: Not serving read requests while running nodetool repair

[Nicolas Guyomar]

Hi Thomas,

AFAIK  temporarily reading at LOCAL_QUORUM/QUORUM until nodetool repair is
finished is the way to go. You can still disable binary/thrift on the node
to "protect" it from acting as a coordinator, and complete its repair
quietly, but I'm not sure that would make such a huge difference in
recovery time.

If you disable gossip the node will be see as down, thus disabling repair
if I'm correct

If repair is taking too much time, or switching to Read Quorum not feasible
within your application, is it OK for you to shutdown the node, wipe its
data and launch a repair on it at an appropriate time windows ?



On 18 October 2017 at 08:04, Steinmaurer, Thomas <
thomas.steinmau...@dynatrace.com> wrote:

> Hello,
>
>
>
> due to performance/latency reasons, we are currently reading and writing
> time series data at consistency level  ONE/ANY.
>
>
>
> In case of a node being down and recovering after the default hinted
> handoff window of 3 hrs, we may potentially read stale data from the
> recovering node. Of course, from an operational POV, we will trigger a
> nodetool repair after the recovered node has start up, but to my
> understanding, this still may cause reading stale data from this particular
> node until nodetool repair is finished, which may take several hours. Is
> this correct?
>
>
>
> Is there a way (e.g. in newer releases 3.0/3.11) to tell a node not being
> part of read requests? Something like a counterpart of nodetool drain for
> writes?
>
>
>
> Other options:
>
>
>
> -  Disabling binary/thrift: Although the node won’t act as
> coordinator then for client requests, as it won’t be available from a
> client perspective, inter-node, the recovering node is being contacted by
> other nodes for serving requests, right?
>
> -  Disabling gossip: Basically the node is marked as DOWN, so
> treated like it is not available, thus not an option?
>
> -  Temporarily reading at LOCAL_QUORUM/QUORUM until nodetool
> repair is finished?
>
>
>
> Did I miss something?
>
>
>
> Thanks,
>
> Thomas
>
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freist
> <https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313=gmail=g>
> ädterstra
> <https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313=gmail=g>
> ße 313
> <https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313=gmail=g>
>

Not serving read requests while running nodetool repair

[Steinmaurer, Thomas]

Hello,

due to performance/latency reasons, we are currently reading and writing time 
series data at consistency level  ONE/ANY.

In case of a node being down and recovering after the default hinted handoff 
window of 3 hrs, we may potentially read stale data from the recovering node. 
Of course, from an operational POV, we will trigger a nodetool repair after the 
recovered node has start up, but to my understanding, this still may cause 
reading stale data from this particular node until nodetool repair is finished, 
which may take several hours. Is this correct?

Is there a way (e.g. in newer releases 3.0/3.11) to tell a node not being part 
of read requests? Something like a counterpart of nodetool drain for writes?

Other options:


-  Disabling binary/thrift: Although the node won't act as coordinator 
then for client requests, as it won't be available from a client perspective, 
inter-node, the recovering node is being contacted by other nodes for serving 
requests, right?

-  Disabling gossip: Basically the node is marked as DOWN, so treated 
like it is not available, thus not an option?

-  Temporarily reading at LOCAL_QUORUM/QUORUM until nodetool repair is 
finished?

Did I miss something?

Thanks,
Thomas

The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freist?dterstra?e 313

Re: Nodetool repair -pr

[Blake Eggleston]

It will on 2.2 and higher, yes.

Also, just want to point out that it would be worth it for you to compare how 
long incremental repairs take vs full repairs in your cluster. There are some 
problems (which are fixed in 4.0) that can cause significant overstreaming when 
using incremental repair.

On September 28, 2017 at 11:46:47 AM, Dmitry Buzolin (dbuz5ga...@gmail.com) 
wrote:

Hi All, 

Can someone confirm if 

"nodetool repair -pr -j2" does run with -inc too? I see the docs mention -inc 
is set by default, but I am not sure if it is enabled when -pr option is used. 

Thanks! 
- 
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
For additional commands, e-mail: user-h...@cassandra.apache.org 

Re: quietness of full nodetool repair on large dataset

[Jeff Jirsa]

Screen and/or subrange repair (e.g. reaper)

-- 
Jeff Jirsa


> On Sep 28, 2017, at 8:23 PM, Mitch Gitman <mgit...@gmail.com> wrote:
> 
> I'm on Apache Cassandra 3.10. I'm interested in moving over to Reaper for 
> repairs, but in the meantime, I want to get nodetool repair working a little 
> more gracefully. 
> 
> What I'm noticing is that, when I'm running a repair for the first time with 
> the --full option after a large initial load of data, the client will say 
> it's starting on a repair job and then cease to produce any output for not 
> just minutes but a few hours. This causes SSH inactivity timeouts. I have 
> tried running the repair with the --trace option, but then that leads to the 
> other extreme where there's just a torrent of output, scarcely any of which 
> I'll typically need. 
> 
> As a literal solution to my SSH inactivity timeouts, I could extend the 
> timeouts, or I could do some scripting jujitsu with StrictHostKeyChecking=no 
> and a loop that spits some arbitrary output until the command finishes. But 
> even if the timeouts were no concern, the sheer unresponsiveness is apt to 
> make an operator nervous. And I'd like to think there's a Goldilocks way to 
> run a full nodetool repair on a large dataset where it's just a bit more 
> responsive without going all TMI. Thoughts? Anyone else notice this?

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

quietness of full nodetool repair on large dataset

[Mitch Gitman]

I'm on Apache Cassandra 3.10. I'm interested in moving over to Reaper for
repairs, but in the meantime, I want to get nodetool repair working a
little more gracefully.

What I'm noticing is that, when I'm running a repair for the first time
with the --full option after a large initial load of data, the client will
say it's starting on a repair job and then cease to produce any output for
not just minutes but a few hours. This causes SSH inactivity timeouts. I
have tried running the repair with the --trace option, but then that leads
to the other extreme where there's just a torrent of output, scarcely any
of which I'll typically need.

As a literal solution to my SSH inactivity timeouts, I could extend the
timeouts, or I could do some scripting jujitsu with
StrictHostKeyChecking=no and a loop that spits some arbitrary output until
the command finishes. But even if the timeouts were no concern, the sheer
unresponsiveness is apt to make an operator nervous. And I'd like to think
there's a Goldilocks way to run a full nodetool repair on a large dataset
where it's just a bit more responsive without going all TMI. Thoughts?
Anyone else notice this?

Nodetool repair -pr

[Dmitry Buzolin]

Hi All,

Can someone confirm if

"nodetool repair -pr -j2" does run with -inc too? I see the docs mention -inc 
is set by default, but I am not sure if it is enabled when -pr option is used.

Thanks!
-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: nodetool repair failure

[Fay Hou [Storage Service] ­]

What is your GC_GRACE_SECONDS ?
What kind repair option do you use for nodetool repair on a keyspace ?
Did you start the repair on one node? did you use nodetool repair -pr ? or
just "nodetool repair keyspace" ? How many nodetool repair processes do you
use on the nodes?





On Sun, Jul 30, 2017 at 10:53 PM, Jeff Jirsa <jji...@apache.org> wrote:

>
>
> On 2017-07-27 21:36 (-0700), Mitch Gitman <mgit...@gmail.com> wrote:
> > Now, the particular symptom to which that response refers is not what I
> was
> > seeing, but the response got me thinking that perhaps the failures I was
> > getting were on account of attempting to run "nodetool repair
> > --partitioner-range" simultaneously on all the nodes in my cluster. These
> > are only three-node dev clusters, and what I would see is that the repair
> > would pass on one node but fail on the other two.
>
>
> Running nodetool repair --partitioner-range simultaneously on all nodes in
> the cluster will indeed be a problem, and the symptoms will vary widely
> based on node state / write load / compaction load. This is one of the
> times when the right answer is "don't do that" until the project comes up
> with a way to prevent you from doing it in order to protect you from
> yourself.
>
>
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Re: Question about nodetool repair

[Blake Eggleston]

Specifying a dc will only repair the data in that dc. If you leave out the dc 
flag, it will repair data in both dcs. You probably shouldn't be restricting 
repair to one dc without a good rationale for doing so.

On August 31, 2017 at 8:56:24 AM, Harper, Paul (paul.har...@aspect.com) wrote:

Hello All,

 

I have a 6 node ring with 3 nodes in DC1 and 3 nodes in DC2. I ssh into node5 
on DC2 was in a “DN” state. I ran “nodetool repair”. I’ve had this situation 
before and ran “nodetool repair -dc DC2”.  I’m trying what if anything is 
different between those commands. What are they actually doing?

 

Thanks

This email (including any attachments) is proprietary to Aspect Software, Inc. 
and may contain information that is confidential. If you have received this 
message in error, please do not read, copy or forward this message. Please 
notify the sender immediately, delete it from your system and destroy any 
copies. You may not further disclose or distribute this email or its 
attachments.

Question about nodetool repair

[Harper, Paul]

Hello All,

I have a 6 node ring with 3 nodes in DC1 and 3 nodes in DC2. I ssh into node5 
on DC2 was in a “DN” state. I ran “nodetool repair”. I’ve had this situation 
before and ran “nodetool repair -dc DC2”.  I’m trying what if anything is 
different between those commands. What are they actually doing?

Thanks
This email (including any attachments) is proprietary to Aspect Software, Inc. 
and may contain information that is confidential. If you have received this 
message in error, please do not read, copy or forward this message. Please 
notify the sender immediately, delete it from your system and destroy any 
copies. You may not further disclose or distribute this email or its 
attachments.

Re: nodetool repair failure

[Jeff Jirsa]

On 2017-07-27 21:36 (-0700), Mitch Gitman <mgit...@gmail.com> wrote: 
> Now, the particular symptom to which that response refers is not what I was
> seeing, but the response got me thinking that perhaps the failures I was
> getting were on account of attempting to run "nodetool repair
> --partitioner-range" simultaneously on all the nodes in my cluster. These
> are only three-node dev clusters, and what I would see is that the repair
> would pass on one node but fail on the other two.


Running nodetool repair --partitioner-range simultaneously on all nodes in the 
cluster will indeed be a problem, and the symptoms will vary widely based on 
node state / write load / compaction load. This is one of the times when the 
right answer is "don't do that" until the project comes up with a way to 
prevent you from doing it in order to protect you from yourself.




-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: nodetool repair failure

[kurt greaves]

You need check the node that failed validation to find the relevant error.
The IP should be in the logs of the node you started repair on.

You shouldn't run multiple repairs on the same table from multiple nodes
unless you really know what you're doing and not using vnodes. The failure
you are likely seeing is that multiple repairs are trying to occur on the
same SSTable, which will cause the repair to fail.

Re: nodetool repair failure

[Mitch Gitman]

Michael, thanks for the input. I don't think I'm going to need to upgrade
to 3.11 for the sake of getting nodetool repair working for me. Instead, I
have another plausible explanation and solution for my particular situation.

First, I should say that disk usage proved to be a red herring. There was
plenty of disk space available.

When I said that the error message I was seeing was no more precise than
"Some repair failed," I misstated things. Just above that error message was
another further detail: "Validation failed in /(IP address of host)." Of
course, that's still vague. What validation failed?

However, that extra information led me to this JIRA ticket:
https://issues.apache.org/jira/browse/CASSANDRA-10057. In particular this
comment: "If you invoke repair on multiple node at once, this can be
happen. Can you confirm? And once it happens, the error will continue
unless you restart the node since some resources remain due to the hang. I
will post the patch not to hang."

Now, the particular symptom to which that response refers is not what I was
seeing, but the response got me thinking that perhaps the failures I was
getting were on account of attempting to run "nodetool repair
--partitioner-range" simultaneously on all the nodes in my cluster. These
are only three-node dev clusters, and what I would see is that the repair
would pass on one node but fail on the other two.

So I tried running the repairs sequentially on each of the nodes. With this
change the repair works, and I have every expectation that it will continue
to work--that running repair sequentially is the solution to my particular
problem. If this is the case and repairs are intended to be run
sequentially, then that constitutes a contract change for nodetool repair.
This is the first time I'm running a repair on a multi-node cluster on
Cassandra 3.10, and only with 3.10 was I seeing this problem. I'd never
seen it previously running repairs on Cassandra 2.1 clusters, which is what
I was upgrading from.

The last comment in that particular JIRA ticket is coming from someone
reporting the same problem I'm seeing, and their experience indirectly
corroborates mine, or at least it doesn't contradict mine.

On Thu, Jul 27, 2017 at 10:26 AM, Michael Shuler <mich...@pbandjelly.org>
wrote:

> On 07/27/2017 12:10 PM, Mitch Gitman wrote:
> > I'm using Apache Cassandra 3.10.
> 
> > this is a dev cluster I'm talking about.
> 
> > Further insights welcome...
>
> Upgrade and see if one of the many fixes for 3.11.0 helped?
>
> https://github.com/apache/cassandra/blob/cassandra-3.11.
> 0/CHANGES.txt#L1-L129
>
> If you can reproduce on 3.11.0, hit JIRA with the steps to repro. There
> are several bug fixes committed to the cassandra-3.11 branch, pending a
> 3.11.1 release, but I don't see one that's particularly relevant to your
> trace.
>
> https://github.com/apache/cassandra/blob/cassandra-3.11/CHANGES.txt
>
> --
> Kind regards,
> Michael
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Re: nodetool repair failure

[Michael Shuler]

On 07/27/2017 12:10 PM, Mitch Gitman wrote:
> I'm using Apache Cassandra 3.10.

> this is a dev cluster I'm talking about.

> Further insights welcome...

Upgrade and see if one of the many fixes for 3.11.0 helped?

https://github.com/apache/cassandra/blob/cassandra-3.11.0/CHANGES.txt#L1-L129

If you can reproduce on 3.11.0, hit JIRA with the steps to repro. There
are several bug fixes committed to the cassandra-3.11 branch, pending a
3.11.1 release, but I don't see one that's particularly relevant to your
trace.

https://github.com/apache/cassandra/blob/cassandra-3.11/CHANGES.txt

-- 
Kind regards,
Michael

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: nodetool repair failure

[Mitch Gitman]

I want to add an extra data point to this thread having encountered much
the same problem. I'm using Apache Cassandra 3.10. I attempted to run an
incremental repair that was optimized to take advantage of some downtime
where the cluster is not fielding traffic and only repair each node's
primary partitioner range:
nodetool repair --partitioner-range

On a couple nodes, I was seeing the repair fail with the vague "Some repair
failed" message:
[2017-07-27 15:30:59,283] Some repair failed
[2017-07-27 15:30:59,286] Repair command #2 finished in 10 seconds
error: Repair job has failed with the error message: [2017-07-27
15:30:59,283] Some repair failed
-- StackTrace --
java.lang.RuntimeException: Repair job has failed with the error message:
[2017-07-27 15:30:59,283] Some repair failed
at org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:116)
at
org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:583)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:533)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108)

Running with the --trace option yielded no additional relevant information.

On one node where this was arising, I was able to run the repair again with
just the keyspace of interest, see that work, run the repair another time
across all keyspaces, and see that work as well.

On another node, just trying again did not work. What did work was running
a "nodetool compact". The subsequent repair on that node succeeded, even
though it took inordinately long. Strangely, another repair after that
failed. But then the next couple succeeded.

I proceeded to do a "df -h" on the Ubuntu hosts and noticed that the disk
usage was inordinately high. This is my hypothesis as to the underlying
cause. Fortunately for me, this is a dev cluster I'm talking about.

Pertinent troubleshooting steps:
* nodetool compact
* Check disk usage. Better yet, preemptively alert on disk usage exceeding
a certain threshold.

Further insights welcome...

Re: "nodetool repair -dc"

[Anuj Wadehra]

Hi, 
I have not used dc local repair specifically but generally repair syncs all 
local tokens of the node with other replicas (full repair) or a subset of local 
tokens (-pr and subrange). Full repair with - Dc option should only sync data 
for all the tokens present on the node where the command is run with other 
replicas in local dc.
You should run full repair on all nodes of the DC unless RF of all keyspaces in 
local DC =number of nodes in DC. E.g if you have 3 nodes in dc1 and RF is 
DC1:3, repairing single node should sync all data within a DC. This doesnt hold 
true if you have 5 nodes and no node holds 100% data. 
Running full repair on all nodes in a dc may lead to repairing every data RF 
times. Inefficient!!  And you cant use pr with dc option.   Even if its allowed 
it wont repair entire ring as a dc owns a subset of entire token ring. 
ThanksAnuj
 
 
  On Tue, 11 Jul 2017 at 20:08, vasu gunja<vasu.no...@gmail.com> wrote:   Hi ,
My Question specific to -dc option 
Do we need to run this on all nodes that belongs to that DC ?Or only on one of 
the nodes that belongs to that DC then it will repair all nodes ?

On Sat, Jul 8, 2017 at 10:56 PM, Varun Gupta <var...@uber.com> wrote:

I do not see the need to run repair, as long as cluster was in healthy state on 
adding new nodes.
On Fri, Jul 7, 2017 at 8:37 AM, vasu gunja <vasu.no...@gmail.com> wrote:

Hi , 
I have a question regarding "nodetool repair -dc" option. recently we added 
multiple nodes to one DC center, we want to perform repair only on current DC. 
Here is my question.
Do we need to perform "nodetool repair -dc" on all nodes belongs to that DC ? 
or only one node of that DC?


thanks,V



  

Re: "nodetool repair -dc"

[vasu gunja]

Hi ,

My Question specific to -dc option

Do we need to run this on all nodes that belongs to that DC ?
Or only on one of the nodes that belongs to that DC then it will repair all
nodes ?


On Sat, Jul 8, 2017 at 10:56 PM, Varun Gupta <var...@uber.com> wrote:

> I do not see the need to run repair, as long as cluster was in healthy
> state on adding new nodes.
>
> On Fri, Jul 7, 2017 at 8:37 AM, vasu gunja <vasu.no...@gmail.com> wrote:
>
>> Hi ,
>>
>> I have a question regarding "nodetool repair -dc" option. recently we
>> added multiple nodes to one DC center, we want to perform repair only on
>> current DC.
>>
>> Here is my question.
>>
>> Do we need to perform "nodetool repair -dc" on all nodes belongs to that
>> DC ?
>> or only one node of that DC?
>>
>>
>>
>> thanks,
>> V
>>
>
>

Re: "nodetool repair -dc"

[Varun Gupta]

I do not see the need to run repair, as long as cluster was in healthy
state on adding new nodes.

On Fri, Jul 7, 2017 at 8:37 AM, vasu gunja <vasu.no...@gmail.com> wrote:

> Hi ,
>
> I have a question regarding "nodetool repair -dc" option. recently we
> added multiple nodes to one DC center, we want to perform repair only on
> current DC.
>
> Here is my question.
>
> Do we need to perform "nodetool repair -dc" on all nodes belongs to that
> DC ?
> or only one node of that DC?
>
>
>
> thanks,
> V
>

"nodetool repair -dc"

[vasu gunja]

Hi ,

I have a question regarding "nodetool repair -dc" option. recently we added
multiple nodes to one DC center, we want to perform repair only on current
DC.

Here is my question.

Do we need to perform "nodetool repair -dc" on all nodes belongs to that DC
?
or only one node of that DC?



thanks,
V

RE: nodetool repair failure

[Anubhav Kale]

If possible, simply read the table under question with consistency=ALL. This 
will trigger a repair and is far more reliable than the nodetool command.

From: Balaji Venkatesan [mailto:venkatesan.bal...@gmail.com]
Sent: Thursday, June 29, 2017 7:26 PM
To: user@cassandra.apache.org
Subject: Re: nodetool repair failure

It did not help much. But other issue or error I saw when I repair the keyspace 
was it says

"Sync failed between /xx.xx.xx.93 and /xx.xx.xx.94" this was run from .91 node.



On Thu, Jun 29, 2017 at 4:44 PM, Akhil Mehra 
<akhilme...@gmail.com<mailto:akhilme...@gmail.com>> wrote:
Run the following query and see if it gives you more information:

select * from system_distributed.repair_history;

Also is there any additional logging on the nodes where the error is coming 
from. Seems to be xx.xx.xx.94 for your last run.


On 30/06/2017, at 9:43 AM, Balaji Venkatesan 
<venkatesan.bal...@gmail.com<mailto:venkatesan.bal...@gmail.com>> wrote:

The verify and scrub went without any error on the keyspace. I ran it again 
with trace mode and still the same issue


[2017-06-29 21:37:45,578] Parsing UPDATE 
system_distributed.parent_repair_history SET finished_at = toTimestamp(now()), 
successful_ranges = {'} WHERE parent_id=f1f10af0-5d12-11e7-8df9-59d19ef3dd23
[2017-06-29 21:37:45,580] Preparing statement
[2017-06-29 21:37:45,580] Determining replicas for mutation
[2017-06-29 21:37:45,580] Sending MUTATION message to /xx.xx.xx.95
[2017-06-29 21:37:45,580] Sending MUTATION message to /xx.xx.xx.94
[2017-06-29 21:37:45,580] Sending MUTATION message to /xx.xx.xx.93
[2017-06-29 21:37:45,581] REQUEST_RESPONSE message received from /xx.xx.xx.93
[2017-06-29 21:37:45,581] REQUEST_RESPONSE message received from /xx.xx.xx.94
[2017-06-29 21:37:45,581] Processing response from /xx.xx.xx.93
[2017-06-29 21:37:45,581] /xx.xx.xx.94: MUTATION message received from 
/xx.xx.xx.91
[2017-06-29 21:37:45,582] Processing response from /xx.xx.xx.94
[2017-06-29 21:37:45,582] /xx.xx.xx.93: MUTATION message received from 
/xx.xx.xx.91
[2017-06-29 21:37:45,582] /xx.xx.xx.95: MUTATION message received from 
/xx.xx.xx.91
[2017-06-29 21:37:45,582] /xx.xx.xx.94: Appending to commitlog
[2017-06-29 21:37:45,582] /xx.xx.xx.94: Adding to parent_repair_history memtable
[2017-06-29 21:37:45,582] Some repair failed
[2017-06-29 21:37:45,582] Repair command #3 finished in 1 minute 44 seconds
error: Repair job has failed with the error message: [2017-06-29 21:37:45,582] 
Some repair failed
-- StackTrace --
java.lang.RuntimeException: Repair job has failed with the error message: 
[2017-06-29 21:37:45,582] Some repair failed
at org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:116)
at 
org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77)
at 
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:583)
at 
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:533)
at 
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)
at 
com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108)



On Thu, Jun 29, 2017 at 1:36 PM, Subroto Barua 
<sbarua...@yahoo.com.invalid<mailto:sbarua...@yahoo.com.invalid>> wrote:
Balaji,

Are you repairing a specific keyspace/table? if the failure is tied to a table, 
try 'verify' and 'scrub' options on .91...see if you get any errors.




On Thursday, June 29, 2017, 12:12:14 PM PDT, Balaji Venkatesan 
<venkatesan.bal...@gmail.com<mailto:venkatesan.bal...@gmail.com>> wrote:


Thanks. I tried with trace option and there is not much info. Here are the few 
log lines just before it failed.


[2017-06-29 19:01:54,969] /xx.xx.xx.93: Sending REPAIR_MESSAGE message to 
/xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
[2017-06-29 19:01:54,969] /xx.xx.xx.92: E

Re: nodetool repair failure

[Balaji Venkatesan]

xx.92: Adding to repair_history memtable
>> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
>> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
>> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
>> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
>> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message
>> to /xx.xx.xx.91
>> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message
>> to /xx.xx.xx.91
>> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message
>> to /xx.xx.xx.91
>> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message
>> to /xx.xx.xx.91
>> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message
>> to /xx.xx.xx.91
>> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message
>> to /xx.xx.xx.91
>> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message
>> to /xx.xx.xx.91
>> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message
>> to /xx.xx.xx.91
>> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message
>> to /xx.xx.xx.91
>> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message
>> to /xx.xx.xx.91
>> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message
>> to /xx.xx.xx.91
>> [2017-06-29 19:02:04,842] Some repair failed
>> [2017-06-29 19:02:04,848] Repair command #1 finished in 1 minute 2 seconds
>> error: Repair job has failed with the error message: [2017-06-29
>> 19:02:04,842] Some repair failed
>> -- StackTrace --
>> java.lang.RuntimeException: Repair job has failed with the error message:
>> [2017-06-29 19:02:04,842] Some repair failed
>> at org.apache.cassandra.tools. RepairRunner.progress(
>> RepairRunner.java:116)
>> at org.apache.cassandra.utils. progress.jmx.
>> JMXNotificationProgressListene r.handleNotification(
>> JMXNotificationProgressListene r.java:77)
>> at com.sun.jmx.remote.internal. ClientNotifForwarder$ NotifFetcher.
>> dispatchNotification( ClientNotifForwarder.java:583)
>> at com.sun.jmx.remote.internal. ClientNotifForwarder$ NotifFetcher.doRun(
>> ClientNotifForwarder.java:533)
>> at com.sun.jmx.remote.internal. ClientNotifForwarder$ NotifFetcher.run(
>> ClientNotifForwarder.java:452)
>> at com.sun.jmx.remote.internal. ClientNotifForwarder$
>> LinearExecutor$1.run( ClientNotifForwarder.java:108)
>>
>>
>>
>> FYI I am running repair from xx.xx.xx.91 node and its a 5 node cluster
>> xx.xx.xx.91-xx.xx.xx.95
>>
>> On Wed, Jun 28, 2017 at 5:16 PM, Akhil Mehra <akhilme...@gmail.com>
>> wrote:
>>
>> nodetool repair has a trace option
>>
>> nodetool repair -tr yourkeyspacename
>>
>> see if that provides you with additional information.
>>
>> Regards,
>> Akhil
>>
>> On 28/06/2017, at 2:25 AM, Balaji Venkatesan <venkatesan.bal...@gmail.com>
>> wrote:
>>
>>
>> We use Apache Cassandra 3.10-13
>>
>> On Jun 26, 2017 8:41 PM, "Michael Shuler" <mich...@pbandjelly.org> wrote:
>>
>> What version of Cassandra?
>>
>> --
>> Michael
>>
>> On 06/26/2017 09:53 PM, Balaji Venkatesan wrote:
>> > Hi All,
>> >
>> > When I run nodetool repair on a keyspace I constantly get  "Some repair
>> > failed" error, there are no sufficient info to debug more. Any help?
>> >
>> > Here is the stacktrace
>> >
>> > == == ==
>> > [2017-06-27 02:44:34,275] Some repair failed
>> > [2017-06-27 02:44:34,279] Repair command #3 finished in 33 seconds
>> > error: Repair job has failed with the error message: [2017-06-27
>> > 02:44:34,275] Some repair failed
>> > -- StackTrace --
>> > java.lang.RuntimeException: Repair job has failed with the error
>> > message: [2017-06-27 02:44:34,275] Some repair failed
>> > at org.apache.cassandra.tools.Rep airRunner.progress(RepairRunne
>> r.java:116)
>> > at
>> > org.apache.cassandra.utils.pro gress.jmx.JMXNotificationProgr
>> essListener. handleNotification(JMXNotifica tionProgressListener.java:77)
>> > at
>> > com.sun.jmx.remote.internal.Cl <http://com.sun.jmx.remote.internal.cl/>
>> ientNotifForwarder$NotifFetche r.dispatchNotification(ClientN
>> otifForwarder.java:583)
>> > at
>> > com.sun.jmx.remote.internal.Cl <http://com.sun.jmx.remote.internal.cl/>
>> ientNotifForwarder$NotifFetche r.doRun(ClientNotifForwarder. java:533)
>> > at
>> > com.sun.jmx.remote.internal.Cl <http://com.sun.jmx.remote.internal.cl/>
>> ientNotifForwarder$NotifFetche r.run(ClientNotifForwarder. java:452)
>> > at
>> > com.sun.jmx.remote.internal.Cl <http://com.sun.jmx.remote.internal.cl/>
>> ientNotifForwarder$LinearExecu tor$1.run(ClientNotifForwarder .java:108)
>> > == == ==
>> >
>> >
>> > --
>> > Thanks,
>> > Balaji Venkatesan.
>>
>>
>> -- -- -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apa che.org
>> <user-unsubscr...@cassandra.apache.org>
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>>
>>
>>
>>
>>
>> --
>> Thanks,
>> Balaji Venkatesan.
>>
>
>
>
> --
> Thanks,
> Balaji Venkatesan.
>
>
>


-- 
Thanks,
Balaji Venkatesan.

Re: nodetool repair failure

[Akhil Mehra]

01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to 
> /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to 
> /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to 
> /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to 
> /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to 
> /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to 
> /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to 
> /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to 
> /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to 
> /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to 
> /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to 
> /xx.xx.xx.91
> [2017-06-29 19:02:04,842] Some repair failed
> [2017-06-29 19:02:04,848] Repair command #1 finished in 1 minute 2 seconds
> error: Repair job has failed with the error message: [2017-06-29 
> 19:02:04,842] Some repair failed
> -- StackTrace --
> java.lang.RuntimeException: Repair job has failed with the error message: 
> [2017-06-29 19:02:04,842] Some repair failed
>   at org.apache.cassandra.tools. RepairRunner.progress( 
> RepairRunner.java:116)
>   at org.apache.cassandra.utils. progress.jmx. 
> JMXNotificationProgressListene r.handleNotification( 
> JMXNotificationProgressListene r.java:77)
>   at com.sun.jmx.remote.internal. ClientNotifForwarder$ NotifFetcher. 
> dispatchNotification( ClientNotifForwarder.java:583)
>   at com.sun.jmx.remote.internal. ClientNotifForwarder$ 
> NotifFetcher.doRun( ClientNotifForwarder.java:533)
>   at com.sun.jmx.remote.internal. ClientNotifForwarder$ NotifFetcher.run( 
> ClientNotifForwarder.java:452)
>   at com.sun.jmx.remote.internal. ClientNotifForwarder$ 
> LinearExecutor$1.run( ClientNotifForwarder.java:108)
> 
> 
> 
> FYI I am running repair from xx.xx.xx.91 node and its a 5 node cluster 
> xx.xx.xx.91-xx.xx.xx.95
> 
> On Wed, Jun 28, 2017 at 5:16 PM, Akhil Mehra <akhilme...@gmail.com 
> <mailto:akhilme...@gmail.com>> wrote:
> nodetool repair has a trace option 
> 
> nodetool repair -tr yourkeyspacename
> 
> see if that provides you with additional information.
> 
> Regards,
> Akhil 
> 
>> On 28/06/2017, at 2:25 AM, Balaji Venkatesan <venkatesan.bal...@gmail.com 
>> <mailto:venkatesan.bal...@gmail.com>> wrote:
>> 
>> 
>> We use Apache Cassandra 3.10-13 
>> 
>> On Jun 26, 2017 8:41 PM, "Michael Shuler" <mich...@pbandjelly.org 
>> <mailto:mich...@pbandjelly.org>> wrote:
>> What version of Cassandra?
>> 
>> --
>> Michael
>> 
>> On 06/26/2017 09:53 PM, Balaji Venkatesan wrote:
>> > Hi All,
>> >
>> > When I run nodetool repair on a keyspace I constantly get  "Some repair
>> > failed" error, there are no sufficient info to debug more. Any help?
>> >
>> > Here is the stacktrace
>> >
>> > == == ==
>> > [2017-06-27 02:44:34,275] Some repair failed
>> > [2017-06-27 02:44:34,279] Repair command #3 finished in 33 seconds
>> > error: Repair job has failed with the error message: [2017-06-27
>> > 02:44:34,275] Some repair failed
>> > -- StackTrace --
>> > java.lang.RuntimeException: Repair job has failed with the error
>> > message: [2017-06-27 02:44:34,275] Some repair failed
>> > at org.apache.cassandra.tools.Rep airRunner.progress(RepairRunne 
>> > r.java:116)
>> > at
>> > org.apache.cassandra.utils.pro <http://org.apache.cassandra.utils.pro/> 
>> > gress.jmx.JMXNotificationProgr essListener. handleNotification(JMXNotifica 
>> > tionProgressListener.java:77)
>> > at
>> > com.sun.jmx.remote.internal.Cl <http://com.sun.jmx.remote.internal.cl/> 
>> > ientNotifForwarder$NotifFetche r.dispatchNotification(ClientN 
>> > otifForwarder.java:583)
>> > at
>> > com.sun.jmx.remote.internal.Cl <http://com.sun.jmx.remote.internal.cl/> 
>> > ientNotifForwarder$NotifFetche r.doRun(ClientNotifForwarder. java:533)
>> > at
>> > com.sun.jmx.remote.internal.Cl <http://com.sun.jmx.remote.internal.cl/> 
>> > ientNotifForwarder$NotifFetche r.run(ClientNotifForwarder. java:452)
>> > at
>> > com.sun.jmx.remote.internal.Cl <http://com.sun.jmx.remote.internal.cl/> 
>> > ientNotifForwarder$LinearExecu tor$1.run(ClientNotifForwarder .java:108)
>> > == == ==
>> >
>> >
>> > --
>> > Thanks,
>> > Balaji Venkatesan.
>> 
>> 
>> -- -- -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apa che.org 
>> <mailto:user-unsubscr...@cassandra.apache.org>
>> For additional commands, e-mail: user-h...@cassandra.apache.org 
>> <mailto:user-h...@cassandra.apache.org>
>> 
>> 
> 
> 
> 
> 
> -- 
> Thanks,
> Balaji Venkatesan.
> 
> 
> 
> -- 
> Thanks,
> Balaji Venkatesan.

Re: nodetool repair failure

[Balaji Venkatesan]

91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message
> to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message
> to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message
> to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message
> to /xx.xx.xx.91
> [2017-06-29 19:02:04,842] Some repair failed
> [2017-06-29 19:02:04,848] Repair command #1 finished in 1 minute 2 seconds
> error: Repair job has failed with the error message: [2017-06-29
> 19:02:04,842] Some repair failed
> -- StackTrace --
> java.lang.RuntimeException: Repair job has failed with the error message:
> [2017-06-29 19:02:04,842] Some repair failed
> at org.apache.cassandra.tools. RepairRunner.progress(
> RepairRunner.java:116)
> at org.apache.cassandra.utils. progress.jmx.
> JMXNotificationProgressListene r.handleNotification(
> JMXNotificationProgressListene r.java:77)
> at com.sun.jmx.remote.internal. ClientNotifForwarder$ NotifFetcher.
> dispatchNotification( ClientNotifForwarder.java:583)
> at com.sun.jmx.remote.internal. ClientNotifForwarder$ NotifFetcher.doRun(
> ClientNotifForwarder.java:533)
> at com.sun.jmx.remote.internal. ClientNotifForwarder$ NotifFetcher.run(
> ClientNotifForwarder.java:452)
> at com.sun.jmx.remote.internal. ClientNotifForwarder$
> LinearExecutor$1.run( ClientNotifForwarder.java:108)
>
>
>
> FYI I am running repair from xx.xx.xx.91 node and its a 5 node cluster
> xx.xx.xx.91-xx.xx.xx.95
>
> On Wed, Jun 28, 2017 at 5:16 PM, Akhil Mehra <akhilme...@gmail.com> wrote:
>
> nodetool repair has a trace option
>
> nodetool repair -tr yourkeyspacename
>
> see if that provides you with additional information.
>
> Regards,
> Akhil
>
> On 28/06/2017, at 2:25 AM, Balaji Venkatesan <venkatesan.bal...@gmail.com>
> wrote:
>
>
> We use Apache Cassandra 3.10-13
>
> On Jun 26, 2017 8:41 PM, "Michael Shuler" <mich...@pbandjelly.org> wrote:
>
> What version of Cassandra?
>
> --
> Michael
>
> On 06/26/2017 09:53 PM, Balaji Venkatesan wrote:
> > Hi All,
> >
> > When I run nodetool repair on a keyspace I constantly get  "Some repair
> > failed" error, there are no sufficient info to debug more. Any help?
> >
> > Here is the stacktrace
> >
> > == == ==
> > [2017-06-27 02:44:34,275] Some repair failed
> > [2017-06-27 02:44:34,279] Repair command #3 finished in 33 seconds
> > error: Repair job has failed with the error message: [2017-06-27
> > 02:44:34,275] Some repair failed
> > -- StackTrace --
> > java.lang.RuntimeException: Repair job has failed with the error
> > message: [2017-06-27 02:44:34,275] Some repair failed
> > at org.apache.cassandra.tools.Rep airRunner.progress(RepairRunne
> r.java:116)
> > at
> > org.apache.cassandra.utils.pro gress.jmx.JMXNotificationProgr
> essListener. handleNotification(JMXNotifica tionProgressListener.java:77)
> > at
> > com.sun.jmx.remote.internal.Cl ientNotifForwarder$NotifFetche
> r.dispatchNotification(ClientN otifForwarder.java:583)
> > at
> > com.sun.jmx.remote.internal.Cl ientNotifForwarder$NotifFetche
> r.doRun(ClientNotifForwarder. java:533)
> > at
> > com.sun.jmx.remote.internal.Cl ientNotifForwarder$NotifFetche
> r.run(ClientNotifForwarder. java:452)
> > at
> > com.sun.jmx.remote.internal.Cl ientNotifForwarder$LinearExecu
> tor$1.run(ClientNotifForwarder .java:108)
> > == == ==
> >
> >
> > --
> > Thanks,
> > Balaji Venkatesan.
>
>
> -- -- -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apa che.org
> <user-unsubscr...@cassandra.apache.org>
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>
>
>
>
>
> --
> Thanks,
> Balaji Venkatesan.
>



-- 
Thanks,
Balaji Venkatesan.

Re: nodetool repair failure

[Subroto Barua]

Balaji,
Are you repairing a specific keyspace/table? if the failure is tied to a table, 
try 'verify' and 'scrub' options on .91...see if you get any errors.



On Thursday, June 29, 2017, 12:12:14 PM PDT, Balaji Venkatesan 
<venkatesan.bal...@gmail.com> wrote:

Thanks. I tried with trace option and there is not much info. Here are the few 
log lines just before it failed.

[2017-06-29 19:01:54,969] /xx.xx.xx.93: Sending REPAIR_MESSAGE message to 
/xx.xx.xx.91[2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to 
commitlog[2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history 
memtable[2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to 
/xx.xx.xx.91[2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to 
commitlog[2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history 
memtable[2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to 
/xx.xx.xx.91[2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to 
commitlog[2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history 
memtable[2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to 
/xx.xx.xx.91[2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to 
commitlog[2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history 
memtable[2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to 
/xx.xx.xx.91[2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to 
commitlog[2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history 
memtable[2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to 
/xx.xx.xx.91[2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to 
commitlog[2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history 
memtable[2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to 
/xx.xx.xx.91[2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE 
message to /xx.xx.xx.91[2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending 
REQUEST_RESPONSE message to /xx.xx.xx.91[2017-06-29 19:01:54,969] /xx.xx.xx.92: 
Sending REQUEST_RESPONSE message to /xx.xx.xx.91[2017-06-29 19:01:54,969] 
/xx.xx.xx.92: Sending REQUEST_RESPONSE message to /xx.xx.xx.91[2017-06-29 
19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to 
/xx.xx.xx.91[2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE 
message to /xx.xx.xx.91[2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending 
REQUEST_RESPONSE message to /xx.xx.xx.91[2017-06-29 19:01:54,969] /xx.xx.xx.92: 
Sending REQUEST_RESPONSE message to /xx.xx.xx.91[2017-06-29 19:01:54,969] 
/xx.xx.xx.92: Sending REQUEST_RESPONSE message to /xx.xx.xx.91[2017-06-29 
19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to 
/xx.xx.xx.91[2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE 
message to /xx.xx.xx.91[2017-06-29 19:02:04,842] Some repair failed[2017-06-29 
19:02:04,848] Repair command #1 finished in 1 minute 2 secondserror: Repair job 
has failed with the error message: [2017-06-29 19:02:04,842] Some repair 
failed-- StackTrace --java.lang.RuntimeException: Repair job has failed with 
the error message: [2017-06-29 19:02:04,842] Some repair failed at 
org.apache.cassandra.tools. RepairRunner.progress( RepairRunner.java:116) at 
org.apache.cassandra.utils. progress.jmx. JMXNotificationProgressListene 
r.handleNotification( JMXNotificationProgressListene r.java:77) at 
com.sun.jmx.remote.internal. ClientNotifForwarder$ NotifFetcher. 
dispatchNotification( ClientNotifForwarder.java:583) at 
com.sun.jmx.remote.internal. ClientNotifForwarder$ NotifFetcher.doRun( 
ClientNotifForwarder.java:533) at com.sun.jmx.remote.internal. 
ClientNotifForwarder$ NotifFetcher.run( ClientNotifForwarder.java:452) at 
com.sun.jmx.remote.internal. ClientNotifForwarder$ LinearExecutor$1.run( 
ClientNotifForwarder.java:108)


FYI I am running repair from xx.xx.xx.91 node and its a 5 node cluster 
xx.xx.xx.91-xx.xx.xx.95
On Wed, Jun 28, 2017 at 5:16 PM, Akhil Mehra <akhilme...@gmail.com> wrote:

nodetool repair has a trace option 
nodetool repair -tr yourkeyspacename
see if that provides you with additional information.
Regards,Akhil 

On 28/06/2017, at 2:25 AM, Balaji Venkatesan <venkatesan.bal...@gmail.com> 
wrote:

We use Apache Cassandra 3.10-13 

On Jun 26, 2017 8:41 PM, "Michael Shuler" <mich...@pbandjelly.org> wrote:

What version of Cassandra?

--
Michael

On 06/26/2017 09:53 PM, Balaji Venkatesan wrote:
> Hi All,
>
> When I run nodetool repair on a keyspace I constantly get  "Some repair
> failed" error, there are no sufficient info to debug more. Any help?
>
> Here is the stacktrace
>
> == == ==
> [2017-06-27 02:44:34,275] Some repair failed
> [2017-06-27 02:44:34,279] Repair command #3 finished in 33 seconds
> error: Repair job has failed with the error message: [2017-06-27
> 02:44:34,275] Some repair failed
> -- StackTrace --
> java.lang.RuntimeException: Repair job has failed with the error
> 

Re: nodetool repair failure

[Balaji Venkatesan]

Thanks. I tried with trace option and there is not much info. Here are the
few log lines just before it failed.


[2017-06-29 19:01:54,969] /xx.xx.xx.93: Sending REPAIR_MESSAGE message to
/xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to
/xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to
/xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to
/xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to
/xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to
/xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to
/xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to
/xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to
/xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to
/xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to
/xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to
/xx.xx.xx.91
[2017-06-29 19:02:04,842] Some repair failed
[2017-06-29 19:02:04,848] Repair command #1 finished in 1 minute 2 seconds
error: Repair job has failed with the error message: [2017-06-29
19:02:04,842] Some repair failed
-- StackTrace --
java.lang.RuntimeException: Repair job has failed with the error message:
[2017-06-29 19:02:04,842] Some repair failed
at org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:116)
at org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListene
r.handleNotification(JMXNotificationProgressListener.java:77)
at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.
dispatchNotification(ClientNotifForwarder.java:583)
at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(
ClientNotifForwarder.java:533)
at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(
ClientNotifForwarder.java:452)
at com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(
ClientNotifForwarder.java:108)



FYI I am running repair from xx.xx.xx.91 node and its a 5 node cluster
xx.xx.xx.91-xx.xx.xx.95

On Wed, Jun 28, 2017 at 5:16 PM, Akhil Mehra <akhilme...@gmail.com> wrote:

> nodetool repair has a trace option
>
> nodetool repair -tr yourkeyspacename
>
> see if that provides you with additional information.
>
> Regards,
> Akhil
>
> On 28/06/2017, at 2:25 AM, Balaji Venkatesan <venkatesan.bal...@gmail.com>
> wrote:
>
>
> We use Apache Cassandra 3.10-13
>
> On Jun 26, 2017 8:41 PM, "Michael Shuler" <mich...@pbandjelly.org> wrote:
>
> What version of Cassandra?
>
> --
> Michael
>
> On 06/26/2017 09:53 PM, Balaji Venkatesan wrote:
> > Hi All,
> >
> > When I run nodetool repair on a keyspace I constantly get  "Some repair
> > failed" error, there are no sufficient info to debug more. Any help?
> >
> > Here is the stacktrace
> >
> > ==
> > [2017-06-27 02:44:34,275] Some repair failed
> > [2017-06-27 02:44:34,279] Repair command #3 finished in 33 seconds
> > error: Repair job has failed with the error message: [2017-06-27
> > 02:44:34,275] Some repair failed
> > -- StackTrace --
> > java.lang.RuntimeException: Repair job has failed with the error
> > message: [2017-06-27 02:44:34,275] Some repair failed
> > at org.apache.cassa

Re: nodetool repair failure

[Akhil Mehra]

nodetool repair has a trace option 

nodetool repair -tr yourkeyspacename

see if that provides you with additional information.

Regards,
Akhil 

> On 28/06/2017, at 2:25 AM, Balaji Venkatesan <venkatesan.bal...@gmail.com> 
> wrote:
> 
> 
> We use Apache Cassandra 3.10-13 
> 
> On Jun 26, 2017 8:41 PM, "Michael Shuler" <mich...@pbandjelly.org 
> <mailto:mich...@pbandjelly.org>> wrote:
> What version of Cassandra?
> 
> --
> Michael
> 
> On 06/26/2017 09:53 PM, Balaji Venkatesan wrote:
> > Hi All,
> >
> > When I run nodetool repair on a keyspace I constantly get  "Some repair
> > failed" error, there are no sufficient info to debug more. Any help?
> >
> > Here is the stacktrace
> >
> > ==
> > [2017-06-27 02:44:34,275] Some repair failed
> > [2017-06-27 02:44:34,279] Repair command #3 finished in 33 seconds
> > error: Repair job has failed with the error message: [2017-06-27
> > 02:44:34,275] Some repair failed
> > -- StackTrace --
> > java.lang.RuntimeException: Repair job has failed with the error
> > message: [2017-06-27 02:44:34,275] Some repair failed
> > at org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:116)
> > at
> > org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77)
> > at
> > com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:583)
> > at
> > com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:533)
> > at
> > com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)
> > at
> > com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108)
> > ==
> >
> >
> > --
> > Thanks,
> > Balaji Venkatesan.
> 
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
> <mailto:user-unsubscr...@cassandra.apache.org>
> For additional commands, e-mail: user-h...@cassandra.apache.org 
> <mailto:user-h...@cassandra.apache.org>
> 
> 

Re: nodetool repair failure

[Balaji Venkatesan]

We use Apache Cassandra 3.10-13

On Jun 26, 2017 8:41 PM, "Michael Shuler" <mich...@pbandjelly.org> wrote:

What version of Cassandra?

--
Michael

On 06/26/2017 09:53 PM, Balaji Venkatesan wrote:
> Hi All,
>
> When I run nodetool repair on a keyspace I constantly get  "Some repair
> failed" error, there are no sufficient info to debug more. Any help?
>
> Here is the stacktrace
>
> ==
> [2017-06-27 02:44:34,275] Some repair failed
> [2017-06-27 02:44:34,279] Repair command #3 finished in 33 seconds
> error: Repair job has failed with the error message: [2017-06-27
> 02:44:34,275] Some repair failed
> -- StackTrace --
> java.lang.RuntimeException: Repair job has failed with the error
> message: [2017-06-27 02:44:34,275] Some repair failed
> at org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:116)
> at
> org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListene
r.handleNotification(JMXNotificationProgressListener.java:77)
> at
> com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.
dispatchNotification(ClientNotifForwarder.java:583)
> at
> com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(
ClientNotifForwarder.java:533)
> at
> com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(
ClientNotifForwarder.java:452)
> at
> com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(
ClientNotifForwarder.java:108)
> ==
>
>
> --
> Thanks,
> Balaji Venkatesan.


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: nodetool repair failure

[Michael Shuler]

What version of Cassandra?

-- 
Michael

On 06/26/2017 09:53 PM, Balaji Venkatesan wrote:
> Hi All,
> 
> When I run nodetool repair on a keyspace I constantly get  "Some repair
> failed" error, there are no sufficient info to debug more. Any help? 
> 
> Here is the stacktrace
> 
> ==
> [2017-06-27 02:44:34,275] Some repair failed
> [2017-06-27 02:44:34,279] Repair command #3 finished in 33 seconds
> error: Repair job has failed with the error message: [2017-06-27
> 02:44:34,275] Some repair failed
> -- StackTrace --
> java.lang.RuntimeException: Repair job has failed with the error
> message: [2017-06-27 02:44:34,275] Some repair failed
> at org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:116)
> at
> org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77)
> at
> com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:583)
> at
> com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:533)
> at
> com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)
> at
> com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108)
> ==
> 
> 
> -- 
> Thanks,
> Balaji Venkatesan.


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

nodetool repair failure

[Balaji Venkatesan]

Hi All,

When I run nodetool repair on a keyspace I constantly get  "Some repair
failed" error, there are no sufficient info to debug more. Any help?

Here is the stacktrace

==
[2017-06-27 02:44:34,275] Some repair failed
[2017-06-27 02:44:34,279] Repair command #3 finished in 33 seconds
error: Repair job has failed with the error message: [2017-06-27
02:44:34,275] Some repair failed
-- StackTrace --
java.lang.RuntimeException: Repair job has failed with the error message:
[2017-06-27 02:44:34,275] Some repair failed
at org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:116)
at
org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:583)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:533)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108)
==


-- 
Thanks,
Balaji Venkatesan.