Re: nodetool repair failing with "Validation failed in /X.X.X.X

[Rhys Campbell]

Hello Shalom,

Someone already tried a rolling restart of Cassandra. I will probably try
rebooting the OS.

Repair seems to work if you do it a keyspace at a time.

Thanks for your input.

Rhys

On Sun, May 5, 2019 at 2:14 PM shalom sagges  wrote:

> Hi Rhys,
>
> I encountered this error after adding new SSTables to a cluster and
> running nodetool refresh (v3.0.12).
> The refresh worked, but after starting repairs on the cluster, I got the
> "Validation failed in /X.X.X.X" error on the remote DC.
> A rolling restart solved the issue for me.
>
> Hope this helps!
>
>
>
> On Sat, May 4, 2019 at 3:58 PM Rhys Campbell
>  wrote:
>
>>
>> > Hello,
>> >
>> > I’m having issues running repair on an Apache Cassandra Cluster. I’m
>> getting "Failed creating a merkle tree“ errors on the replication partner
>> nodes. Anyone have any experience of this? I am running 2.2.13.
>> >
>> > Further details here…
>> https://issues.apache.org/jira/projects/CASSANDRA/issues/CASSANDRA-15109?filter=allopenissues
>> >
>> > Best,
>> >
>> > Rhys
>>
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>>

Re: nodetool repair failing with "Validation failed in /X.X.X.X

[shalom sagges]

Hi Rhys,

I encountered this error after adding new SSTables to a cluster and running
nodetool refresh (v3.0.12).
The refresh worked, but after starting repairs on the cluster, I got the
"Validation failed in /X.X.X.X" error on the remote DC.
A rolling restart solved the issue for me.

Hope this helps!



On Sat, May 4, 2019 at 3:58 PM Rhys Campbell
 wrote:

>
> > Hello,
> >
> > I’m having issues running repair on an Apache Cassandra Cluster. I’m
> getting "Failed creating a merkle tree“ errors on the replication partner
> nodes. Anyone have any experience of this? I am running 2.2.13.
> >
> > Further details here…
> https://issues.apache.org/jira/projects/CASSANDRA/issues/CASSANDRA-15109?filter=allopenissues
> >
> > Best,
> >
> > Rhys
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Re: nodetool repair -pr

[Arvinder Dhillon]

It depends on your data model. -pr only repair primary range. So if there
is a keyspace with replication 'DC2:3', and you run repair -pr only on all
nodes of DC1, it is not going to repair token ranges corsponding to DC2. So
you will have to run on each node.

-Arvinder

On Fri, Jun 8, 2018, 8:42 PM Igor Zubchenok  wrote:

> According docs at
> http://cassandra.apache.org/doc/latest/operating/repair.html?highlight=single
>
>
> *The -pr flag will only repair the “primary” ranges on a node, so you can
> repair your entire cluster by running nodetool repair -pr on each node in
> a single datacenter.*
> But I saw many places, where it is noted that I should run it at ALL data
> centers.
>
> Looking for a qualified answer.
>
>
> On Fri, 8 Jun 2018 at 18:08 Igor Zubchenok  wrote:
>
>> I want to repair all nodes at all data centers.
>>
>> Example:
>> DC1
>>  nodeA
>>  nodeB
>>  nodeC
>> DC2
>>  node D
>>  node E
>>  node F
>>
>> If I run `nodetool repair -pr` at nodeA nodeB and nodeC, will all ranges
>> be repaired?
>>
>>
>> On Fri, 8 Jun 2018 at 17:57 Rahul Singh 
>> wrote:
>>
>>> From DS dox : "Do not use -pr with this option to repair only a local
>>> data center."
>>> On Jun 8, 2018, 10:42 AM -0400, user@cassandra.apache.org, wrote:
>>>
>>>
>>> *nodetool repair -pr*
>>>
>>>

Re: nodetool repair -pr

[Igor Zubchenok]

According docs at
http://cassandra.apache.org/doc/latest/operating/repair.html?highlight=single


*The -pr flag will only repair the “primary” ranges on a node, so you can
repair your entire cluster by running nodetool repair -pr on each node in
a single datacenter.*
But I saw many places, where it is noted that I should run it at ALL data
centers.

Looking for a qualified answer.


On Fri, 8 Jun 2018 at 18:08 Igor Zubchenok  wrote:

> I want to repair all nodes at all data centers.
>
> Example:
> DC1
>  nodeA
>  nodeB
>  nodeC
> DC2
>  node D
>  node E
>  node F
>
> If I run `nodetool repair -pr` at nodeA nodeB and nodeC, will all ranges
> be repaired?
>
>
> On Fri, 8 Jun 2018 at 17:57 Rahul Singh 
> wrote:
>
>> From DS dox : "Do not use -pr with this option to repair only a local
>> data center."
>> On Jun 8, 2018, 10:42 AM -0400, user@cassandra.apache.org, wrote:
>>
>>
>> *nodetool repair -pr*
>>
>>

Re: nodetool repair -pr

[Igor Zubchenok]

I want to repair all nodes at all data centers.

Example:
DC1
 nodeA
 nodeB
 nodeC
DC2
 node D
 node E
 node F

If I run `nodetool repair -pr` at nodeA nodeB and nodeC, will all ranges be
repaired?


On Fri, 8 Jun 2018 at 17:57 Rahul Singh 
wrote:

> From DS dox : "Do not use -pr with this option to repair only a local
> data center."
> On Jun 8, 2018, 10:42 AM -0400, user@cassandra.apache.org, wrote:
>
>
> *nodetool repair -pr*
>
> --
Regards,
Igor Zubchenok

CTO at Multi Brains LLC
Founder of taxistartup.com saytaxi.com chauffy.com
Skype: igor.zubchenok

Re: nodetool repair -pr

[Rahul Singh]

>From DS dox : "Do not use -pr with this option to repair only a local data 
>center."
On Jun 8, 2018, 10:42 AM -0400, user@cassandra.apache.org, wrote:
>
> nodetool repair -pr

Re: Nodetool repair multiple dc

[Abdul Patel]

One quick question on reaper ..what data is stored in reaper_db keyspace ?
And how much does it grow?
Do we have to cleanup that frequently or reaper has mechnism to slef clean ?

On Friday, April 13, 2018, Alexander Dejanovski 
wrote:

> Hi Abdul,
>
> Reaper has been used in production for several years now, by many
> companies.
> I've seen it handling 100s of clusters and 1000s of nodes with a single
> Reaper process.
> Check the docs on cassandra-reaper.io to see which architecture matches
> your cluster : http://cassandra-reaper.io/docs/usage/multi_dc/
>
> Cheers,
>
> On Fri, Apr 13, 2018 at 4:38 PM Rahul Singh 
> wrote:
>
>> Makes sense it takes a long time since it has to reconcile against
>> replicas in all DCs. I leverage commercial tools for production clusters,
>> but I’m pretty sure Reaper is the best open source option. Otherwise you’ll
>> waste a lot of time trying to figure it out own your own. No need to
>> reinvent the wheel.
>>
>> On Apr 12, 2018, 11:02 PM -0400, Abdul Patel ,
>> wrote:
>>
>> Hi All,
>>
>> I have 18 node cluster across 3 dc , if i tey to run incremental repair
>> on singke node it takes forever sometome 45 to 1hr and sometime times out
>> ..so i started running "nodetool repair -dc dc1" for each dc one by one
>> ..which works fine ..do we have an better way to handle this?
>> I am thinking abouy exploring cassandra reaper ..does anyone has used
>> that in prod?
>>
>> --
> -
> Alexander Dejanovski
> France
> @alexanderdeja
>
> Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>

Re: Nodetool repair multiple dc

[Alexander Dejanovski]

Hi Abdul,

Reaper has been used in production for several years now, by many companies.
I've seen it handling 100s of clusters and 1000s of nodes with a single
Reaper process.
Check the docs on cassandra-reaper.io to see which architecture matches
your cluster : http://cassandra-reaper.io/docs/usage/multi_dc/

Cheers,

On Fri, Apr 13, 2018 at 4:38 PM Rahul Singh 
wrote:

> Makes sense it takes a long time since it has to reconcile against
> replicas in all DCs. I leverage commercial tools for production clusters,
> but I’m pretty sure Reaper is the best open source option. Otherwise you’ll
> waste a lot of time trying to figure it out own your own. No need to
> reinvent the wheel.
>
> On Apr 12, 2018, 11:02 PM -0400, Abdul Patel , wrote:
>
> Hi All,
>
> I have 18 node cluster across 3 dc , if i tey to run incremental repair on
> singke node it takes forever sometome 45 to 1hr and sometime times out ..so
> i started running "nodetool repair -dc dc1" for each dc one by one ..which
> works fine ..do we have an better way to handle this?
> I am thinking abouy exploring cassandra reaper ..does anyone has used that
> in prod?
>
> --
-
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

Re: Nodetool repair multiple dc

[Rahul Singh]

Makes sense it takes a long time since it has to reconcile against replicas in 
all DCs. I leverage commercial tools for production clusters, but I’m pretty 
sure Reaper is the best open source option. Otherwise you’ll waste a lot of 
time trying to figure it out own your own. No need to reinvent the wheel.

On Apr 12, 2018, 11:02 PM -0400, Abdul Patel , wrote:
> Hi All,
>
> I have 18 node cluster across 3 dc , if i tey to run incremental repair on 
> singke node it takes forever sometome 45 to 1hr and sometime times out ..so i 
> started running "nodetool repair -dc dc1" for each dc one by one ..which 
> works fine ..do we have an better way to handle this?
> I am thinking abouy exploring cassandra reaper ..does anyone has used that in 
> prod?

Re: nodetool repair and compact

[Alain RODRIGUEZ]

I have just this been told that my first statement is inaccurate:

If  'upgradesstable' is run as a routine operation, you might forget about
> it and suffer consequences. 'upgradesstable' is not only doing the
> compaction.


I should probably have checked upgradesstable closely before making this
statement and I definitely will.

Yet, I believe the second point still holds though: 'With UDC, you can
trigger the compaction of the sstables you want to remove the tombstones
from, instead of compacting *all* the sstables for a given table.'

C*heers,

2018-04-02 16:39 GMT+01:00 Alain RODRIGUEZ :

> Hi,
>
> it will re-write this table's sstable files to current version, while
>> re-writing, will evit droppable tombstones (expired +  gc_grace_seconds
>> (default 10 days) ), if partition cross different files, they will still
>> be kept, but most droppable tombstones gone and size reduced.
>>
>
> Nice tip James, I never thought about doing this, it could have been handy
> :).
>
> Now, these compactions can be automatically done using the proper
> tombstone compaction settings in most cases. Generally, tombstone
> compaction is enabled, but if tombstone eviction is still an issue, you
> might want to give a try enabling 'unchecked_tombstone_compaction' in the
> table options. This might claim quite a lot of disk space (depending on the
> sstable overlapping levels).
>
> In case manual action is really needed (even more if it is run
> automatically), I would recommend using 'User Defined Compactions' - UDC
> (accessible through JMX at least) instead of 'uprade sstable':
>
> - It will remove the tombstones the same way, but with no side effect if
> you are currently upgrading for example. If  'upgradesstable' is run as a
> routine operation, you might forget about it and suffer consequences.
> 'upgradesstable' is not only doing the compaction.
> - With UDC, you can trigger the compaction of the sstables you want to
> remove the tombstones from, instead of compacting *all* the sstables for a
> given table.
>
> This last point can prevent harming the cluster with useless compaction,
> and even allow the operator to do things like: 'Compact the 10% biggest
> sstables, that have an estimated tombstone ratio above 0.5, every day' or
> 'compact any sstable having more than 75% of tombstones' as you see fit,
> and using information such as the sstables sizes and sstablemetadata to get
> the tombstone ratio.
>
> C*heers,
> ---
> Alain Rodriguez - @arodream - al...@thelastpickle.com
> France / Spain
>
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> 2018-04-02 14:55 GMT+01:00 James Shaw :
>
>> you may use:  nodetool upgradesstables -a keyspace_name table_name
>> it will re-write this table's sstable files to current version, while
>> re-writing, will evit droppable tombstones (expired +  gc_grace_seconds
>> (default 10 days) ), if partition cross different files, they will still
>> be kept, but most droppable tombstones gone and size reduced.
>> It works well for ours.
>>
>>
>>
>> On Mon, Apr 2, 2018 at 12:45 AM, Jon Haddad  wrote:
>>
>>> You’ll find the answers to your questions (and quite a bit more) in this
>>> blog post from my coworker: http://thelastpickle
>>> .com/blog/2016/07/27/about-deletes-and-tombstones.html
>>>
>>> Repair doesn’t clean up tombstones, they’re only removed through
>>> compaction.  I advise taking care with nodetool compact, most of the time
>>> it’s not a great idea for a variety of reasons.  Check out the above post,
>>> if you still have questions, ask away.
>>>
>>>
>>> On Apr 1, 2018, at 9:41 PM, Xiangfei Ni  wrote:
>>>
>>> Hi All,
>>>   I want to delete the expired tombstone, someone uses nodetool repair
>>> ,but someone uses compact,so I want to know which one is the correct way,
>>>   I have read the below pages from Datastax,but the page just tells us
>>> how to use the command,but doesn’t tell us what it is exactly dose,
>>>   https://docs.datastax.com/en/cassandra/3.0/cassandra/tools
>>> /toolsRepair.html
>>>could anybody tell me how to clean the tombstone and give me some
>>> materials include the detailed instruction about the nodetool command and
>>> options?Web link is also ok.
>>>   Thanks very much
>>> Best Regards,
>>>
>>> 倪项菲*/ **David Ni*
>>> 中移德电网络科技有限公司
>>>
>>> Virtue Intelligent Network Ltd, co.
>>> Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
>>> Mob: +86 13797007811 <+86%20137%209700%207811>|Tel: + 86 27 5024 2516
>>> <+86%2027%205024%202516>
>>>
>>>
>>>
>>
>

Re: nodetool repair and compact

[Alain RODRIGUEZ]

Hi,

it will re-write this table's sstable files to current version, while
> re-writing, will evit droppable tombstones (expired +  gc_grace_seconds
> (default 10 days) ), if partition cross different files, they will still
> be kept, but most droppable tombstones gone and size reduced.
>

Nice tip James, I never thought about doing this, it could have been handy
:).

Now, these compactions can be automatically done using the proper tombstone
compaction settings in most cases. Generally, tombstone compaction is
enabled, but if tombstone eviction is still an issue, you might want to
give a try enabling 'unchecked_tombstone_compaction' in the table options.
This might claim quite a lot of disk space (depending on the sstable
overlapping levels).

In case manual action is really needed (even more if it is run
automatically), I would recommend using 'User Defined Compactions' - UDC
(accessible through JMX at least) instead of 'uprade sstable':

- It will remove the tombstones the same way, but with no side effect if
you are currently upgrading for example. If  'upgradesstable' is run as a
routine operation, you might forget about it and suffer consequences.
'upgradesstable' is not only doing the compaction.
- With UDC, you can trigger the compaction of the sstables you want to
remove the tombstones from, instead of compacting *all* the sstables for a
given table.

This last point can prevent harming the cluster with useless compaction,
and even allow the operator to do things like: 'Compact the 10% biggest
sstables, that have an estimated tombstone ratio above 0.5, every day' or
'compact any sstable having more than 75% of tombstones' as you see fit,
and using information such as the sstables sizes and sstablemetadata to get
the tombstone ratio.

C*heers,
---
Alain Rodriguez - @arodream - al...@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2018-04-02 14:55 GMT+01:00 James Shaw :

> you may use:  nodetool upgradesstables -a keyspace_name table_name
> it will re-write this table's sstable files to current version, while
> re-writing, will evit droppable tombstones (expired +  gc_grace_seconds
> (default 10 days) ), if partition cross different files, they will still
> be kept, but most droppable tombstones gone and size reduced.
> It works well for ours.
>
>
>
> On Mon, Apr 2, 2018 at 12:45 AM, Jon Haddad  wrote:
>
>> You’ll find the answers to your questions (and quite a bit more) in this
>> blog post from my coworker: http://thelastpickle
>> .com/blog/2016/07/27/about-deletes-and-tombstones.html
>>
>> Repair doesn’t clean up tombstones, they’re only removed through
>> compaction.  I advise taking care with nodetool compact, most of the time
>> it’s not a great idea for a variety of reasons.  Check out the above post,
>> if you still have questions, ask away.
>>
>>
>> On Apr 1, 2018, at 9:41 PM, Xiangfei Ni  wrote:
>>
>> Hi All,
>>   I want to delete the expired tombstone, someone uses nodetool repair
>> ,but someone uses compact,so I want to know which one is the correct way,
>>   I have read the below pages from Datastax,but the page just tells us
>> how to use the command,but doesn’t tell us what it is exactly dose,
>>   https://docs.datastax.com/en/cassandra/3.0/cassandra/tools
>> /toolsRepair.html
>>could anybody tell me how to clean the tombstone and give me some
>> materials include the detailed instruction about the nodetool command and
>> options?Web link is also ok.
>>   Thanks very much
>> Best Regards,
>>
>> 倪项菲*/ **David Ni*
>> 中移德电网络科技有限公司
>>
>> Virtue Intelligent Network Ltd, co.
>> Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
>> Mob: +86 13797007811 <+86%20137%209700%207811>|Tel: + 86 27 5024 2516
>> <+86%2027%205024%202516>
>>
>>
>>
>

Re: nodetool repair and compact

[James Shaw]

you may use:  nodetool upgradesstables -a keyspace_name table_name
it will re-write this table's sstable files to current version, while
re-writing, will evit droppable tombstones (expired +  gc_grace_seconds
(default 10 days) ), if partition cross different files, they will still be
kept, but most droppable tombstones gone and size reduced.
It works well for ours.



On Mon, Apr 2, 2018 at 12:45 AM, Jon Haddad  wrote:

> You’ll find the answers to your questions (and quite a bit more) in this
> blog post from my coworker: http://thelastpickle.com/blog/2016/
> 07/27/about-deletes-and-tombstones.html
>
> Repair doesn’t clean up tombstones, they’re only removed through
> compaction.  I advise taking care with nodetool compact, most of the time
> it’s not a great idea for a variety of reasons.  Check out the above post,
> if you still have questions, ask away.
>
>
> On Apr 1, 2018, at 9:41 PM, Xiangfei Ni  wrote:
>
> Hi All,
>   I want to delete the expired tombstone, someone uses nodetool repair
> ,but someone uses compact,so I want to know which one is the correct way,
>   I have read the below pages from Datastax,but the page just tells us how
> to use the command,but doesn’t tell us what it is exactly dose,
>   https://docs.datastax.com/en/cassandra/3.0/cassandra/
> tools/toolsRepair.html
>could anybody tell me how to clean the tombstone and give me some
> materials include the detailed instruction about the nodetool command and
> options?Web link is also ok.
>   Thanks very much
> Best Regards,
>
> 倪项菲*/ **David Ni*
> 中移德电网络科技有限公司
>
> Virtue Intelligent Network Ltd, co.
> Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
> Mob: +86 13797007811 <+86%20137%209700%207811>|Tel: + 86 27 5024 2516
> <+86%2027%205024%202516>
>
>
>

Re: nodetool repair and compact

[Jon Haddad]

You’ll find the answers to your questions (and quite a bit more) in this blog 
post from my coworker: 
http://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html 


Repair doesn’t clean up tombstones, they’re only removed through compaction.  I 
advise taking care with nodetool compact, most of the time it’s not a great 
idea for a variety of reasons.  Check out the above post, if you still have 
questions, ask away.  


> On Apr 1, 2018, at 9:41 PM, Xiangfei Ni  wrote:
> 
> Hi All,
>   I want to delete the expired tombstone, someone uses nodetool repair ,but 
> someone uses compact,so I want to know which one is the correct way,
>   I have read the below pages from Datastax,but the page just tells us how to 
> use the command,but doesn’t tell us what it is exactly dose,
>   https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsRepair.html 
> 
>could anybody tell me how to clean the tombstone and give me some 
> materials include the detailed instruction about the nodetool command and 
> options?Web link is also ok.
>   Thanks very much
> Best Regards,
>  
> 倪项菲/ David Ni
> 中移德电网络科技有限公司
> Virtue Intelligent Network Ltd, co.
> 
> Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
> Mob: +86 13797007811|Tel: + 86 27 5024 2516

Re: Nodetool Repair --full

[kurt greaves]

Worth noting that if you have racks == RF you only need to repair one rack
to repair all the data in the cluster if you *don't* use -pr. Also note
that full repairs on >=3.0 case anti-compactions and will mark things as
repaired, so once you start repairs you need to keep repairing to ensure
you don't have any zombie data or other problems.

On 17 March 2018 at 15:52, Hannu Kröger  wrote:

> Hi Jonathan,
>
> If you want to repair just one node (for example if it has been down for
> more than 3h), run “nodetool repair -full” on that node. This will bring
> all data on that node up to date.
>
> If you want to repair all data on the cluster, run “nodetool repair -full
> -pr” on each node. This will run full repair on all nodes but it will do it
> so only the primary range for each node is fixed. If you do it on all
> nodes, effectively the whole token range is repaired. You can run the same
> without -pr to get the same effect but it’s not efficient because then you
> are doing the repair RF times on all data instead of just repairing the
> whole data once.
>
> I hope this clarifies,
> Hannu
>
> On 17 Mar 2018, at 17:20, Jonathan Baynes 
> wrote:
>
> Hi Community,
>
> Can someone confirm, as the documentation out on the web is so
> contradictory and vague.
>
> Nodetool repair –full if I call this, do I need to run this on ALL my
> nodes or is just the once sufficient?
>
> Thanks
> J
>
> *Jonathan Baynes*
> DBA
> Tradeweb Europe Limited
> Moor Place  •  1 Fore Street Avenue
> 
>   •
> 
>   London EC2Y 9DT
> 
> P +44 (0)20 77760988 <+44%2020%207776%200988>  •  F +44 (0)20 7776 3201
> <+44%2020%207776%203201>  •  M +44 (0)7884111546 <+44%207884%20111546>
> jonathan.bay...@tradeweb.com
>
>     follow us:  **
>    <
> image003.jpg> 
> —
> A leading marketplace  for
> electronic fixed income, derivatives and ETF trading
>
>
> 
>
> This e-mail may contain confidential and/or privileged information. If you
> are not the intended recipient (or have received this e-mail in error)
> please notify the sender immediately and destroy it. Any unauthorized
> copying, disclosure or distribution of the material in this e-mail is
> strictly forbidden. Tradeweb reserves the right to monitor all e-mail
> communications through its networks. If you do not wish to receive
> marketing emails about our products / services, please let us know by
> contacting us, either by email at contac...@tradeweb.com or by writing to
> us at the registered office of Tradeweb in the UK, which is: Tradeweb
> Europe Limited (company number 3912826), 1 Fore Street Avenue London EC2Y
> 9DT
> .
> To see our privacy policy, visit our website @ www.tradeweb.com.
>
>
>

Re: Nodetool Repair --full

[Hannu Kröger]

Hi Jonathan,

If you want to repair just one node (for example if it has been down for more 
than 3h), run “nodetool repair -full” on that node. This will bring all data on 
that node up to date.

If you want to repair all data on the cluster, run “nodetool repair -full -pr” 
on each node. This will run full repair on all nodes but it will do it so only 
the primary range for each node is fixed. If you do it on all nodes, 
effectively the whole token range is repaired. You can run the same without -pr 
to get the same effect but it’s not efficient because then you are doing the 
repair RF times on all data instead of just repairing the whole data once.

I hope this clarifies,
Hannu

> On 17 Mar 2018, at 17:20, Jonathan Baynes  
> wrote:
> 
> Hi Community,
>  
> Can someone confirm, as the documentation out on the web is so contradictory 
> and vague.
>  
> Nodetool repair –full if I call this, do I need to run this on ALL my nodes 
> or is just the once sufficient?
>  
> Thanks
> J
>  
> Jonathan Baynes
> DBA
> Tradeweb Europe Limited
> Moor Place  •  1 Fore Street Avenue  •  London EC2Y 9DT
> P +44 (0)20 77760988  •  F +44 (0)20 7776 3201  •  M +44 (0)7884111546
> jonathan.bay...@tradeweb.com 
>  
>     follow us:   
> 
> 
> —
> A leading marketplace  for 
> electronic fixed income, derivatives and ETF trading
>  
> 
> 
> This e-mail may contain confidential and/or privileged information. If you 
> are not the intended recipient (or have received this e-mail in error) please 
> notify the sender immediately and destroy it. Any unauthorized copying, 
> disclosure or distribution of the material in this e-mail is strictly 
> forbidden. Tradeweb reserves the right to monitor all e-mail communications 
> through its networks. If you do not wish to receive marketing emails about 
> our products / services, please let us know by contacting us, either by email 
> at contac...@tradeweb.com  or by writing to us 
> at the registered office of Tradeweb in the UK, which is: Tradeweb Europe 
> Limited (company number 3912826), 1 Fore Street Avenue London EC2Y 9DT. To 
> see our privacy policy, visit our website @ www.tradeweb.com 
> .
> 

Re: Nodetool repair on read only cluster

[Jeff Jirsa]

Over time the various nodes likely got slightly out of sync - dropped mutations 
primarily, during Long GC pauses or maybe network failures

In that case, repair will make all of the data match - how long it takes 
depends on size of data (more data takes longer to validate), size of your 
partitions (big partitions are more work to repair), and how you invoke repair



-- 
Jeff Jirsa


> On Nov 29, 2017, at 5:42 PM, Roger Warner  wrote:
> 
>  
> What would running a repair on a cluster do when there are no deletes nor 
> have there ever been?I have no deletes yet on my data.Yet running a 
> repair took over 9 hours on a 5 node cluster?
>  
> Roger?

Re: Nodetool repair on read only cluster

[@Nandan@]

Hi Roger,
As you provide incomplete information which is so tough to analyse .
But if you like to refer then please check below JIRA link to check out is
it useful or not. ?
https://issues.apache.org/jira/browse/CASSANDRA-6616

Thanks.

On Thu, Nov 30, 2017 at 9:42 AM, Roger Warner  wrote:

>
>
> What would running a repair on a cluster do when there are no deletes nor
> have there ever been?I have no deletes yet on my data.Yet running a
> repair took over 9 hours on a 5 node cluster?
>
>
>
> Roger?
>

Re: Nodetool repair -pr

[Blake Eggleston]

It will on 2.2 and higher, yes.

Also, just want to point out that it would be worth it for you to compare how 
long incremental repairs take vs full repairs in your cluster. There are some 
problems (which are fixed in 4.0) that can cause significant overstreaming when 
using incremental repair.

On September 28, 2017 at 11:46:47 AM, Dmitry Buzolin (dbuz5ga...@gmail.com) 
wrote:

Hi All, 

Can someone confirm if 

"nodetool repair -pr -j2" does run with -inc too? I see the docs mention -inc 
is set by default, but I am not sure if it is enabled when -pr option is used. 

Thanks! 
- 
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
For additional commands, e-mail: user-h...@cassandra.apache.org 

Re: nodetool repair failure

[Fay Hou [Storage Service] ­]

What is your GC_GRACE_SECONDS ?
What kind repair option do you use for nodetool repair on a keyspace ?
Did you start the repair on one node? did you use nodetool repair -pr ? or
just "nodetool repair keyspace" ? How many nodetool repair processes do you
use on the nodes?





On Sun, Jul 30, 2017 at 10:53 PM, Jeff Jirsa  wrote:

>
>
> On 2017-07-27 21:36 (-0700), Mitch Gitman  wrote:
> > Now, the particular symptom to which that response refers is not what I
> was
> > seeing, but the response got me thinking that perhaps the failures I was
> > getting were on account of attempting to run "nodetool repair
> > --partitioner-range" simultaneously on all the nodes in my cluster. These
> > are only three-node dev clusters, and what I would see is that the repair
> > would pass on one node but fail on the other two.
>
>
> Running nodetool repair --partitioner-range simultaneously on all nodes in
> the cluster will indeed be a problem, and the symptoms will vary widely
> based on node state / write load / compaction load. This is one of the
> times when the right answer is "don't do that" until the project comes up
> with a way to prevent you from doing it in order to protect you from
> yourself.
>
>
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Re: nodetool repair failure

[Jeff Jirsa]

On 2017-07-27 21:36 (-0700), Mitch Gitman  wrote: 
> Now, the particular symptom to which that response refers is not what I was
> seeing, but the response got me thinking that perhaps the failures I was
> getting were on account of attempting to run "nodetool repair
> --partitioner-range" simultaneously on all the nodes in my cluster. These
> are only three-node dev clusters, and what I would see is that the repair
> would pass on one node but fail on the other two.


Running nodetool repair --partitioner-range simultaneously on all nodes in the 
cluster will indeed be a problem, and the symptoms will vary widely based on 
node state / write load / compaction load. This is one of the times when the 
right answer is "don't do that" until the project comes up with a way to 
prevent you from doing it in order to protect you from yourself.




-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: nodetool repair failure

[kurt greaves]

You need check the node that failed validation to find the relevant error.
The IP should be in the logs of the node you started repair on.

You shouldn't run multiple repairs on the same table from multiple nodes
unless you really know what you're doing and not using vnodes. The failure
you are likely seeing is that multiple repairs are trying to occur on the
same SSTable, which will cause the repair to fail.

Re: nodetool repair failure

[Mitch Gitman]

Michael, thanks for the input. I don't think I'm going to need to upgrade
to 3.11 for the sake of getting nodetool repair working for me. Instead, I
have another plausible explanation and solution for my particular situation.

First, I should say that disk usage proved to be a red herring. There was
plenty of disk space available.

When I said that the error message I was seeing was no more precise than
"Some repair failed," I misstated things. Just above that error message was
another further detail: "Validation failed in /(IP address of host)." Of
course, that's still vague. What validation failed?

However, that extra information led me to this JIRA ticket:
https://issues.apache.org/jira/browse/CASSANDRA-10057. In particular this
comment: "If you invoke repair on multiple node at once, this can be
happen. Can you confirm? And once it happens, the error will continue
unless you restart the node since some resources remain due to the hang. I
will post the patch not to hang."

Now, the particular symptom to which that response refers is not what I was
seeing, but the response got me thinking that perhaps the failures I was
getting were on account of attempting to run "nodetool repair
--partitioner-range" simultaneously on all the nodes in my cluster. These
are only three-node dev clusters, and what I would see is that the repair
would pass on one node but fail on the other two.

So I tried running the repairs sequentially on each of the nodes. With this
change the repair works, and I have every expectation that it will continue
to work--that running repair sequentially is the solution to my particular
problem. If this is the case and repairs are intended to be run
sequentially, then that constitutes a contract change for nodetool repair.
This is the first time I'm running a repair on a multi-node cluster on
Cassandra 3.10, and only with 3.10 was I seeing this problem. I'd never
seen it previously running repairs on Cassandra 2.1 clusters, which is what
I was upgrading from.

The last comment in that particular JIRA ticket is coming from someone
reporting the same problem I'm seeing, and their experience indirectly
corroborates mine, or at least it doesn't contradict mine.

On Thu, Jul 27, 2017 at 10:26 AM, Michael Shuler 
wrote:

> On 07/27/2017 12:10 PM, Mitch Gitman wrote:
> > I'm using Apache Cassandra 3.10.
> 
> > this is a dev cluster I'm talking about.
> 
> > Further insights welcome...
>
> Upgrade and see if one of the many fixes for 3.11.0 helped?
>
> https://github.com/apache/cassandra/blob/cassandra-3.11.
> 0/CHANGES.txt#L1-L129
>
> If you can reproduce on 3.11.0, hit JIRA with the steps to repro. There
> are several bug fixes committed to the cassandra-3.11 branch, pending a
> 3.11.1 release, but I don't see one that's particularly relevant to your
> trace.
>
> https://github.com/apache/cassandra/blob/cassandra-3.11/CHANGES.txt
>
> --
> Kind regards,
> Michael
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Re: nodetool repair failure

[Michael Shuler]

On 07/27/2017 12:10 PM, Mitch Gitman wrote:
> I'm using Apache Cassandra 3.10.

> this is a dev cluster I'm talking about.

> Further insights welcome...

Upgrade and see if one of the many fixes for 3.11.0 helped?

https://github.com/apache/cassandra/blob/cassandra-3.11.0/CHANGES.txt#L1-L129

If you can reproduce on 3.11.0, hit JIRA with the steps to repro. There
are several bug fixes committed to the cassandra-3.11 branch, pending a
3.11.1 release, but I don't see one that's particularly relevant to your
trace.

https://github.com/apache/cassandra/blob/cassandra-3.11/CHANGES.txt

-- 
Kind regards,
Michael

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: nodetool repair failure

[Mitch Gitman]

I want to add an extra data point to this thread having encountered much
the same problem. I'm using Apache Cassandra 3.10. I attempted to run an
incremental repair that was optimized to take advantage of some downtime
where the cluster is not fielding traffic and only repair each node's
primary partitioner range:
nodetool repair --partitioner-range

On a couple nodes, I was seeing the repair fail with the vague "Some repair
failed" message:
[2017-07-27 15:30:59,283] Some repair failed
[2017-07-27 15:30:59,286] Repair command #2 finished in 10 seconds
error: Repair job has failed with the error message: [2017-07-27
15:30:59,283] Some repair failed
-- StackTrace --
java.lang.RuntimeException: Repair job has failed with the error message:
[2017-07-27 15:30:59,283] Some repair failed
at org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:116)
at
org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:583)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:533)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108)

Running with the --trace option yielded no additional relevant information.

On one node where this was arising, I was able to run the repair again with
just the keyspace of interest, see that work, run the repair another time
across all keyspaces, and see that work as well.

On another node, just trying again did not work. What did work was running
a "nodetool compact". The subsequent repair on that node succeeded, even
though it took inordinately long. Strangely, another repair after that
failed. But then the next couple succeeded.

I proceeded to do a "df -h" on the Ubuntu hosts and noticed that the disk
usage was inordinately high. This is my hypothesis as to the underlying
cause. Fortunately for me, this is a dev cluster I'm talking about.

Pertinent troubleshooting steps:
* nodetool compact
* Check disk usage. Better yet, preemptively alert on disk usage exceeding
a certain threshold.

Further insights welcome...

Re: "nodetool repair -dc"

[Anuj Wadehra]

Hi, 
I have not used dc local repair specifically but generally repair syncs all 
local tokens of the node with other replicas (full repair) or a subset of local 
tokens (-pr and subrange). Full repair with - Dc option should only sync data 
for all the tokens present on the node where the command is run with other 
replicas in local dc.
You should run full repair on all nodes of the DC unless RF of all keyspaces in 
local DC =number of nodes in DC. E.g if you have 3 nodes in dc1 and RF is 
DC1:3, repairing single node should sync all data within a DC. This doesnt hold 
true if you have 5 nodes and no node holds 100% data. 
Running full repair on all nodes in a dc may lead to repairing every data RF 
times. Inefficient!!  And you cant use pr with dc option.   Even if its allowed 
it wont repair entire ring as a dc owns a subset of entire token ring. 
ThanksAnuj
 
 
  On Tue, 11 Jul 2017 at 20:08, vasu gunja wrote:   Hi ,
My Question specific to -dc option 
Do we need to run this on all nodes that belongs to that DC ?Or only on one of 
the nodes that belongs to that DC then it will repair all nodes ?

On Sat, Jul 8, 2017 at 10:56 PM, Varun Gupta  wrote:

I do not see the need to run repair, as long as cluster was in healthy state on 
adding new nodes.
On Fri, Jul 7, 2017 at 8:37 AM, vasu gunja  wrote:

Hi , 
I have a question regarding "nodetool repair -dc" option. recently we added 
multiple nodes to one DC center, we want to perform repair only on current DC. 
Here is my question.
Do we need to perform "nodetool repair -dc" on all nodes belongs to that DC ? 
or only one node of that DC?


thanks,V



  

Re: "nodetool repair -dc"

[vasu gunja]

Hi ,

My Question specific to -dc option

Do we need to run this on all nodes that belongs to that DC ?
Or only on one of the nodes that belongs to that DC then it will repair all
nodes ?


On Sat, Jul 8, 2017 at 10:56 PM, Varun Gupta  wrote:

> I do not see the need to run repair, as long as cluster was in healthy
> state on adding new nodes.
>
> On Fri, Jul 7, 2017 at 8:37 AM, vasu gunja  wrote:
>
>> Hi ,
>>
>> I have a question regarding "nodetool repair -dc" option. recently we
>> added multiple nodes to one DC center, we want to perform repair only on
>> current DC.
>>
>> Here is my question.
>>
>> Do we need to perform "nodetool repair -dc" on all nodes belongs to that
>> DC ?
>> or only one node of that DC?
>>
>>
>>
>> thanks,
>> V
>>
>
>

Re: "nodetool repair -dc"

[Varun Gupta]

I do not see the need to run repair, as long as cluster was in healthy
state on adding new nodes.

On Fri, Jul 7, 2017 at 8:37 AM, vasu gunja  wrote:

> Hi ,
>
> I have a question regarding "nodetool repair -dc" option. recently we
> added multiple nodes to one DC center, we want to perform repair only on
> current DC.
>
> Here is my question.
>
> Do we need to perform "nodetool repair -dc" on all nodes belongs to that
> DC ?
> or only one node of that DC?
>
>
>
> thanks,
> V
>

RE: nodetool repair failure

[Anubhav Kale]

If possible, simply read the table under question with consistency=ALL. This 
will trigger a repair and is far more reliable than the nodetool command.

From: Balaji Venkatesan [mailto:venkatesan.bal...@gmail.com]
Sent: Thursday, June 29, 2017 7:26 PM
To: user@cassandra.apache.org
Subject: Re: nodetool repair failure

It did not help much. But other issue or error I saw when I repair the keyspace 
was it says

"Sync failed between /xx.xx.xx.93 and /xx.xx.xx.94" this was run from .91 node.



On Thu, Jun 29, 2017 at 4:44 PM, Akhil Mehra 
<akhilme...@gmail.com<mailto:akhilme...@gmail.com>> wrote:
Run the following query and see if it gives you more information:

select * from system_distributed.repair_history;

Also is there any additional logging on the nodes where the error is coming 
from. Seems to be xx.xx.xx.94 for your last run.


On 30/06/2017, at 9:43 AM, Balaji Venkatesan 
<venkatesan.bal...@gmail.com<mailto:venkatesan.bal...@gmail.com>> wrote:

The verify and scrub went without any error on the keyspace. I ran it again 
with trace mode and still the same issue


[2017-06-29 21:37:45,578] Parsing UPDATE 
system_distributed.parent_repair_history SET finished_at = toTimestamp(now()), 
successful_ranges = {'} WHERE parent_id=f1f10af0-5d12-11e7-8df9-59d19ef3dd23
[2017-06-29 21:37:45,580] Preparing statement
[2017-06-29 21:37:45,580] Determining replicas for mutation
[2017-06-29 21:37:45,580] Sending MUTATION message to /xx.xx.xx.95
[2017-06-29 21:37:45,580] Sending MUTATION message to /xx.xx.xx.94
[2017-06-29 21:37:45,580] Sending MUTATION message to /xx.xx.xx.93
[2017-06-29 21:37:45,581] REQUEST_RESPONSE message received from /xx.xx.xx.93
[2017-06-29 21:37:45,581] REQUEST_RESPONSE message received from /xx.xx.xx.94
[2017-06-29 21:37:45,581] Processing response from /xx.xx.xx.93
[2017-06-29 21:37:45,581] /xx.xx.xx.94: MUTATION message received from 
/xx.xx.xx.91
[2017-06-29 21:37:45,582] Processing response from /xx.xx.xx.94
[2017-06-29 21:37:45,582] /xx.xx.xx.93: MUTATION message received from 
/xx.xx.xx.91
[2017-06-29 21:37:45,582] /xx.xx.xx.95: MUTATION message received from 
/xx.xx.xx.91
[2017-06-29 21:37:45,582] /xx.xx.xx.94: Appending to commitlog
[2017-06-29 21:37:45,582] /xx.xx.xx.94: Adding to parent_repair_history memtable
[2017-06-29 21:37:45,582] Some repair failed
[2017-06-29 21:37:45,582] Repair command #3 finished in 1 minute 44 seconds
error: Repair job has failed with the error message: [2017-06-29 21:37:45,582] 
Some repair failed
-- StackTrace --
java.lang.RuntimeException: Repair job has failed with the error message: 
[2017-06-29 21:37:45,582] Some repair failed
at org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:116)
at 
org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77)
at 
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:583)
at 
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:533)
at 
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)
at 
com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108)



On Thu, Jun 29, 2017 at 1:36 PM, Subroto Barua 
<sbarua...@yahoo.com.invalid<mailto:sbarua...@yahoo.com.invalid>> wrote:
Balaji,

Are you repairing a specific keyspace/table? if the failure is tied to a table, 
try 'verify' and 'scrub' options on .91...see if you get any errors.




On Thursday, June 29, 2017, 12:12:14 PM PDT, Balaji Venkatesan 
<venkatesan.bal...@gmail.com<mailto:venkatesan.bal...@gmail.com>> wrote:


Thanks. I tried with trace option and there is not much info. Here are the few 
log lines just before it failed.


[2017-06-29 19:01:54,969] /xx.xx.xx.93: Sending REPAIR_MESSAGE message to 
/xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
[2017-06-29 19:01:54,969] /xx.xx.xx.92: E

Re: nodetool repair failure

[Balaji Venkatesan]

It did not help much. But other issue or error I saw when I repair the
keyspace was it says

"Sync failed between /xx.xx.xx.93 and /xx.xx.xx.94" this was run from .91
node.



On Thu, Jun 29, 2017 at 4:44 PM, Akhil Mehra  wrote:

> Run the following query and see if it gives you more information:
>
> select * from system_distributed.repair_history;
>
> Also is there any additional logging on the nodes where the error is
> coming from. Seems to be xx.xx.xx.94 for your last run.
>
>
> On 30/06/2017, at 9:43 AM, Balaji Venkatesan 
> wrote:
>
> The verify and scrub went without any error on the keyspace. I ran it
> again with trace mode and still the same issue
>
>
> [2017-06-29 21:37:45,578] Parsing UPDATE 
> system_distributed.parent_repair_history
> SET finished_at = toTimestamp(now()), successful_ranges = {'} WHERE
> parent_id=f1f10af0-5d12-11e7-8df9-59d19ef3dd23
> [2017-06-29 21:37:45,580] Preparing statement
> [2017-06-29 21:37:45,580] Determining replicas for mutation
> [2017-06-29 21:37:45,580] Sending MUTATION message to /xx.xx.xx.95
> [2017-06-29 21:37:45,580] Sending MUTATION message to /xx.xx.xx.94
> [2017-06-29 21:37:45,580] Sending MUTATION message to /xx.xx.xx.93
> [2017-06-29 21:37:45,581] REQUEST_RESPONSE message received from
> /xx.xx.xx.93
> [2017-06-29 21:37:45,581] REQUEST_RESPONSE message received from
> /xx.xx.xx.94
> [2017-06-29 21:37:45,581] Processing response from /xx.xx.xx.93
> [2017-06-29 21:37:45,581] /xx.xx.xx.94: MUTATION message received from
> /xx.xx.xx.91
> [2017-06-29 21:37:45,582] Processing response from /xx.xx.xx.94
> [2017-06-29 21:37:45,582] /xx.xx.xx.93: MUTATION message received from
> /xx.xx.xx.91
> [2017-06-29 21:37:45,582] /xx.xx.xx.95: MUTATION message received from
> /xx.xx.xx.91
> [2017-06-29 21:37:45,582] /xx.xx.xx.94: Appending to commitlog
> [2017-06-29 21:37:45,582] /xx.xx.xx.94: Adding to parent_repair_history
> memtable
> [2017-06-29 21:37:45,582] Some repair failed
> [2017-06-29 21:37:45,582] Repair command #3 finished in 1 minute 44 seconds
> error: Repair job has failed with the error message: [2017-06-29
> 21:37:45,582] Some repair failed
> -- StackTrace --
> java.lang.RuntimeException: Repair job has failed with the error message:
> [2017-06-29 21:37:45,582] Some repair failed
> at org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:116)
> at org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListene
> r.handleNotification(JMXNotificationProgressListener.java:77)
> at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.
> dispatchNotification(ClientNotifForwarder.java:583)
> at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(
> ClientNotifForwarder.java:533)
> at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(
> ClientNotifForwarder.java:452)
> at com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(
> ClientNotifForwarder.java:108)
>
>
>
> On Thu, Jun 29, 2017 at 1:36 PM, Subroto Barua <
> sbarua...@yahoo.com.invalid> wrote:
>
>> Balaji,
>>
>> Are you repairing a specific keyspace/table? if the failure is tied to a
>> table, try 'verify' and 'scrub' options on .91...see if you get any errors.
>>
>>
>>
>>
>> On Thursday, June 29, 2017, 12:12:14 PM PDT, Balaji Venkatesan <
>> venkatesan.bal...@gmail.com> wrote:
>>
>>
>> Thanks. I tried with trace option and there is not much info. Here are
>> the few log lines just before it failed.
>>
>>
>> [2017-06-29 19:01:54,969] /xx.xx.xx.93: Sending REPAIR_MESSAGE message to
>> /xx.xx.xx.91
>> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
>> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
>> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
>> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
>> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
>> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
>> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
>> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
>> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
>> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
>> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
>> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
>> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
>> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
>> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
>> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
>> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
>> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
>> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message

Re: nodetool repair failure

[Akhil Mehra]

Run the following query and see if it gives you more information:

select * from system_distributed.repair_history;

Also is there any additional logging on the nodes where the error is coming 
from. Seems to be xx.xx.xx.94 for your last run.


> On 30/06/2017, at 9:43 AM, Balaji Venkatesan  
> wrote:
> 
> The verify and scrub went without any error on the keyspace. I ran it again 
> with trace mode and still the same issue
> 
> 
> [2017-06-29 21:37:45,578] Parsing UPDATE 
> system_distributed.parent_repair_history SET finished_at = 
> toTimestamp(now()), successful_ranges = {'} WHERE 
> parent_id=f1f10af0-5d12-11e7-8df9-59d19ef3dd23
> [2017-06-29 21:37:45,580] Preparing statement
> [2017-06-29 21:37:45,580] Determining replicas for mutation
> [2017-06-29 21:37:45,580] Sending MUTATION message to /xx.xx.xx.95
> [2017-06-29 21:37:45,580] Sending MUTATION message to /xx.xx.xx.94
> [2017-06-29 21:37:45,580] Sending MUTATION message to /xx.xx.xx.93
> [2017-06-29 21:37:45,581] REQUEST_RESPONSE message received from /xx.xx.xx.93
> [2017-06-29 21:37:45,581] REQUEST_RESPONSE message received from /xx.xx.xx.94
> [2017-06-29 21:37:45,581] Processing response from /xx.xx.xx.93
> [2017-06-29 21:37:45,581] /xx.xx.xx.94: MUTATION message received from 
> /xx.xx.xx.91
> [2017-06-29 21:37:45,582] Processing response from /xx.xx.xx.94
> [2017-06-29 21:37:45,582] /xx.xx.xx.93: MUTATION message received from 
> /xx.xx.xx.91
> [2017-06-29 21:37:45,582] /xx.xx.xx.95: MUTATION message received from 
> /xx.xx.xx.91
> [2017-06-29 21:37:45,582] /xx.xx.xx.94: Appending to commitlog
> [2017-06-29 21:37:45,582] /xx.xx.xx.94: Adding to parent_repair_history 
> memtable
> [2017-06-29 21:37:45,582] Some repair failed
> [2017-06-29 21:37:45,582] Repair command #3 finished in 1 minute 44 seconds
> error: Repair job has failed with the error message: [2017-06-29 
> 21:37:45,582] Some repair failed
> -- StackTrace --
> java.lang.RuntimeException: Repair job has failed with the error message: 
> [2017-06-29 21:37:45,582] Some repair failed
>   at 
> org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:116)
>   at 
> org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77)
>   at 
> com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:583)
>   at 
> com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:533)
>   at 
> com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)
>   at 
> com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108)
> 
> 
> 
> On Thu, Jun 29, 2017 at 1:36 PM, Subroto Barua  > wrote:
> Balaji,
> 
> Are you repairing a specific keyspace/table? if the failure is tied to a 
> table, try 'verify' and 'scrub' options on .91...see if you get any errors.
> 
> 
> 
> 
> On Thursday, June 29, 2017, 12:12:14 PM PDT, Balaji Venkatesan 
> > wrote:
> 
> 
> Thanks. I tried with trace option and there is not much info. Here are the 
> few log lines just before it failed.
> 
> 
> [2017-06-29 19:01:54,969] /xx.xx.xx.93: Sending REPAIR_MESSAGE message to 
> /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to 
> /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to 
> /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending 

Re: nodetool repair failure

[Balaji Venkatesan]

The verify and scrub went without any error on the keyspace. I ran it again
with trace mode and still the same issue


[2017-06-29 21:37:45,578] Parsing UPDATE
system_distributed.parent_repair_history SET finished_at =
toTimestamp(now()), successful_ranges = {'} WHERE
parent_id=f1f10af0-5d12-11e7-8df9-59d19ef3dd23
[2017-06-29 21:37:45,580] Preparing statement
[2017-06-29 21:37:45,580] Determining replicas for mutation
[2017-06-29 21:37:45,580] Sending MUTATION message to /xx.xx.xx.95
[2017-06-29 21:37:45,580] Sending MUTATION message to /xx.xx.xx.94
[2017-06-29 21:37:45,580] Sending MUTATION message to /xx.xx.xx.93
[2017-06-29 21:37:45,581] REQUEST_RESPONSE message received from
/xx.xx.xx.93
[2017-06-29 21:37:45,581] REQUEST_RESPONSE message received from
/xx.xx.xx.94
[2017-06-29 21:37:45,581] Processing response from /xx.xx.xx.93
[2017-06-29 21:37:45,581] /xx.xx.xx.94: MUTATION message received from
/xx.xx.xx.91
[2017-06-29 21:37:45,582] Processing response from /xx.xx.xx.94
[2017-06-29 21:37:45,582] /xx.xx.xx.93: MUTATION message received from
/xx.xx.xx.91
[2017-06-29 21:37:45,582] /xx.xx.xx.95: MUTATION message received from
/xx.xx.xx.91
[2017-06-29 21:37:45,582] /xx.xx.xx.94: Appending to commitlog
[2017-06-29 21:37:45,582] /xx.xx.xx.94: Adding to parent_repair_history
memtable
[2017-06-29 21:37:45,582] Some repair failed
[2017-06-29 21:37:45,582] Repair command #3 finished in 1 minute 44 seconds
error: Repair job has failed with the error message: [2017-06-29
21:37:45,582] Some repair failed
-- StackTrace --
java.lang.RuntimeException: Repair job has failed with the error message:
[2017-06-29 21:37:45,582] Some repair failed
at org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:116)
at
org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:583)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:533)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108)



On Thu, Jun 29, 2017 at 1:36 PM, Subroto Barua 
wrote:

> Balaji,
>
> Are you repairing a specific keyspace/table? if the failure is tied to a
> table, try 'verify' and 'scrub' options on .91...see if you get any errors.
>
>
>
>
> On Thursday, June 29, 2017, 12:12:14 PM PDT, Balaji Venkatesan <
> venkatesan.bal...@gmail.com> wrote:
>
>
> Thanks. I tried with trace option and there is not much info. Here are the
> few log lines just before it failed.
>
>
> [2017-06-29 19:01:54,969] /xx.xx.xx.93: Sending REPAIR_MESSAGE message to
> /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message
> to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message
> to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message
> to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message
> to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message
> to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message
> to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message
> to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message
> to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE 

Re: nodetool repair failure

[Subroto Barua]

Balaji,
Are you repairing a specific keyspace/table? if the failure is tied to a table, 
try 'verify' and 'scrub' options on .91...see if you get any errors.



On Thursday, June 29, 2017, 12:12:14 PM PDT, Balaji Venkatesan 
 wrote:

Thanks. I tried with trace option and there is not much info. Here are the few 
log lines just before it failed.

[2017-06-29 19:01:54,969] /xx.xx.xx.93: Sending REPAIR_MESSAGE message to 
/xx.xx.xx.91[2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to 
commitlog[2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history 
memtable[2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to 
/xx.xx.xx.91[2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to 
commitlog[2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history 
memtable[2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to 
/xx.xx.xx.91[2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to 
commitlog[2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history 
memtable[2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to 
/xx.xx.xx.91[2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to 
commitlog[2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history 
memtable[2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to 
/xx.xx.xx.91[2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to 
commitlog[2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history 
memtable[2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to 
/xx.xx.xx.91[2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to 
commitlog[2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history 
memtable[2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to 
/xx.xx.xx.91[2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE 
message to /xx.xx.xx.91[2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending 
REQUEST_RESPONSE message to /xx.xx.xx.91[2017-06-29 19:01:54,969] /xx.xx.xx.92: 
Sending REQUEST_RESPONSE message to /xx.xx.xx.91[2017-06-29 19:01:54,969] 
/xx.xx.xx.92: Sending REQUEST_RESPONSE message to /xx.xx.xx.91[2017-06-29 
19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to 
/xx.xx.xx.91[2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE 
message to /xx.xx.xx.91[2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending 
REQUEST_RESPONSE message to /xx.xx.xx.91[2017-06-29 19:01:54,969] /xx.xx.xx.92: 
Sending REQUEST_RESPONSE message to /xx.xx.xx.91[2017-06-29 19:01:54,969] 
/xx.xx.xx.92: Sending REQUEST_RESPONSE message to /xx.xx.xx.91[2017-06-29 
19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to 
/xx.xx.xx.91[2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE 
message to /xx.xx.xx.91[2017-06-29 19:02:04,842] Some repair failed[2017-06-29 
19:02:04,848] Repair command #1 finished in 1 minute 2 secondserror: Repair job 
has failed with the error message: [2017-06-29 19:02:04,842] Some repair 
failed-- StackTrace --java.lang.RuntimeException: Repair job has failed with 
the error message: [2017-06-29 19:02:04,842] Some repair failed at 
org.apache.cassandra.tools. RepairRunner.progress( RepairRunner.java:116) at 
org.apache.cassandra.utils. progress.jmx. JMXNotificationProgressListene 
r.handleNotification( JMXNotificationProgressListene r.java:77) at 
com.sun.jmx.remote.internal. ClientNotifForwarder$ NotifFetcher. 
dispatchNotification( ClientNotifForwarder.java:583) at 
com.sun.jmx.remote.internal. ClientNotifForwarder$ NotifFetcher.doRun( 
ClientNotifForwarder.java:533) at com.sun.jmx.remote.internal. 
ClientNotifForwarder$ NotifFetcher.run( ClientNotifForwarder.java:452) at 
com.sun.jmx.remote.internal. ClientNotifForwarder$ LinearExecutor$1.run( 
ClientNotifForwarder.java:108)


FYI I am running repair from xx.xx.xx.91 node and its a 5 node cluster 
xx.xx.xx.91-xx.xx.xx.95
On Wed, Jun 28, 2017 at 5:16 PM, Akhil Mehra  wrote:

nodetool repair has a trace option 
nodetool repair -tr yourkeyspacename
see if that provides you with additional information.
Regards,Akhil 

On 28/06/2017, at 2:25 AM, Balaji Venkatesan  
wrote:

We use Apache Cassandra 3.10-13 

On Jun 26, 2017 8:41 PM, "Michael Shuler"  wrote:

What version of Cassandra?

--
Michael

On 06/26/2017 09:53 PM, Balaji Venkatesan wrote:
> Hi All,
>
> When I run nodetool repair on a keyspace I constantly get  "Some repair
> failed" error, there are no sufficient info to debug more. Any help?
>
> Here is the stacktrace
>
> == == ==
> [2017-06-27 02:44:34,275] Some repair failed
> [2017-06-27 02:44:34,279] Repair command #3 finished in 33 seconds
> error: Repair job has failed with the error message: [2017-06-27
> 02:44:34,275] Some repair failed
> -- StackTrace --
> java.lang.RuntimeException: Repair job has failed with the error
> message: [2017-06-27 02:44:34,275] Some repair failed
> at org.apache.cassandra.tools.Rep 

Re: nodetool repair failure

[Balaji Venkatesan]

Thanks. I tried with trace option and there is not much info. Here are the
few log lines just before it failed.


[2017-06-29 19:01:54,969] /xx.xx.xx.93: Sending REPAIR_MESSAGE message to
/xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to
/xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to
/xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to
/xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to
/xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to
/xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to
/xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to
/xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to
/xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to
/xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to
/xx.xx.xx.91
[2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to
/xx.xx.xx.91
[2017-06-29 19:02:04,842] Some repair failed
[2017-06-29 19:02:04,848] Repair command #1 finished in 1 minute 2 seconds
error: Repair job has failed with the error message: [2017-06-29
19:02:04,842] Some repair failed
-- StackTrace --
java.lang.RuntimeException: Repair job has failed with the error message:
[2017-06-29 19:02:04,842] Some repair failed
at org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:116)
at org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListene
r.handleNotification(JMXNotificationProgressListener.java:77)
at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.
dispatchNotification(ClientNotifForwarder.java:583)
at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(
ClientNotifForwarder.java:533)
at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(
ClientNotifForwarder.java:452)
at com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(
ClientNotifForwarder.java:108)



FYI I am running repair from xx.xx.xx.91 node and its a 5 node cluster
xx.xx.xx.91-xx.xx.xx.95

On Wed, Jun 28, 2017 at 5:16 PM, Akhil Mehra  wrote:

> nodetool repair has a trace option
>
> nodetool repair -tr yourkeyspacename
>
> see if that provides you with additional information.
>
> Regards,
> Akhil
>
> On 28/06/2017, at 2:25 AM, Balaji Venkatesan 
> wrote:
>
>
> We use Apache Cassandra 3.10-13
>
> On Jun 26, 2017 8:41 PM, "Michael Shuler"  wrote:
>
> What version of Cassandra?
>
> --
> Michael
>
> On 06/26/2017 09:53 PM, Balaji Venkatesan wrote:
> > Hi All,
> >
> > When I run nodetool repair on a keyspace I constantly get  "Some repair
> > failed" error, there are no sufficient info to debug more. Any help?
> >
> > Here is the stacktrace
> >
> > ==
> > [2017-06-27 02:44:34,275] Some repair failed
> > [2017-06-27 02:44:34,279] Repair command #3 finished in 33 seconds
> > error: Repair job has failed with the error message: [2017-06-27
> > 02:44:34,275] Some repair failed
> > -- StackTrace --
> > java.lang.RuntimeException: Repair job has failed with the error
> > message: [2017-06-27 02:44:34,275] Some repair failed
> > at org.apache.cassandra.tools.RepairRunner.progress(RepairRunne
> r.java:116)
> > at
> > org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.
> handleNotification(JMXNotificationProgressListener.java:77)
> > at
> > 

Re: nodetool repair failure

[Akhil Mehra]

nodetool repair has a trace option 

nodetool repair -tr yourkeyspacename

see if that provides you with additional information.

Regards,
Akhil 

> On 28/06/2017, at 2:25 AM, Balaji Venkatesan  
> wrote:
> 
> 
> We use Apache Cassandra 3.10-13 
> 
> On Jun 26, 2017 8:41 PM, "Michael Shuler"  > wrote:
> What version of Cassandra?
> 
> --
> Michael
> 
> On 06/26/2017 09:53 PM, Balaji Venkatesan wrote:
> > Hi All,
> >
> > When I run nodetool repair on a keyspace I constantly get  "Some repair
> > failed" error, there are no sufficient info to debug more. Any help?
> >
> > Here is the stacktrace
> >
> > ==
> > [2017-06-27 02:44:34,275] Some repair failed
> > [2017-06-27 02:44:34,279] Repair command #3 finished in 33 seconds
> > error: Repair job has failed with the error message: [2017-06-27
> > 02:44:34,275] Some repair failed
> > -- StackTrace --
> > java.lang.RuntimeException: Repair job has failed with the error
> > message: [2017-06-27 02:44:34,275] Some repair failed
> > at org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:116)
> > at
> > org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77)
> > at
> > com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:583)
> > at
> > com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:533)
> > at
> > com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)
> > at
> > com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108)
> > ==
> >
> >
> > --
> > Thanks,
> > Balaji Venkatesan.
> 
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
> 
> For additional commands, e-mail: user-h...@cassandra.apache.org 
> 
> 
> 

Re: nodetool repair failure

[Balaji Venkatesan]

We use Apache Cassandra 3.10-13

On Jun 26, 2017 8:41 PM, "Michael Shuler"  wrote:

What version of Cassandra?

--
Michael

On 06/26/2017 09:53 PM, Balaji Venkatesan wrote:
> Hi All,
>
> When I run nodetool repair on a keyspace I constantly get  "Some repair
> failed" error, there are no sufficient info to debug more. Any help?
>
> Here is the stacktrace
>
> ==
> [2017-06-27 02:44:34,275] Some repair failed
> [2017-06-27 02:44:34,279] Repair command #3 finished in 33 seconds
> error: Repair job has failed with the error message: [2017-06-27
> 02:44:34,275] Some repair failed
> -- StackTrace --
> java.lang.RuntimeException: Repair job has failed with the error
> message: [2017-06-27 02:44:34,275] Some repair failed
> at org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:116)
> at
> org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListene
r.handleNotification(JMXNotificationProgressListener.java:77)
> at
> com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.
dispatchNotification(ClientNotifForwarder.java:583)
> at
> com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(
ClientNotifForwarder.java:533)
> at
> com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(
ClientNotifForwarder.java:452)
> at
> com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(
ClientNotifForwarder.java:108)
> ==
>
>
> --
> Thanks,
> Balaji Venkatesan.


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: nodetool repair failure

[Michael Shuler]

What version of Cassandra?

-- 
Michael

On 06/26/2017 09:53 PM, Balaji Venkatesan wrote:
> Hi All,
> 
> When I run nodetool repair on a keyspace I constantly get  "Some repair
> failed" error, there are no sufficient info to debug more. Any help? 
> 
> Here is the stacktrace
> 
> ==
> [2017-06-27 02:44:34,275] Some repair failed
> [2017-06-27 02:44:34,279] Repair command #3 finished in 33 seconds
> error: Repair job has failed with the error message: [2017-06-27
> 02:44:34,275] Some repair failed
> -- StackTrace --
> java.lang.RuntimeException: Repair job has failed with the error
> message: [2017-06-27 02:44:34,275] Some repair failed
> at org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:116)
> at
> org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77)
> at
> com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:583)
> at
> com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:533)
> at
> com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)
> at
> com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108)
> ==
> 
> 
> -- 
> Thanks,
> Balaji Venkatesan.


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Nodetool repair

[Li, Guangxing]

Romain,

I was trying what you mentioned as below:

a. nodetool stop VALIDATION
b. echo run -b org.apache.cassandra.db:type=StorageService
forceTerminateAllRepairSessions | java -jar
/tmp/jmxterm/jmxterm-1.0-alpha-4-uber.jar
-l 127.0.0.1:7199

to stop a seemingly forever-going repair but seeing really odd behavior
with C* 2.0.9. Here is what I did:
1. First, I run 'nodetool tpstats' on all nodes in the cluster and seeing
only one node have 1 active pending AntiEntropySessions. All other nodes do
not have any pending or active AntiEntropySessions.
2. Then I grep 'Repair' on all logs on all nodes and seeing absolutely no
repair related activity in these logs for the past day.
3. Then on the node that has active AntiEntropySessions, I did steps 'a'
and 'b' above. Now all the sudden I start seeing repair activities, on
nodes that did not have pending AntiEntropySessions, I am seeing the
following in their logs:
INFO [NonPeriodicTasks:1] 2016-09-29 17:12:53,469 StreamingRepairTask.java
(line 87) [repair #e80e17d0-8667-11e6-a801-e172d7a67134] streaming task
succeed, returning response to /10.253.2.166
On node 10.253.2.166 which has active pending AntiEntropySessions, I am
seeing the following in the log:
INFO [AntiEntropySessions:136] 2016-09-29 17:03:02,405 RepairSession.java
(line 282) [repair #812dafe0-8666-11e6-a801-e172d7a67134] session completed
successfully

So it seems to me that by doing forceTerminateAllRepairSessions, it
actually 'wakes up' the dormant repair so it goes again. So far, the only
way I can get working to stop a repair is to restart C* node where the
repair command is initiated.

Thanks.

George.

On Fri, Sep 23, 2016 at 6:20 AM, Romain Hardouin 
wrote:

> OK. If you still have issues after setting streaming_socket_timeout_in_ms
> != 0, consider increasing request_timeout_in_ms to a high value, say 1 or 2
> minutes. See comments in https://issues.apache.org/
> jira/browse/CASSANDRA-7904
> Regarding 2.1, be sure to test incremental repair on your data before to
> run it in production ;-)
>
> Romain
>

Re: Nodetool repair

[Romain Hardouin]

OK. If you still have issues after setting streaming_socket_timeout_in_ms != 0, 
consider increasing request_timeout_in_ms to a high value, say 1 or 2 minutes. 
See comments in https://issues.apache.org/jira/browse/CASSANDRA-7904Regarding 
2.1, be sure to test incremental repair on your data before to run it in 
production ;-)
Romain

Re: Nodetool repair

[Li, Guangxing]

Thanks a lot, guys. That is lots of useful info to digest.
In my cassandra.ymal, request_timeout_in_ms is set to
1, streaming_socket_timeout_in_ms is not set hence takes default of 0.
Looks like 2.1x has made quite some improvement on this area. Besides, I
can use incremental repair. So for right now, I will kill the repair using
JMX when it hangs. I am looking into upgrading to 2.1x.
Many thank again. Great stuff!

George.

On Thu, Sep 22, 2016 at 9:47 AM, Romain Hardouin 
wrote:

> Alain, you replied faster, I didn't see your answer :-D
>

Re: Nodetool repair

[Romain Hardouin]

Alain, you replied faster, I didn't see your answer :-D

Re: Nodetool repair

[Alain RODRIGUEZ]

As Matija mentioned, my coworker Alexander worked on Reaper. I believe the
branches of most interest would be:

Incremental repairs on Reaper:
https://github.com/adejanovski/cassandra-reaper/tree/inc-repair-that-works
UI integration with incremental repairs on Reaper:
https://github.com/adejanovski/cassandra-reaper/tree/inc-repair-support-with-ui

@George

When I check the log for pattern "session completed successfully" in
> system.log, I see the last finished range occurred in 14 hours ago. So I
> think it is safe to say that the repair has hanged somehow.
>

What is your current setting for 'streaming_socket_timeout_in_ms'. You
might want to be aware of
https://issues.apache.org/jira/browse/CASSANDRA-8611 and
https://issues.apache.org/jira/browse/CASSANDRA-11840

Depending on how long the streams are expected to be, you might want to try
'360 ms (1 hour)', if you are currently using 0, or increasing this
value it is already set if you think you might be hitting
https://issues.apache.org/jira/browse/CASSANDRA-11840

In order to start another repair, do we need to 'kill' this repair. If so,
> how do we do that?


Restarting the node is a straightforward way of doing that.

If you do not want to restart for some reason, you can use JMX (
forceTerminateAllRepairSessions). If you are going to use JMX and don't
know much about it, this video of the presentation done by Nate, , another
coworker, at the Cassandra Summit 2016 might be of interest
https://www.youtube.com/watch?v=uiUThbonnpc=21=PLm-EPIkBI3YoiA-02vufoEj4CgYvIQgIk
.

C*heers,
---
Alain Rodriguez - @arodream - al...@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com


2016-09-22 16:45 GMT+02:00 Li, Guangxing :

> Romain,
>
> I had another repair that seems to just hang last night. When I did 'nodetool
> tpstats' on nodes, I see the following in the node where I initiated the
> repair:
> AntiEntropySessions   1 1
> On all other nodes, I see:
> AntiEntropySessions   0 0
> When I check the log for pattern "session completed successfully" in
> system.log, I see the last finished range occurred in 14 hours ago. So I
> think it is safe to say that the repair has hanged somehow. In order to
> start another repair, do we need to 'kill' this repair. If so, how do we do
> that?
>
> Thanks.
>
> George.
>
> On Thu, Sep 22, 2016 at 6:23 AM, Romain Hardouin 
> wrote:
>
>> I meant that pending (and active) AntiEntropySessions are a simple way to
>> check if a repair is still running on a cluster. Also have a look at
>> Cassandra reaper:
>> - https://github.com/spotify/cassandra-reaper
>>
>> - https://github.com/spodkowinski/cassandra-reaper-ui
>>
>> Best,
>> Romain
>>
>>
>>
>> Le Mercredi 21 septembre 2016 22h32, "Li, Guangxing" <
>> guangxing...@pearson.com> a écrit :
>>
>> Romain,
>>
>> I started running a new repair. If I see such behavior again, I will try
>> what you mentioned.
>>
>> Thanks.
>>
>
>

Re: Nodetool repair

[Li, Guangxing]

Romain,

I had another repair that seems to just hang last night. When I did 'nodetool
tpstats' on nodes, I see the following in the node where I initiated the
repair:
AntiEntropySessions   1 1
On all other nodes, I see:
AntiEntropySessions   0 0
When I check the log for pattern "session completed successfully" in
system.log, I see the last finished range occurred in 14 hours ago. So I
think it is safe to say that the repair has hanged somehow. In order to
start another repair, do we need to 'kill' this repair. If so, how do we do
that?

Thanks.

George.

On Thu, Sep 22, 2016 at 6:23 AM, Romain Hardouin 
wrote:

> I meant that pending (and active) AntiEntropySessions are a simple way to
> check if a repair is still running on a cluster. Also have a look at
> Cassandra reaper:
> - https://github.com/spotify/cassandra-reaper
>
> - https://github.com/spodkowinski/cassandra-reaper-ui
>
> Best,
> Romain
>
>
>
> Le Mercredi 21 septembre 2016 22h32, "Li, Guangxing" <
> guangxing...@pearson.com> a écrit :
>
> Romain,
>
> I started running a new repair. If I see such behavior again, I will try
> what you mentioned.
>
> Thanks.
>

Re: Nodetool repair

[Romain Hardouin]

I meant that pending (and active) AntiEntropySessions are a simple way to check 
if a repair is still running on a cluster. Also have a look at Cassandra reaper:
- https://github.com/spotify/cassandra-reaper

- https://github.com/spodkowinski/cassandra-reaper-ui

Best,
Romain



Le Mercredi 21 septembre 2016 22h32, "Li, Guangxing"  
a écrit :

Romain,

I started running a new repair. If I see such behavior again, I will try what 
you mentioned.

Thanks.

Re: Nodetool repair

[Li, Guangxing]

Romain,

I started running a new repair. If I see such behavior again, I will try
what you mentioned.

Thanks.

On Wed, Sep 21, 2016 at 9:51 AM, Romain Hardouin 
wrote:

> Do you see any pending AntiEntropySessions (not AntiEntropyStage) with
> nodetool tpstats on nodes?
>
> Romain
>
>
> Le Mercredi 21 septembre 2016 16h45, "Li, Guangxing" <
> guangxing...@pearson.com> a écrit :
>
>
> Alain,
>
> my script actually grep through all the log files, including those
> system.log.*. So it was probably due to a failed session. So now my script
> assumes the repair has finished (possibly due to failure) if it does not
> see any more repair related logs after 2 hours.
>
> Thanks.
>
> George.
>
> On Wed, Sep 21, 2016 at 3:03 AM, Alain RODRIGUEZ 
> wrote:
>
> Hi George,
>
> That's the best way to monitor repairs "out of the box" I could think of.
> When you're not seeing 2048 (in your case), it might be due to log rotation
> or to a session failure. Have you had a look at repair failures?
>
> I am wondering why the implementor did not put something in the log (e.g.
> ... Repair command #41 has ended...) to clearly state that the repair has
> completed.
>
>
> +1, and some informations about ranges successfully repaired and the
> ranges that failed could be a very good thing as well. It would be easy to
> then read the repair result and to know what to do next (re-run repair on
> some ranges, move to the next node, etc).
>
>
> 2016-09-20 17:00 GMT+02:00 Li, Guangxing :
>
> Hi,
>
> I am using version 2.0.9. I have been looking into the logs to see if a
> repair is finished. Each time a repair is started on a node, I am seeing
> log line like "INFO [Thread-112920] 2016-09-16 19:00:43,805
> StorageService.java (line 2646) Starting repair command #41, repairing 2048
> ranges for keyspace groupmanager" in system.log. So I know that I am
> expecting to see 2048 log lines like "INFO [AntiEntropySessions:109]
> 2016-09-16 19:27:20,662 RepairSession.java (line 282) [repair
> #8b910950-7c43-11e6-88f3-f147e a74230b] session completed successfully".
> Once I see 2048 such log lines, I know this repair has completed. But this
> is not dependable since sometimes I am seeing less than 2048 but I know
> there is no repair going on since I do not see any trace of repair in
> system.log for a long time. So it seems to me that there is a clear way to
> tell that a repair has started but there is no clear way to tell a repair
> has ended. The only thing you can do is to watch the log and if you do not
> see repair activity for a long time, the repair is done somehow. I am
> wondering why the implementor did not put something in the log (e.g. ...
> Repair command #41 has ended...) to clearly state that the repair has
> completed.
>
> Thanks.
>
> George.
>
> On Tue, Sep 20, 2016 at 2:54 AM, Jens Rantil  wrote:
>
> On Mon, Sep 19, 2016 at 3:07 PM Alain RODRIGUEZ 
> wrote:
>
> ...
>
> - The size of your data
> - The number of vnodes
> - The compaction throughput
> - The streaming throughput
> - The hardware available
> - The load of the cluster
> - ...
>
>
> I've also heard that the number of clustering keys per partition key could
> have an impact. Might be worth investigating.
>
> Cheers,
> Jens
> --
> Jens Rantil
> Backend Developer @ Tink
> Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
> For urgent matters you can reach me at +46-708-84 18 32.
>
>
>
>
>
>
>

Re: Nodetool repair

[Romain Hardouin]

Do you see any pending AntiEntropySessions (not AntiEntropyStage) with nodetool 
tpstats on nodes?
Romain
 

Le Mercredi 21 septembre 2016 16h45, "Li, Guangxing" 
 a écrit :
 

 Alain,
my script actually grep through all the log files, including those 
system.log.*. So it was probably due to a failed session. So now my script 
assumes the repair has finished (possibly due to failure) if it does not see 
any more repair related logs after 2 hours.
Thanks.
George.
On Wed, Sep 21, 2016 at 3:03 AM, Alain RODRIGUEZ  wrote:

Hi George,
That's the best way to monitor repairs "out of the box" I could think of. When 
you're not seeing 2048 (in your case), it might be due to log rotation or to a 
session failure. Have you had a look at repair failures?

I am wondering why the implementor did not put something in the log (e.g. ... 
Repair command #41 has ended...) to clearly state that the repair has completed.

+1, and some informations about ranges successfully repaired and the ranges 
that failed could be a very good thing as well. It would be easy to then read 
the repair result and to know what to do next (re-run repair on some ranges, 
move to the next node, etc).

2016-09-20 17:00 GMT+02:00 Li, Guangxing :

Hi,
I am using version 2.0.9. I have been looking into the logs to see if a repair 
is finished. Each time a repair is started on a node, I am seeing log line like 
"INFO [Thread-112920] 2016-09-16 19:00:43,805 StorageService.java (line 2646) 
Starting repair command #41, repairing 2048 ranges for keyspace groupmanager" 
in system.log. So I know that I am expecting to see 2048 log lines like "INFO 
[AntiEntropySessions:109] 2016-09-16 19:27:20,662 RepairSession.java (line 282) 
[repair #8b910950-7c43-11e6-88f3-f147e a74230b] session completed 
successfully". Once I see 2048 such log lines, I know this repair has 
completed. But this is not dependable since sometimes I am seeing less than 
2048 but I know there is no repair going on since I do not see any trace of 
repair in system.log for a long time. So it seems to me that there is a clear 
way to tell that a repair has started but there is no clear way to tell a 
repair has ended. The only thing you can do is to watch the log and if you do 
not see repair activity for a long time, the repair is done somehow. I am 
wondering why the implementor did not put something in the log (e.g. ... Repair 
command #41 has ended...) to clearly state that the repair has completed.
Thanks.
George.
On Tue, Sep 20, 2016 at 2:54 AM, Jens Rantil  wrote:

On Mon, Sep 19, 2016 at 3:07 PM Alain RODRIGUEZ  wrote:

...
- The size of your data- The number of vnodes- The compaction throughput- The 
streaming throughput- The hardware available- The load of the cluster- ...

I've also heard that the number of clustering keys per partition key could have 
an impact. Might be worth investigating.
Cheers,Jens -- 
Jens Rantil
Backend Developer @ TinkTink AB, Wallingatan 5, 111 60 Stockholm, Sweden
For urgent matters you can reach me at +46-708-84 18 32.







   

Re: Nodetool repair

[Li, Guangxing]

Alain,

my script actually grep through all the log files, including those
system.log.*. So it was probably due to a failed session. So now my script
assumes the repair has finished (possibly due to failure) if it does not
see any more repair related logs after 2 hours.

Thanks.

George.

On Wed, Sep 21, 2016 at 3:03 AM, Alain RODRIGUEZ  wrote:

> Hi George,
>
> That's the best way to monitor repairs "out of the box" I could think of.
> When you're not seeing 2048 (in your case), it might be due to log rotation
> or to a session failure. Have you had a look at repair failures?
>
> I am wondering why the implementor did not put something in the log (e.g.
>> ... Repair command #41 has ended...) to clearly state that the repair has
>> completed.
>
>
> +1, and some informations about ranges successfully repaired and the
> ranges that failed could be a very good thing as well. It would be easy to
> then read the repair result and to know what to do next (re-run repair on
> some ranges, move to the next node, etc).
>
>
> 2016-09-20 17:00 GMT+02:00 Li, Guangxing :
>
>> Hi,
>>
>> I am using version 2.0.9. I have been looking into the logs to see if a
>> repair is finished. Each time a repair is started on a node, I am seeing
>> log line like "INFO [Thread-112920] 2016-09-16 19:00:43,805
>> StorageService.java (line 2646) Starting repair command #41, repairing 2048
>> ranges for keyspace groupmanager" in system.log. So I know that I am
>> expecting to see 2048 log lines like "INFO [AntiEntropySessions:109]
>> 2016-09-16 19:27:20,662 RepairSession.java (line 282) [repair
>> #8b910950-7c43-11e6-88f3-f147ea74230b] session completed successfully".
>> Once I see 2048 such log lines, I know this repair has completed. But this
>> is not dependable since sometimes I am seeing less than 2048 but I know
>> there is no repair going on since I do not see any trace of repair in
>> system.log for a long time. So it seems to me that there is a clear way to
>> tell that a repair has started but there is no clear way to tell a repair
>> has ended. The only thing you can do is to watch the log and if you do not
>> see repair activity for a long time, the repair is done somehow. I am
>> wondering why the implementor did not put something in the log (e.g. ...
>> Repair command #41 has ended...) to clearly state that the repair has
>> completed.
>>
>> Thanks.
>>
>> George.
>>
>> On Tue, Sep 20, 2016 at 2:54 AM, Jens Rantil  wrote:
>>
>>> On Mon, Sep 19, 2016 at 3:07 PM Alain RODRIGUEZ 
>>> wrote:
>>>
>>> ...
>>>
 - The size of your data
 - The number of vnodes
 - The compaction throughput
 - The streaming throughput
 - The hardware available
 - The load of the cluster
 - ...

>>>
>>> I've also heard that the number of clustering keys per partition key
>>> could have an impact. Might be worth investigating.
>>>
>>> Cheers,
>>> Jens
>>> --
>>>
>>> Jens Rantil
>>> Backend Developer @ Tink
>>>
>>> Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
>>> For urgent matters you can reach me at +46-708-84 18 32.
>>>
>>
>>
>

Re: Nodetool repair

[Alain RODRIGUEZ]

Hi George,

That's the best way to monitor repairs "out of the box" I could think of.
When you're not seeing 2048 (in your case), it might be due to log rotation
or to a session failure. Have you had a look at repair failures?

I am wondering why the implementor did not put something in the log (e.g.
> ... Repair command #41 has ended...) to clearly state that the repair has
> completed.


+1, and some informations about ranges successfully repaired and the ranges
that failed could be a very good thing as well. It would be easy to then
read the repair result and to know what to do next (re-run repair on some
ranges, move to the next node, etc).


2016-09-20 17:00 GMT+02:00 Li, Guangxing :

> Hi,
>
> I am using version 2.0.9. I have been looking into the logs to see if a
> repair is finished. Each time a repair is started on a node, I am seeing
> log line like "INFO [Thread-112920] 2016-09-16 19:00:43,805
> StorageService.java (line 2646) Starting repair command #41, repairing 2048
> ranges for keyspace groupmanager" in system.log. So I know that I am
> expecting to see 2048 log lines like "INFO [AntiEntropySessions:109]
> 2016-09-16 19:27:20,662 RepairSession.java (line 282) [repair
> #8b910950-7c43-11e6-88f3-f147ea74230b] session completed successfully".
> Once I see 2048 such log lines, I know this repair has completed. But this
> is not dependable since sometimes I am seeing less than 2048 but I know
> there is no repair going on since I do not see any trace of repair in
> system.log for a long time. So it seems to me that there is a clear way to
> tell that a repair has started but there is no clear way to tell a repair
> has ended. The only thing you can do is to watch the log and if you do not
> see repair activity for a long time, the repair is done somehow. I am
> wondering why the implementor did not put something in the log (e.g. ...
> Repair command #41 has ended...) to clearly state that the repair has
> completed.
>
> Thanks.
>
> George.
>
> On Tue, Sep 20, 2016 at 2:54 AM, Jens Rantil  wrote:
>
>> On Mon, Sep 19, 2016 at 3:07 PM Alain RODRIGUEZ 
>> wrote:
>>
>> ...
>>
>>> - The size of your data
>>> - The number of vnodes
>>> - The compaction throughput
>>> - The streaming throughput
>>> - The hardware available
>>> - The load of the cluster
>>> - ...
>>>
>>
>> I've also heard that the number of clustering keys per partition key
>> could have an impact. Might be worth investigating.
>>
>> Cheers,
>> Jens
>> --
>>
>> Jens Rantil
>> Backend Developer @ Tink
>>
>> Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
>> For urgent matters you can reach me at +46-708-84 18 32.
>>
>
>

Re: Nodetool repair

[Li, Guangxing]

Hi,

I am using version 2.0.9. I have been looking into the logs to see if a
repair is finished. Each time a repair is started on a node, I am seeing
log line like "INFO [Thread-112920] 2016-09-16 19:00:43,805
StorageService.java (line 2646) Starting repair command #41, repairing 2048
ranges for keyspace groupmanager" in system.log. So I know that I am
expecting to see 2048 log lines like "INFO [AntiEntropySessions:109]
2016-09-16 19:27:20,662 RepairSession.java (line 282) [repair
#8b910950-7c43-11e6-88f3-f147ea74230b] session completed successfully".
Once I see 2048 such log lines, I know this repair has completed. But this
is not dependable since sometimes I am seeing less than 2048 but I know
there is no repair going on since I do not see any trace of repair in
system.log for a long time. So it seems to me that there is a clear way to
tell that a repair has started but there is no clear way to tell a repair
has ended. The only thing you can do is to watch the log and if you do not
see repair activity for a long time, the repair is done somehow. I am
wondering why the implementor did not put something in the log (e.g. ...
Repair command #41 has ended...) to clearly state that the repair has
completed.

Thanks.

George.

On Tue, Sep 20, 2016 at 2:54 AM, Jens Rantil  wrote:

> On Mon, Sep 19, 2016 at 3:07 PM Alain RODRIGUEZ 
> wrote:
>
> ...
>
>> - The size of your data
>> - The number of vnodes
>> - The compaction throughput
>> - The streaming throughput
>> - The hardware available
>> - The load of the cluster
>> - ...
>>
>
> I've also heard that the number of clustering keys per partition key could
> have an impact. Might be worth investigating.
>
> Cheers,
> Jens
> --
>
> Jens Rantil
> Backend Developer @ Tink
>
> Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
> For urgent matters you can reach me at +46-708-84 18 32.
>

Re: Nodetool repair

[Jens Rantil]

On Mon, Sep 19, 2016 at 3:07 PM Alain RODRIGUEZ  wrote:

...

> - The size of your data
> - The number of vnodes
> - The compaction throughput
> - The streaming throughput
> - The hardware available
> - The load of the cluster
> - ...
>

I've also heard that the number of clustering keys per partition key could
have an impact. Might be worth investigating.

Cheers,
Jens
-- 

Jens Rantil
Backend Developer @ Tink

Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
For urgent matters you can reach me at +46-708-84 18 32.

Re: Nodetool repair

[Alain RODRIGUEZ]

Hi Lokesh,

Repair is a regular, very common and yet non trivial operations in
Cassandra. A lot of people are struggling with it.

Some good talks were done about repairs during the summit, you might want
to have a look in the Datastax youtube channel in a few days :-).
https://www.youtube.com/user/DataStaxMedia

Is there a way to know in advance the ETA of manual repair before
> triggering it
>

There is not such a thing. And it is probably because the duration of the
repair is going to depend on:

- The size of your data
- The number of vnodes
- The compaction throughput
- The streaming throughput
- The hardware available
- The load of the cluster
- ...

So the best thing to do is to benchmark it in your own environment. You can
track repairs using logs. I used something like that in the past:

for i in $(echo "SELECT columnfamily_name FROM system.schema_columns WHERE
keyspace_name = ‘my_keyspace';" | cqlsh | uniq | tail -n +4 | head -n -2);
do echo Sessions synced for $i: $(grep -i "$i is fully synced"
/var/log/cassandra/system.log* | wc -l); done

Depending on your version of Cassandra - and the path to your logs - this
might work or not, you might need to adjust it. The number of "sessions"
depends on the number of nodes and of vnodes. But the number of session
will be the same on all the tables, from all the nodes if you are using the
same number of vnodes.

So you will soon have a good idea on how long it takes to repair a table /
a keyspace and some informations about the completeness of the repairs (be
aware of the rotations in the logs and of the previous repairs logs if
using the command above).

How fast repair can go will also depend on the options and techniques you
are using:

- Subranges: https://github.com/BrianGallew/cassandra_range_repair ?
- Incremental / Full repairs ?

I believe repair performs following operations -
>
> 1) Major compaction
> 2) Exchange of merkle trees with neighbouring nodes.
>

 AFAIK, a repair doesn't trigger a major compaction, but I might be wrong
> here.


Jens is right, no major compaction in there. This is how repairs (roughly)
works. There are 2 main steps:

- Compare / exchange merkle trees (done through a VALIDATION compaction,
like a compaction, but without the write phase)
- Streaming: Any mismatch detected in the previous validation is fixed by
streaming a larger block of data (read more about that:
http://www.datastax.com/dev/blog/advanced-repair-techniques)

To monitor those operations use

- validation: nodetool compactionstats -H (Look for "VALIDATION COMPACTION"
off the top of my head)
- streaming: watch -d 'nodetool netstats -H | grep -v 100%'

You should think about what would be a good repair strategy according to
your use case and workload (run repairs by night ? Use subranges ?). Keep
in mind that "nodetool repair" is useful to reduce entropy in your cluster,
and so reducing the risk of inconsistencies. Repair also prevents deleted
data from reappearing (Zombies) as long as it is run cluster-wide within
gc_grace_seconds (per table option).

What if I kill the process in the middle?


This is safe, some parts of the data will not be repair on this node,
that's it. You can either restart the node or find the right JMX command.

C*heers,
---
Alain Rodriguez - @arodream - al...@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2016-09-19 11:18 GMT+02:00 Jens Rantil :

> Hi Lokesh,
>
> Which version of Cassandra are you using? Which compaction strategy are
> you using?
>
> AFAIK, a repair doesn't trigger a major compaction, but I might be wrong
> here.
>
> What you could do is to run a repair for a subset of the ring (see `-st`
> and `-et` `nodetool repair` parameters). If you repair 1/1000 or the ring,
> repairing the whole ring will take ~1000 longer than your sample.
>
> Also, you might want to look at incremental repairs.
>
> If you kill the process in the middle the repair will not start again. You
> will need to reissue it.
>
> Cheers,
> Jens
>
> On Sun, Sep 18, 2016 at 2:58 PM Lokesh Shrivastava <
> lokesh.shrivast...@gmail.com> wrote:
>
>> Hi,
>>
>> I tried to run nodetool repair command on one of my keyspaces and found
>> that it took lot more time than I anticipated. Is there a way to know in
>> advance the ETA of manual repair before triggering it? I believe repair
>> performs following operations -
>>
>> 1) Major compaction
>> 2) Exchange of merkle trees with neighbouring nodes.
>>
>> Is there any other operation performed during manual repair? What if I
>> kill the process in the middle?
>>
>> Thanks.
>> Lokesh
>>
> --
>
> Jens Rantil
> Backend Developer @ Tink
>
> Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
> For urgent matters you can reach me at +46-708-84 18 32.
>

Re: Nodetool repair

[Jens Rantil]

Hi Lokesh,

Which version of Cassandra are you using? Which compaction strategy are you
using?

AFAIK, a repair doesn't trigger a major compaction, but I might be wrong
here.

What you could do is to run a repair for a subset of the ring (see `-st`
and `-et` `nodetool repair` parameters). If you repair 1/1000 or the ring,
repairing the whole ring will take ~1000 longer than your sample.

Also, you might want to look at incremental repairs.

If you kill the process in the middle the repair will not start again. You
will need to reissue it.

Cheers,
Jens

On Sun, Sep 18, 2016 at 2:58 PM Lokesh Shrivastava <
lokesh.shrivast...@gmail.com> wrote:

> Hi,
>
> I tried to run nodetool repair command on one of my keyspaces and found
> that it took lot more time than I anticipated. Is there a way to know in
> advance the ETA of manual repair before triggering it? I believe repair
> performs following operations -
>
> 1) Major compaction
> 2) Exchange of merkle trees with neighbouring nodes.
>
> Is there any other operation performed during manual repair? What if I
> kill the process in the middle?
>
> Thanks.
> Lokesh
>
-- 

Jens Rantil
Backend Developer @ Tink

Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
For urgent matters you can reach me at +46-708-84 18 32.

Re: nodetool repair uses option '-local' and '-pr' togather

[Paulo Motta]

You're right Christopher, I missed the fact that with RF=3 NTS will always
place a replica on us-east-1d, so in this case repair on this node would be
sufficient. Thanks for clarifying!

2016-09-05 11:28 GMT-03:00 Christopher Bradford :

> If each AZ has a different rack identifier and the keyspace uses
> NetworkTopologyStrategy with a replication factor of 3 then the single host
> in us-east-1d *will receive 100% of the data*. This is due
> to NetworkTopologyStrategy's preference for placing replicas across
> different racks before placing a second replica in a rack where data
> already resides. Check it out with CCM:
>
> > ccm node1 status
>
> Datacenter: us-east-1
> =
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  AddressLoad   Tokens   Owns (effective)  Host ID
> Rack
> UN  127.0.0.1  98.31 KiB  140.0%
> a887ef23-c7ea-4f7a-94a4-1ed12b1caa38  us-east-1b
> UN  127.0.0.2  98.31 KiB  140.0%
> 30152aaa-cc5e-485d-9b98-1c51f1141155  us-east-1b
> UN  127.0.0.3  98.3 KiB   140.0%
> 8e1f68f7-571e-4479-bb1f-1ed526fefa9e  us-east-1c
> UN  127.0.0.4  98.31 KiB  140.0%
> 1c9b45ed-02ca-48b5-b619-a87107ff8eba  us-east-1c
> UN  127.0.0.5  98.31 KiB  140.0%
> 2a33751a-c718-44fc-8442-cce9996ebc0c  us-east-1d
>
> cqlsh> CREATE KEYSPACE replication_test WITH replication = {'class':
> 'NetworkTopologyStrategy', 'us-east-1': 3};
>
> > ccm node1 status replication_test
> Datacenter: us-east-1
> =
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  AddressLoad   Tokens   Owns (effective)  Host ID
> Rack
> UN  127.0.0.1  88.38 KiB  180.0%
> a887ef23-c7ea-4f7a-94a4-1ed12b1caa38  us-east-1b
> UN  127.0.0.2  98.31 KiB  120.0%
> 30152aaa-cc5e-485d-9b98-1c51f1141155  us-east-1b
> UN  127.0.0.3  98.3 KiB   180.0%
> 8e1f68f7-571e-4479-bb1f-1ed526fefa9e  us-east-1c
> UN  127.0.0.4  98.31 KiB  120.0%
> 1c9b45ed-02ca-48b5-b619-a87107ff8eba  us-east-1c
> UN  127.0.0.5  98.31 KiB  1100.0%
>  2a33751a-c718-44fc-8442-cce9996ebc0c  us-east-1d
>
> This can be tested further with a simple table and nodetool getendpoints.
>
> > ccm node1 nodetool getendpoints replication_test sample bar
>
> 127.0.0.2
> 127.0.0.3
> 127.0.0.5
>
> > ccm node1 nodetool getendpoints replication_test sample baz
>
> 127.0.0.1
> 127.0.0.3
> 127.0.0.5
>
> > ccm node1 nodetool getendpoints replication_test sample bif
>
> 127.0.0.3
> 127.0.0.5
> 127.0.0.1
>
> > ccm node1 nodetool getendpoints replication_test sample biz
>
> 127.0.0.2
> 127.0.0.3
> 127.0.0.5
>
> On Fri, Sep 2, 2016 at 9:41 AM Paulo Motta 
> wrote:
>
>> If I understand the way replication is done, the node in us-east-1d has
>> all the (data) replicas, right?
>>
>> No, for this to be correct, you'd need to have one DC per AZ, which is
>> not this case since you have a single DC encompassing multiple AZs. Right
>> now, replicas will be spread in 3 distinct AZs, which are represented as
>> racks in the single NTS DC if you are using EC2*Snitch. So your best bet is
>> probably to run repair -pr in all nodes.
>>
>>
>> 2016-09-01 14:28 GMT-03:00 Li, Guangxing :
>>
>>> Thanks for the info, Paulo.
>>>
>>> My cluster is in AWS, the keyspace has replication factor 3 with
>>> NetworkTopologyStrategy in one DC which have 5 nodes: 2 in us-east-1b, 2 in
>>> us-east-1c and 1 in us-east-1d. If I understand the way replication is
>>> done, the node in us-east-1d has all the (data) replicas, right? If so, if
>>> I do not use '-pr' option, would it be enough to run 'nodetool repair' ONLY
>>> on the node in us-east-1d? In other words, does 'nodetool repair' started
>>> on node in us-east-1d also cause repairs on replicas on other nodes? I am
>>> seeing different answers in discussion like this
>>> http://dba.stackexchange.com/questions/82414/do-you-hav
>>> e-to-run-nodetool-repair-on-every-node.
>>>
>>> Thanks again.
>>>
>>> George
>>>
>>> On Thu, Sep 1, 2016 at 10:22 AM, Paulo Motta 
>>> wrote:
>>>
 https://issues.apache.org/jira/browse/CASSANDRA-7450

 2016-09-01 13:11 GMT-03:00 Li, Guangxing :

> Hi,
>
> I have a cluster running 2.0.9 with 2 data centers. I noticed that
> 'nodetool repair -pr keyspace cf' runs very slow (OpsCenter shows that the
> node's data size is 39 GB and the largest SSTable size is like 7 GB so the
> column family is not huge, SizeTieredCompactionStrategy is used). 
> Repairing
> a column family on a single node takes over 5 hours. So I am wondering if 
> I
> can use option '-local' and '-pr' together, hoping to get some speed up.
> But according to documentation at https://docs.datastax.com/e
> n/cassandra/2.0/cassandra/tools/toolsRepair.html '...Do not use -pr

Re: nodetool repair uses option '-local' and '-pr' togather

[Christopher Bradford]

If each AZ has a different rack identifier and the keyspace uses
NetworkTopologyStrategy with a replication factor of 3 then the single host
in us-east-1d *will receive 100% of the data*. This is due
to NetworkTopologyStrategy's preference for placing replicas across
different racks before placing a second replica in a rack where data
already resides. Check it out with CCM:

> ccm node1 status

Datacenter: us-east-1
=
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  AddressLoad   Tokens   Owns (effective)  Host ID
Rack
UN  127.0.0.1  98.31 KiB  140.0%
a887ef23-c7ea-4f7a-94a4-1ed12b1caa38  us-east-1b
UN  127.0.0.2  98.31 KiB  140.0%
30152aaa-cc5e-485d-9b98-1c51f1141155  us-east-1b
UN  127.0.0.3  98.3 KiB   140.0%
8e1f68f7-571e-4479-bb1f-1ed526fefa9e  us-east-1c
UN  127.0.0.4  98.31 KiB  140.0%
1c9b45ed-02ca-48b5-b619-a87107ff8eba  us-east-1c
UN  127.0.0.5  98.31 KiB  140.0%
2a33751a-c718-44fc-8442-cce9996ebc0c  us-east-1d

cqlsh> CREATE KEYSPACE replication_test WITH replication = {'class':
'NetworkTopologyStrategy', 'us-east-1': 3};

> ccm node1 status replication_test
Datacenter: us-east-1
=
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  AddressLoad   Tokens   Owns (effective)  Host ID
Rack
UN  127.0.0.1  88.38 KiB  180.0%
a887ef23-c7ea-4f7a-94a4-1ed12b1caa38  us-east-1b
UN  127.0.0.2  98.31 KiB  120.0%
30152aaa-cc5e-485d-9b98-1c51f1141155  us-east-1b
UN  127.0.0.3  98.3 KiB   180.0%
8e1f68f7-571e-4479-bb1f-1ed526fefa9e  us-east-1c
UN  127.0.0.4  98.31 KiB  120.0%
1c9b45ed-02ca-48b5-b619-a87107ff8eba  us-east-1c
UN  127.0.0.5  98.31 KiB  1100.0%
 2a33751a-c718-44fc-8442-cce9996ebc0c  us-east-1d

This can be tested further with a simple table and nodetool getendpoints.

> ccm node1 nodetool getendpoints replication_test sample bar

127.0.0.2
127.0.0.3
127.0.0.5

> ccm node1 nodetool getendpoints replication_test sample baz

127.0.0.1
127.0.0.3
127.0.0.5

> ccm node1 nodetool getendpoints replication_test sample bif

127.0.0.3
127.0.0.5
127.0.0.1

> ccm node1 nodetool getendpoints replication_test sample biz

127.0.0.2
127.0.0.3
127.0.0.5

On Fri, Sep 2, 2016 at 9:41 AM Paulo Motta  wrote:

> If I understand the way replication is done, the node in us-east-1d has
> all the (data) replicas, right?
>
> No, for this to be correct, you'd need to have one DC per AZ, which is not
> this case since you have a single DC encompassing multiple AZs. Right now,
> replicas will be spread in 3 distinct AZs, which are represented as racks
> in the single NTS DC if you are using EC2*Snitch. So your best bet is
> probably to run repair -pr in all nodes.
>
>
> 2016-09-01 14:28 GMT-03:00 Li, Guangxing :
>
>> Thanks for the info, Paulo.
>>
>> My cluster is in AWS, the keyspace has replication factor 3 with
>> NetworkTopologyStrategy in one DC which have 5 nodes: 2 in us-east-1b, 2 in
>> us-east-1c and 1 in us-east-1d. If I understand the way replication is
>> done, the node in us-east-1d has all the (data) replicas, right? If so, if
>> I do not use '-pr' option, would it be enough to run 'nodetool repair' ONLY
>> on the node in us-east-1d? In other words, does 'nodetool repair' started
>> on node in us-east-1d also cause repairs on replicas on other nodes? I am
>> seeing different answers in discussion like this
>> http://dba.stackexchange.com/questions/82414/do-you-have-to-run-nodetool-repair-on-every-node
>> .
>>
>> Thanks again.
>>
>> George
>>
>> On Thu, Sep 1, 2016 at 10:22 AM, Paulo Motta 
>> wrote:
>>
>>> https://issues.apache.org/jira/browse/CASSANDRA-7450
>>>
>>> 2016-09-01 13:11 GMT-03:00 Li, Guangxing :
>>>
 Hi,

 I have a cluster running 2.0.9 with 2 data centers. I noticed that
 'nodetool repair -pr keyspace cf' runs very slow (OpsCenter shows that the
 node's data size is 39 GB and the largest SSTable size is like 7 GB so the
 column family is not huge, SizeTieredCompactionStrategy is used). Repairing
 a column family on a single node takes over 5 hours. So I am wondering if I
 can use option '-local' and '-pr' together, hoping to get some speed up.
 But according to documentation at
 https://docs.datastax.com/en/cassandra/2.0/cassandra/tools/toolsRepair.html
 '...Do not use -pr with this option to repair only a local data
 center...'. Can someone tell me the reason why we should not use options
 '-local' and '-pr' together?

 Thanks.

 George

>>>
>>>
>>
>

Re: nodetool repair uses option '-local' and '-pr' togather

[Paulo Motta]

 If I understand the way replication is done, the node in us-east-1d has
all the (data) replicas, right?

No, for this to be correct, you'd need to have one DC per AZ, which is not
this case since you have a single DC encompassing multiple AZs. Right now,
replicas will be spread in 3 distinct AZs, which are represented as racks
in the single NTS DC if you are using EC2*Snitch. So your best bet is
probably to run repair -pr in all nodes.


2016-09-01 14:28 GMT-03:00 Li, Guangxing :

> Thanks for the info, Paulo.
>
> My cluster is in AWS, the keyspace has replication factor 3 with
> NetworkTopologyStrategy in one DC which have 5 nodes: 2 in us-east-1b, 2 in
> us-east-1c and 1 in us-east-1d. If I understand the way replication is
> done, the node in us-east-1d has all the (data) replicas, right? If so, if
> I do not use '-pr' option, would it be enough to run 'nodetool repair' ONLY
> on the node in us-east-1d? In other words, does 'nodetool repair' started
> on node in us-east-1d also cause repairs on replicas on other nodes? I am
> seeing different answers in discussion like this http://dba.stackexchange.
> com/questions/82414/do-you-have-to-run-nodetool-repair-on-every-node.
>
> Thanks again.
>
> George
>
> On Thu, Sep 1, 2016 at 10:22 AM, Paulo Motta 
> wrote:
>
>> https://issues.apache.org/jira/browse/CASSANDRA-7450
>>
>> 2016-09-01 13:11 GMT-03:00 Li, Guangxing :
>>
>>> Hi,
>>>
>>> I have a cluster running 2.0.9 with 2 data centers. I noticed that
>>> 'nodetool repair -pr keyspace cf' runs very slow (OpsCenter shows that the
>>> node's data size is 39 GB and the largest SSTable size is like 7 GB so the
>>> column family is not huge, SizeTieredCompactionStrategy is used). Repairing
>>> a column family on a single node takes over 5 hours. So I am wondering if I
>>> can use option '-local' and '-pr' together, hoping to get some speed up.
>>> But according to documentation at https://docs.datastax.com/e
>>> n/cassandra/2.0/cassandra/tools/toolsRepair.html '...Do not use -pr
>>> with this option to repair only a local data center...'. Can someone tell
>>> me the reason why we should not use options '-local' and '-pr' together?
>>>
>>> Thanks.
>>>
>>> George
>>>
>>
>>
>

Re: nodetool repair uses option '-local' and '-pr' togather

[Li, Guangxing]

Thanks for the info, Paulo.

My cluster is in AWS, the keyspace has replication factor 3 with
NetworkTopologyStrategy in one DC which have 5 nodes: 2 in us-east-1b, 2 in
us-east-1c and 1 in us-east-1d. If I understand the way replication is
done, the node in us-east-1d has all the (data) replicas, right? If so, if
I do not use '-pr' option, would it be enough to run 'nodetool repair' ONLY
on the node in us-east-1d? In other words, does 'nodetool repair' started
on node in us-east-1d also cause repairs on replicas on other nodes? I am
seeing different answers in discussion like this
http://dba.stackexchange.com/questions/82414/do-you-have-to-run-nodetool-repair-on-every-node
.

Thanks again.

George

On Thu, Sep 1, 2016 at 10:22 AM, Paulo Motta 
wrote:

> https://issues.apache.org/jira/browse/CASSANDRA-7450
>
> 2016-09-01 13:11 GMT-03:00 Li, Guangxing :
>
>> Hi,
>>
>> I have a cluster running 2.0.9 with 2 data centers. I noticed that
>> 'nodetool repair -pr keyspace cf' runs very slow (OpsCenter shows that the
>> node's data size is 39 GB and the largest SSTable size is like 7 GB so the
>> column family is not huge, SizeTieredCompactionStrategy is used). Repairing
>> a column family on a single node takes over 5 hours. So I am wondering if I
>> can use option '-local' and '-pr' together, hoping to get some speed up.
>> But according to documentation at https://docs.datastax.com/e
>> n/cassandra/2.0/cassandra/tools/toolsRepair.html '...Do not use -pr with
>> this option to repair only a local data center...'. Can someone tell me the
>> reason why we should not use options '-local' and '-pr' together?
>>
>> Thanks.
>>
>> George
>>
>
>

Re: nodetool repair uses option '-local' and '-pr' togather

[Paulo Motta]

https://issues.apache.org/jira/browse/CASSANDRA-7450

2016-09-01 13:11 GMT-03:00 Li, Guangxing :

> Hi,
>
> I have a cluster running 2.0.9 with 2 data centers. I noticed that
> 'nodetool repair -pr keyspace cf' runs very slow (OpsCenter shows that the
> node's data size is 39 GB and the largest SSTable size is like 7 GB so the
> column family is not huge, SizeTieredCompactionStrategy is used). Repairing
> a column family on a single node takes over 5 hours. So I am wondering if I
> can use option '-local' and '-pr' together, hoping to get some speed up.
> But according to documentation at https://docs.datastax.com/
> en/cassandra/2.0/cassandra/tools/toolsRepair.html '...Do not use -pr with
> this option to repair only a local data center...'. Can someone tell me the
> reason why we should not use options '-local' and '-pr' together?
>
> Thanks.
>
> George
>

Re: nodetool repair with -pr and -dc

[Jérôme Mainaud]

Hi Romain,

Thank you for your answer, I will open a ticket soon.

Best

-- 
Jérôme Mainaud
jer...@mainaud.com

2016-08-19 12:16 GMT+02:00 Romain Hardouin :

> Hi Jérôme,
>
> The code in 2.2.6 allows -local and -pr:
> https://github.com/apache/cassandra/blob/cassandra-2.2.
> 6/src/java/org/apache/cassandra/service/StorageService.java#L2899
>
> But... the options validation introduced in CASSANDRA-6455 seems to break
> this feature!
> https://github.com/apache/cassandra/blob/cassandra-2.2.
> 6/src/java/org/apache/cassandra/repair/messages/RepairOption.java#L211
>
> I suggest to open a ticket https://issues.apache.org/
> jira/browse/cassandra/
>
> Best,
>
> Romain
>
>
> Le Vendredi 19 août 2016 11h47, Jérôme Mainaud  a
> écrit :
>
>
> Hello,
>
> I've got a repair command with both -pr and -local rejected on an 2.2.6
> cluster.
> The exact command was : nodetool repair --full -par -pr -local -j 4
>
> The message is  “You need to run primary range repair on all nodes in the
> cluster”.
>
> Reading the code and previously cited CASSANDRA-7450, it should have been
> accepted.
>
> Did anyone meet this error before ?
>
> Thanks
>
>
> --
> Jérôme Mainaud
> jer...@mainaud.com
>
> 2016-08-12 1:14 GMT+02:00 kurt Greaves :
>
> -D does not do what you think it does. I've quoted the relevant
> documentation from the README:
>
>
> Multiple
> Datacenters
> If you have multiple datacenters in your ring, then you MUST specify the
> name of the datacenter containing the node you are repairing as part of the
> command-line options (--datacenter=DCNAME). Failure to do so will result in
> only a subset of your data being repaired (approximately
> data/number-of-datacenters). This is because nodetool has no way to
> determine the relevant DC on its own, which in turn means it will use the
> tokens from every ring member in every datacenter.
>
>
>
> On 11 August 2016 at 12:24, Paulo Motta  wrote:
>
> > if we want to use -pr option ( which i suppose we should to prevent
> duplicate checks) in 2.0 then if we run the repair on all nodes in a single
> DC then it should be sufficient and we should not need to run it on all
> nodes across DC's?
>
> No, because the primary ranges of the nodes in other DCs will be missing
> repair, so you should either run with -pr in all nodes in all DCs, or
> restrict repair to a specific DC with -local (and have duplicate checks).
> Combined -pr and -local are only supported on 2.1
>
>
> 2016-08-11 1:29 GMT-03:00 Anishek Agarwal :
>
> ok thanks, so if we want to use -pr option ( which i suppose we should to
> prevent duplicate checks) in 2.0 then if we run the repair on all nodes in
> a single DC then it should be sufficient and we should not need to run it
> on all nodes across DC's ?
>
>
>
> On Wed, Aug 10, 2016 at 5:01 PM, Paulo Motta 
> wrote:
>
> On 2.0 repair -pr option is not supported together with -local, -hosts or
> -dc, since it assumes you need to repair all nodes in all DCs and it will
> throw and error if you try to run with nodetool, so perhaps there's
> something wrong with range_repair options parsing.
>
> On 2.1 it was added support to simultaneous -pr and -local options on
> CASSANDRA-7450, so if you need that you can either upgade to 2.1 or
> backport that to 2.0.
>
>
> 2016-08-10 5:20 GMT-03:00 Anishek Agarwal :
>
> Hello,
>
> We have 2.0.17 cassandra cluster(*DC1*) with a cross dc setup with a
> smaller cluster(*DC2*).  After reading various blogs about
> scheduling/running repairs looks like its good to run it with the following
>
>
> -pr for primary range only
> -st -et for sub ranges
> -par for parallel
> -dc to make sure we can schedule repairs independently on each Data centre
> we have.
>
> i have configured the above using the repair utility @ 
> https://github.com/BrianGallew
> /cassandra_range_repair.git
> 
>
> which leads to the following command :
>
> ./src/range_repair.py -k [keyspace] -c [columnfamily name] -v -H localhost
> -p -D* DC1*
>
> but looks like the merkle tree is being calculated on nodes which are part
> of other *DC2.*
>
> why does this happen? i thought it should only look at the nodes in local
> cluster. however on nodetool the* -pr* option cannot be used with *-local* 
> according
> to docs @https://docs.datastax.com/en/ cassandra/2.0/cassandra/tools/
> toolsRepair.html
> 
>
> so i am may be missing something, can someone help explain this please.
>
> thanks
> anishek
>
>
>
>
>
>
>
> --
> Kurt Greaves
> k...@instaclustr.com
> www.instaclustr.com
>
>
>
>
>

Re: nodetool repair with -pr and -dc

[Romain Hardouin]

Hi Jérôme,
The code in 2.2.6 allows -local and 
-pr:https://github.com/apache/cassandra/blob/cassandra-2.2.6/src/java/org/apache/cassandra/service/StorageService.java#L2899
But... the options validation introduced in CASSANDRA-6455 seems to break this 
feature!https://github.com/apache/cassandra/blob/cassandra-2.2.6/src/java/org/apache/cassandra/repair/messages/RepairOption.java#L211
I suggest to open a ticket https://issues.apache.org/jira/browse/cassandra/
Best,
Romain 

Le Vendredi 19 août 2016 11h47, Jérôme Mainaud  a écrit 
:
 

 Hello,

I've got a repair command with both -pr and -local rejected on an 2.2.6 cluster.
The exact command was : nodetool repair --full -par -pr -local -j 4
The message is  “You need to run primary range repair on all nodes in the 
cluster”.

Reading the code and previously cited CASSANDRA-7450, it should have been 
accepted.

Did anyone meet this error before ?

Thanks


-- 
Jérôme Mainaud
jer...@mainaud.com

2016-08-12 1:14 GMT+02:00 kurt Greaves :

-D does not do what you think it does. I've quoted the relevant documentation 
from the README:


Multiple Datacenters
If you have multiple datacenters in your ring, then you MUST specify the name 
of the datacenter containing the node you are repairing as part of the 
command-line options (--datacenter=DCNAME). Failure to do so will result in 
only a subset of your data being repaired (approximately 
data/number-of-datacenters). This is because nodetool has no way to determine 
the relevant DC on its own, which in turn means it will use the tokens from 
every ring member in every datacenter.


On 11 August 2016 at 12:24, Paulo Motta  wrote:

> if we want to use -pr option ( which i suppose we should to prevent duplicate 
> checks) in 2.0 then if we run the repair on all nodes in a single DC then it 
> should be sufficient and we should not need to run it on all nodes across 
> DC's?

No, because the primary ranges of the nodes in other DCs will be missing 
repair, so you should either run with -pr in all nodes in all DCs, or restrict 
repair to a specific DC with -local (and have duplicate checks). Combined -pr 
and -local are only supported on 2.1


2016-08-11 1:29 GMT-03:00 Anishek Agarwal :

ok thanks, so if we want to use -pr option ( which i suppose we should to 
prevent duplicate checks) in 2.0 then if we run the repair on all nodes in a 
single DC then it should be sufficient and we should not need to run it on all 
nodes across DC's ?


On Wed, Aug 10, 2016 at 5:01 PM, Paulo Motta  wrote:

On 2.0 repair -pr option is not supported together with -local, -hosts or -dc, 
since it assumes you need to repair all nodes in all DCs and it will throw and 
error if you try to run with nodetool, so perhaps there's something wrong with 
range_repair options parsing.

On 2.1 it was added support to simultaneous -pr and -local options on 
CASSANDRA-7450, so if you need that you can either upgade to 2.1 or backport 
that to 2.0.

2016-08-10 5:20 GMT-03:00 Anishek Agarwal :

Hello,
We have 2.0.17 cassandra cluster(DC1) with a cross dc setup with a smaller 
cluster(DC2).  After reading various blogs about scheduling/running repairs 
looks like its good to run it with the following 

-pr for primary range only -st -et for sub ranges -par for parallel -dc to make 
sure we can schedule repairs independently on each Data centre we have. 
i have configured the above using the repair utility @ 
https://github.com/BrianGallew /cassandra_range_repair.git
which leads to the following command :
./src/range_repair.py -k [keyspace] -c [columnfamily name] -v -H localhost -p 
-D DC1

but looks like the merkle tree is being calculated on nodes which are part of 
other DC2.
why does this happen? i thought it should only look at the nodes in local 
cluster. however on nodetool the -pr option cannot be used with -local 
according to docs @https://docs.datastax.com/en/ cassandra/2.0/cassandra/tools/ 
toolsRepair.html
so i am may be missing something, can someone help explain this please.
thanksanishek









-- 
Kurt greavesk...@instaclustr.comwww.instaclustr.com



  

Re: nodetool repair with -pr and -dc

[Jérôme Mainaud]

Hello,

I've got a repair command with both -pr and -local rejected on an 2.2.6
cluster.
The exact command was : nodetool repair --full -par -pr -local -j 4

The message is  “You need to run primary range repair on all nodes in the
cluster”.

Reading the code and previously cited CASSANDRA-7450, it should have been
accepted.

Did anyone meet this error before ?

Thanks


-- 
Jérôme Mainaud
jer...@mainaud.com

2016-08-12 1:14 GMT+02:00 kurt Greaves :

> -D does not do what you think it does. I've quoted the relevant
> documentation from the README:
>
>>
>> Multiple
>> Datacenters
>>
>> If you have multiple datacenters in your ring, then you MUST specify the
>> name of the datacenter containing the node you are repairing as part of the
>> command-line options (--datacenter=DCNAME). Failure to do so will result in
>> only a subset of your data being repaired (approximately
>> data/number-of-datacenters). This is because nodetool has no way to
>> determine the relevant DC on its own, which in turn means it will use the
>> tokens from every ring member in every datacenter.
>>
>
>
> On 11 August 2016 at 12:24, Paulo Motta  wrote:
>
>> > if we want to use -pr option ( which i suppose we should to prevent
>> duplicate checks) in 2.0 then if we run the repair on all nodes in a single
>> DC then it should be sufficient and we should not need to run it on all
>> nodes across DC's?
>>
>> No, because the primary ranges of the nodes in other DCs will be missing
>> repair, so you should either run with -pr in all nodes in all DCs, or
>> restrict repair to a specific DC with -local (and have duplicate checks).
>> Combined -pr and -local are only supported on 2.1
>>
>>
>> 2016-08-11 1:29 GMT-03:00 Anishek Agarwal :
>>
>>> ok thanks, so if we want to use -pr option ( which i suppose we should
>>> to prevent duplicate checks) in 2.0 then if we run the repair on all nodes
>>> in a single DC then it should be sufficient and we should not need to run
>>> it on all nodes across DC's ?
>>>
>>>
>>>
>>> On Wed, Aug 10, 2016 at 5:01 PM, Paulo Motta 
>>> wrote:
>>>
 On 2.0 repair -pr option is not supported together with -local, -hosts
 or -dc, since it assumes you need to repair all nodes in all DCs and it
 will throw and error if you try to run with nodetool, so perhaps there's
 something wrong with range_repair options parsing.

 On 2.1 it was added support to simultaneous -pr and -local options on
 CASSANDRA-7450, so if you need that you can either upgade to 2.1 or
 backport that to 2.0.


 2016-08-10 5:20 GMT-03:00 Anishek Agarwal :

> Hello,
>
> We have 2.0.17 cassandra cluster(*DC1*) with a cross dc setup with a
> smaller cluster(*DC2*).  After reading various blogs about
> scheduling/running repairs looks like its good to run it with the 
> following
>
>
> -pr for primary range only
> -st -et for sub ranges
> -par for parallel
> -dc to make sure we can schedule repairs independently on each Data
> centre we have.
>
> i have configured the above using the repair utility @
> https://github.com/BrianGallew/cassandra_range_repair.git
>
> which leads to the following command :
>
> ./src/range_repair.py -k [keyspace] -c [columnfamily name] -v -H
> localhost -p -D* DC1*
>
> but looks like the merkle tree is being calculated on nodes which are
> part of other *DC2.*
>
> why does this happen? i thought it should only look at the nodes in
> local cluster. however on nodetool the* -pr* option cannot be used
> with *-local* according to docs @https://docs.datastax.com/en/
> cassandra/2.0/cassandra/tools/toolsRepair.html
>
> so i am may be missing something, can someone help explain this please.
>
> thanks
> anishek
>


>>>
>>
>
>
> --
> Kurt Greaves
> k...@instaclustr.com
> www.instaclustr.com
>

Re: nodetool repair with -pr and -dc

[kurt Greaves]

-D does not do what you think it does. I've quoted the relevant
documentation from the README:

>
> Multiple
> Datacenters
>
> If you have multiple datacenters in your ring, then you MUST specify the
> name of the datacenter containing the node you are repairing as part of the
> command-line options (--datacenter=DCNAME). Failure to do so will result in
> only a subset of your data being repaired (approximately
> data/number-of-datacenters). This is because nodetool has no way to
> determine the relevant DC on its own, which in turn means it will use the
> tokens from every ring member in every datacenter.
>


On 11 August 2016 at 12:24, Paulo Motta  wrote:

> > if we want to use -pr option ( which i suppose we should to prevent
> duplicate checks) in 2.0 then if we run the repair on all nodes in a single
> DC then it should be sufficient and we should not need to run it on all
> nodes across DC's?
>
> No, because the primary ranges of the nodes in other DCs will be missing
> repair, so you should either run with -pr in all nodes in all DCs, or
> restrict repair to a specific DC with -local (and have duplicate checks).
> Combined -pr and -local are only supported on 2.1
>
>
> 2016-08-11 1:29 GMT-03:00 Anishek Agarwal :
>
>> ok thanks, so if we want to use -pr option ( which i suppose we should to
>> prevent duplicate checks) in 2.0 then if we run the repair on all nodes in
>> a single DC then it should be sufficient and we should not need to run it
>> on all nodes across DC's ?
>>
>>
>>
>> On Wed, Aug 10, 2016 at 5:01 PM, Paulo Motta 
>> wrote:
>>
>>> On 2.0 repair -pr option is not supported together with -local, -hosts
>>> or -dc, since it assumes you need to repair all nodes in all DCs and it
>>> will throw and error if you try to run with nodetool, so perhaps there's
>>> something wrong with range_repair options parsing.
>>>
>>> On 2.1 it was added support to simultaneous -pr and -local options on
>>> CASSANDRA-7450, so if you need that you can either upgade to 2.1 or
>>> backport that to 2.0.
>>>
>>>
>>> 2016-08-10 5:20 GMT-03:00 Anishek Agarwal :
>>>
 Hello,

 We have 2.0.17 cassandra cluster(*DC1*) with a cross dc setup with a
 smaller cluster(*DC2*).  After reading various blogs about
 scheduling/running repairs looks like its good to run it with the following


 -pr for primary range only
 -st -et for sub ranges
 -par for parallel
 -dc to make sure we can schedule repairs independently on each Data
 centre we have.

 i have configured the above using the repair utility @
 https://github.com/BrianGallew/cassandra_range_repair.git

 which leads to the following command :

 ./src/range_repair.py -k [keyspace] -c [columnfamily name] -v -H
 localhost -p -D* DC1*

 but looks like the merkle tree is being calculated on nodes which are
 part of other *DC2.*

 why does this happen? i thought it should only look at the nodes in
 local cluster. however on nodetool the* -pr* option cannot be used
 with *-local* according to docs @https://docs.datastax.com/en/
 cassandra/2.0/cassandra/tools/toolsRepair.html

 so i am may be missing something, can someone help explain this please.

 thanks
 anishek

>>>
>>>
>>
>


-- 
Kurt Greaves
k...@instaclustr.com
www.instaclustr.com

Re: nodetool repair with -pr and -dc

[Paulo Motta]

> if we want to use -pr option ( which i suppose we should to prevent
duplicate checks) in 2.0 then if we run the repair on all nodes in a single
DC then it should be sufficient and we should not need to run it on all
nodes across DC's?

No, because the primary ranges of the nodes in other DCs will be missing
repair, so you should either run with -pr in all nodes in all DCs, or
restrict repair to a specific DC with -local (and have duplicate checks).
Combined -pr and -local are only supported on 2.1


2016-08-11 1:29 GMT-03:00 Anishek Agarwal :

> ok thanks, so if we want to use -pr option ( which i suppose we should to
> prevent duplicate checks) in 2.0 then if we run the repair on all nodes in
> a single DC then it should be sufficient and we should not need to run it
> on all nodes across DC's ?
>
>
>
> On Wed, Aug 10, 2016 at 5:01 PM, Paulo Motta 
> wrote:
>
>> On 2.0 repair -pr option is not supported together with -local, -hosts or
>> -dc, since it assumes you need to repair all nodes in all DCs and it will
>> throw and error if you try to run with nodetool, so perhaps there's
>> something wrong with range_repair options parsing.
>>
>> On 2.1 it was added support to simultaneous -pr and -local options on
>> CASSANDRA-7450, so if you need that you can either upgade to 2.1 or
>> backport that to 2.0.
>>
>>
>> 2016-08-10 5:20 GMT-03:00 Anishek Agarwal :
>>
>>> Hello,
>>>
>>> We have 2.0.17 cassandra cluster(*DC1*) with a cross dc setup with a
>>> smaller cluster(*DC2*).  After reading various blogs about
>>> scheduling/running repairs looks like its good to run it with the following
>>>
>>>
>>> -pr for primary range only
>>> -st -et for sub ranges
>>> -par for parallel
>>> -dc to make sure we can schedule repairs independently on each Data
>>> centre we have.
>>>
>>> i have configured the above using the repair utility @
>>> https://github.com/BrianGallew/cassandra_range_repair.git
>>>
>>> which leads to the following command :
>>>
>>> ./src/range_repair.py -k [keyspace] -c [columnfamily name] -v -H
>>> localhost -p -D* DC1*
>>>
>>> but looks like the merkle tree is being calculated on nodes which are
>>> part of other *DC2.*
>>>
>>> why does this happen? i thought it should only look at the nodes in
>>> local cluster. however on nodetool the* -pr* option cannot be used with
>>> *-local* according to docs @https://docs.datastax.com/en/
>>> cassandra/2.0/cassandra/tools/toolsRepair.html
>>>
>>> so i am may be missing something, can someone help explain this please.
>>>
>>> thanks
>>> anishek
>>>
>>
>>
>

Re: nodetool repair with -pr and -dc

[Anishek Agarwal]

ok thanks, so if we want to use -pr option ( which i suppose we should to
prevent duplicate checks) in 2.0 then if we run the repair on all nodes in
a single DC then it should be sufficient and we should not need to run it
on all nodes across DC's ?



On Wed, Aug 10, 2016 at 5:01 PM, Paulo Motta 
wrote:

> On 2.0 repair -pr option is not supported together with -local, -hosts or
> -dc, since it assumes you need to repair all nodes in all DCs and it will
> throw and error if you try to run with nodetool, so perhaps there's
> something wrong with range_repair options parsing.
>
> On 2.1 it was added support to simultaneous -pr and -local options on
> CASSANDRA-7450, so if you need that you can either upgade to 2.1 or
> backport that to 2.0.
>
>
> 2016-08-10 5:20 GMT-03:00 Anishek Agarwal :
>
>> Hello,
>>
>> We have 2.0.17 cassandra cluster(*DC1*) with a cross dc setup with a
>> smaller cluster(*DC2*).  After reading various blogs about
>> scheduling/running repairs looks like its good to run it with the following
>>
>>
>> -pr for primary range only
>> -st -et for sub ranges
>> -par for parallel
>> -dc to make sure we can schedule repairs independently on each Data
>> centre we have.
>>
>> i have configured the above using the repair utility @
>> https://github.com/BrianGallew/cassandra_range_repair.git
>>
>> which leads to the following command :
>>
>> ./src/range_repair.py -k [keyspace] -c [columnfamily name] -v -H
>> localhost -p -D* DC1*
>>
>> but looks like the merkle tree is being calculated on nodes which are
>> part of other *DC2.*
>>
>> why does this happen? i thought it should only look at the nodes in local
>> cluster. however on nodetool the* -pr* option cannot be used with
>> *-local* according to docs @https://docs.datastax.com/en/
>> cassandra/2.0/cassandra/tools/toolsRepair.html
>>
>> so i am may be missing something, can someone help explain this please.
>>
>> thanks
>> anishek
>>
>
>

Re: nodetool repair with -pr and -dc

[Paulo Motta]

On 2.0 repair -pr option is not supported together with -local, -hosts or
-dc, since it assumes you need to repair all nodes in all DCs and it will
throw and error if you try to run with nodetool, so perhaps there's
something wrong with range_repair options parsing.

On 2.1 it was added support to simultaneous -pr and -local options on
CASSANDRA-7450, so if you need that you can either upgade to 2.1 or
backport that to 2.0.

2016-08-10 5:20 GMT-03:00 Anishek Agarwal :

> Hello,
>
> We have 2.0.17 cassandra cluster(*DC1*) with a cross dc setup with a
> smaller cluster(*DC2*).  After reading various blogs about
> scheduling/running repairs looks like its good to run it with the following
>
>
> -pr for primary range only
> -st -et for sub ranges
> -par for parallel
> -dc to make sure we can schedule repairs independently on each Data centre
> we have.
>
> i have configured the above using the repair utility @
> https://github.com/BrianGallew/cassandra_range_repair.git
>
> which leads to the following command :
>
> ./src/range_repair.py -k [keyspace] -c [columnfamily name] -v -H localhost
> -p -D* DC1*
>
> but looks like the merkle tree is being calculated on nodes which are part
> of other *DC2.*
>
> why does this happen? i thought it should only look at the nodes in local
> cluster. however on nodetool the* -pr* option cannot be used with *-local* 
> according
> to docs @https://docs.datastax.com/en/cassandra/2.0/cassandra/tools/
> toolsRepair.html
>
> so i am may be missing something, can someone help explain this please.
>
> thanks
> anishek
>

Re: Nodetool repair inconsistencies

[Jason Kania]

Hi Paul,
I have tried running 'nodetool compact' and the situation remains the same 
after I deleted the files that caused 'nodetool compact' to generate an 
exception in the first place.
My concern is that if I delete some sstable sets from a directory or even if I 
completely eliminate the sstables in a directory on one machine, run 'nodetool 
repair' followed by 'nodetool compact', that directory remains empty. My 
understanding has been that these equivalently named directories should contain 
roughly the same amount of content.
Thanks,
Jason

  From: Paul Fife <paulf...@gmail.com>
 To: user@cassandra.apache.org; Jason Kania <jason.ka...@ymail.com> 
 Sent: Wednesday, June 8, 2016 12:55 PM
 Subject: Re: Nodetool repair inconsistencies
   
Hi Jason -
Did you run a major compaction after the repair completed? Do you have other 
reasons besides the number/size of sstables to believe all nodes don't have a 
copy of the current data at the end of the repair operation?
Thanks,Paul
On Wed, Jun 8, 2016 at 8:12 AM, Jason Kania <jason.ka...@ymail.com> wrote:

Hi Romain,
The problem is that there is no error to share. I am focusing on the 
inconsistency that when I run nodetool repair, get no errors and yet the 
content in the same directory on the different nodes is vastly different. This 
lack of an error is nature of my question, not the nodetool compact error.
Thanks,
Jason
  From: Romain Hardouin <romainh...@yahoo.fr>
 To: "user@cassandra.apache.org" <user@cassandra.apache.org>; Jason Kania 
<jason.ka...@ymail.com> 
 Sent: Wednesday, June 8, 2016 8:30 AM
 Subject: Re: Nodetool repair inconsistencies
  
Hi Jason,
It's difficult for the community to help you if you don't share the error 
;-)What the logs said when you ran a major compaction? (i.e. the first error 
you encountered) 
Best,
Romain

Le Mercredi 8 juin 2016 3h34, Jason Kania <jason.ka...@ymail.com> a écrit :
 

 I am running a 3 node cluster of 3.0.6 instances and encountered an error when 
running nodetool compact. I then ran nodetool repair. No errors were returned.
I then attempted to run nodetool compact again, but received the same error so 
the repair made no correction and reported no errors.
After that, I moved the problematic files out of the directory, restarted 
cassandra and attempted the repair again. The repair again completed without 
errors, however, no files were added to the directory that had contained the 
corrupt files. So nodetool repair does not seem to be making actual repairs.
I started looking around and numerous directories have vastly different amounts 
of content across the 3 nodes. There are 3 replicas so I would expect to find 
similar amounts of content in the same data directory on the different nodes.

Is there any way to dig deeper into this? I don't want to be caught because 
replication/repair is silently failing. I noticed that there is always an "some 
repair failed" amongst the repair output but that is so completely unhelpful 
and has always been present.

Thanks,
Jason


   

   



  

Re: Nodetool repair inconsistencies

[Paul Fife]

Hi Jason -

Did you run a major compaction after the repair completed? Do you have
other reasons besides the number/size of sstables to believe all nodes
don't have a copy of the current data at the end of the repair operation?

Thanks,
Paul

On Wed, Jun 8, 2016 at 8:12 AM, Jason Kania <jason.ka...@ymail.com> wrote:

> Hi Romain,
>
> The problem is that there is no error to share. I am focusing on the
> inconsistency that when I run nodetool repair, get no errors and yet the
> content in the same directory on the different nodes is vastly different.
> This lack of an error is nature of my question, not the nodetool compact
> error.
>
> Thanks,
>
> Jason
>
> --
> *From:* Romain Hardouin <romainh...@yahoo.fr>
> *To:* "user@cassandra.apache.org" <user@cassandra.apache.org>; Jason
> Kania <jason.ka...@ymail.com>
> *Sent:* Wednesday, June 8, 2016 8:30 AM
> *Subject:* Re: Nodetool repair inconsistencies
>
> Hi Jason,
>
> It's difficult for the community to help you if you don't share the error
> ;-)
> What the logs said when you ran a major compaction? (i.e. the first error
> you encountered)
>
> Best,
>
> Romain
>
> Le Mercredi 8 juin 2016 3h34, Jason Kania <jason.ka...@ymail.com> a écrit
> :
>
>
> I am running a 3 node cluster of 3.0.6 instances and encountered an error
> when running nodetool compact. I then ran nodetool repair. No errors were
> returned.
>
> I then attempted to run nodetool compact again, but received the same
> error so the repair made no correction and reported no errors.
>
> After that, I moved the problematic files out of the directory, restarted
> cassandra and attempted the repair again. The repair again completed
> without errors, however, no files were added to the directory that had
> contained the corrupt files. So nodetool repair does not seem to be making
> actual repairs.
>
> I started looking around and numerous directories have vastly different
> amounts of content across the 3 nodes. There are 3 replicas so I would
> expect to find similar amounts of content in the same data directory on the
> different nodes.
>
> Is there any way to dig deeper into this? I don't want to be caught
> because replication/repair is silently failing. I noticed that there is
> always an "some repair failed" amongst the repair output but that is so
> completely unhelpful and has always been present.
>
> Thanks,
>
> Jason
>
>
>
>
>

Re: Nodetool repair inconsistencies

[Jason Kania]

Hi Romain,
The problem is that there is no error to share. I am focusing on the 
inconsistency that when I run nodetool repair, get no errors and yet the 
content in the same directory on the different nodes is vastly different. This 
lack of an error is nature of my question, not the nodetool compact error.
Thanks,
Jason
  From: Romain Hardouin <romainh...@yahoo.fr>
 To: "user@cassandra.apache.org" <user@cassandra.apache.org>; Jason Kania 
<jason.ka...@ymail.com> 
 Sent: Wednesday, June 8, 2016 8:30 AM
 Subject: Re: Nodetool repair inconsistencies
   
Hi Jason,
It's difficult for the community to help you if you don't share the error 
;-)What the logs said when you ran a major compaction? (i.e. the first error 
you encountered) 
Best,
Romain

Le Mercredi 8 juin 2016 3h34, Jason Kania <jason.ka...@ymail.com> a écrit :
 

 I am running a 3 node cluster of 3.0.6 instances and encountered an error when 
running nodetool compact. I then ran nodetool repair. No errors were returned.
I then attempted to run nodetool compact again, but received the same error so 
the repair made no correction and reported no errors.
After that, I moved the problematic files out of the directory, restarted 
cassandra and attempted the repair again. The repair again completed without 
errors, however, no files were added to the directory that had contained the 
corrupt files. So nodetool repair does not seem to be making actual repairs.
I started looking around and numerous directories have vastly different amounts 
of content across the 3 nodes. There are 3 replicas so I would expect to find 
similar amounts of content in the same data directory on the different nodes.

Is there any way to dig deeper into this? I don't want to be caught because 
replication/repair is silently failing. I noticed that there is always an "some 
repair failed" amongst the repair output but that is so completely unhelpful 
and has always been present.

Thanks,
Jason


   

  

Re: Nodetool repair inconsistencies

[Romain Hardouin]

Hi Jason,
It's difficult for the community to help you if you don't share the error 
;-)What the logs said when you ran a major compaction? (i.e. the first error 
you encountered) 
Best,
Romain

Le Mercredi 8 juin 2016 3h34, Jason Kania  a écrit :
 

 I am running a 3 node cluster of 3.0.6 instances and encountered an error when 
running nodetool compact. I then ran nodetool repair. No errors were returned.
I then attempted to run nodetool compact again, but received the same error so 
the repair made no correction and reported no errors.
After that, I moved the problematic files out of the directory, restarted 
cassandra and attempted the repair again. The repair again completed without 
errors, however, no files were added to the directory that had contained the 
corrupt files. So nodetool repair does not seem to be making actual repairs.
I started looking around and numerous directories have vastly different amounts 
of content across the 3 nodes. There are 3 replicas so I would expect to find 
similar amounts of content in the same data directory on the different nodes.

Is there any way to dig deeper into this? I don't want to be caught because 
replication/repair is silently failing. I noticed that there is always an "some 
repair failed" amongst the repair output but that is so completely unhelpful 
and has always been present.

Thanks,
Jason


  

Re: Nodetool repair question

[Joel Knighton]

No - repair does not change token ownership. The up/down state of a node is
not related to token ownership.

On Tue, May 10, 2016 at 3:26 PM, Anubhav Kale 
wrote:

> Hello,
>
>
>
> Suppose I have 3 nodes, and stop Cassandra on one of them. Then I run a
> repair. Will repair move the token ranges from down node to other node ? In
> other words in any situation, does repair operation *ever* change token
> ownership ?
>
>
>
> Thanks !
>



-- 



Joel Knighton
Cassandra Developer | joel.knigh...@datastax.com


 

 

Re: Nodetool repair with Load times 5

[Jean Tremblay]

Dear Alain,

Thanks again for your precious help.

I might help, but I need to know what you have done recently (change the RF, 
Add remove node, cleanups, anything else as much as possible...)

I have a cluster of 5 nodes all running Cassandra 2.1.8.
I have a fixed schema which never changes. I have not changed RF, it is 3. I 
have not remove nodes, no cleanups.

Basically here are the important operations I have done:

- Install Cassandra 2.1.7 on a cluster of 5 nodes with RF 3 using Sized-Tiered 
compaction.
- Insert 2 billion rows. (bulk load)
- Made loads of selects statements… Verified that the data is good.
- Did some deletes and a bit more inserts.
- Eventually migrated to 2.1.8
- Then only very few delete/inserts.
- Did a few snapshots.

When I was doing “nodetool status” I always got a load of about 200 GB on 
**all** nodes.

- Then I did a “nodetool -h node0 repair -par -pr -inc” and after that I had a 
completely different picture.

nodetool -h zennode0 status
Datacenter: datacenter1
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  AddressLoad   Tokens  OwnsHost ID   
Rack
UN  192.168.2.104  941.49 GB  256 ?   
c13e0858-091c-47c4-8773-6d6262723435  rack1
UN  192.168.2.100  1.07 TB256 ?   
c32a9357-e37e-452e-8eb1-57d86314b419  rack1
UN  192.168.2.101  189.72 GB  256 ?   
9af90dea-90b3-4a8a-b88a-0aeabe3cea79  rack1
UN  192.168.2.102  948.61 GB  256 ?   
8eb7a5bb-6903-4ae1-a372-5436d0cc170c  rack1
UN  192.168.2.103  197.27 GB  256 ?   
9efc6f13-2b02-4400-8cde-ae831feb86e9  rack1


Also, could you please do the nodetool status myks for your keyspace(s) ? We 
will then be able to know the theoretical ownership of each node on your 
distinct (or unique) keyspace(s) ?

nodetool -h zennode0 status XYZdata
Datacenter: datacenter1
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  AddressLoad   Tokens  Owns (effective)  Host ID 
  Rack
UN  192.168.2.104  941.49 GB  256 62.5% 
c13e0858-091c-47c4-8773-6d6262723435  rack1
UN  192.168.2.100  1.07 TB256 58.4% 
c32a9357-e37e-452e-8eb1-57d86314b419  rack1
UN  192.168.2.101  189.72 GB  256 58.4% 
9af90dea-90b3-4a8a-b88a-0aeabe3cea79  rack1
UN  192.168.2.102  948.61 GB  256 60.1% 
8eb7a5bb-6903-4ae1-a372-5436d0cc170c  rack1
UN  192.168.2.103  197.27 GB  256 60.6% 
9efc6f13-2b02-4400-8cde-ae831feb86e9  rack1


Some ideas:

You repaired only a primary range (-pr) of one node, with a RF of 3 and have 
3 big nodes, if not using vnodes, this would be almost normal (excepted for the 
gap 200 GB -- 1 TB, this is huge, unless you messed up with RF). So are you 
using them ?

My schema is totally fixed and I use RF 3 since the beginning. Sorry I’m not 
too aquinted with vnodes. I have not changed anything in the cassandra.yaml 
except the seeds and the name of the cluster.

2/ Load is barely the size of the data on each node

If it is the size of the data how can it fit on the disk?
My 5 nodes have an SSD drive of 1 TB and here is the disk usage for each of 
them:

node0: 25%
node1: 25%
node2: 24%
node3: 26%
node4: 29%

nodetool status says that the load for node0 is 1.07TB. That is more than fit 
of it’s disk, and the disk usage for node0 is 25%.

This is not clear for me… the Load in nodetool status output seems to be more 
that “the size of the data on a node”.


On 18 Aug 2015, at 19:29 , Alain RODRIGUEZ 
arodr...@gmail.commailto:arodr...@gmail.com wrote:

Hi Jean,

I might help, but I need to know what you have done recently (change the RF, 
Add remove node, cleanups, anything else as much as possible...)

Also, could you please do the nodetool status myks for your keyspace(s) ? We 
will then be able to know the theoretical ownership of each node on your 
distinct (or unique) keyspace(s) ?

Some ideas:

You repaired only a primary range (-pr) of one node, with a RF of 3 and have 
3 big nodes, if not using vnodes, this would be almost normal (excepted for the 
gap 200 GB -- 1 TB, this is huge, unless you messed up with RF). So are you 
using them ?

Answers:

1/ It depends on what happen to this cluster (see my questions above)
2/ Load is barely the size of the data on each node
3/ No, this is not a normal nor stable situation.
4/ No, pr means you repaired only the partition that node is responsible for 
(depends on token), you have to run this on all nodes. But I would wait to find 
out first what's happening to avoid hitting the threshold on disk space or 
whatever.

I guess I have been confused with the -par switch, which means to me that the 
work will be done in parallel and therefore will be done on all nodes.

So if I understand right, one should do a “nodetool repair -par -pr -inc” on 
all nodes one after the other? Is this correct?


I have a second cluster, a smaller one, 

Re: Nodetool repair with Load times 5

[Mark Greene]

Hey Jean,

Did you try running a nodetool cleanup on all your nodes, perhaps one at a
time?

On Tue, Aug 18, 2015 at 3:59 AM, Jean Tremblay 
jean.tremb...@zen-innovations.com wrote:

 Hi,

 I have a phenomena I cannot explain, and I would like to understand.

 I’m running Cassandra 2.1.8 on a cluster of 5 nodes.
 I’m using replication factor 3, with most default settings.

 Last week I done a nodetool status which gave me on each node a load of
 about 200 GB.
 Since then there was no deletes no inserts.

 This weekend I did a nodetool -h 192.168.2.100 repair -pr -par -inc

 And now when I make a nodetool status I see completely a new picture!!

 nodetool -h zennode0 status
 Datacenter: datacenter1
 ===
 Status=Up/Down
 |/ State=Normal/Leaving/Joining/Moving
 --  AddressLoad   Tokens  OwnsHost ID
   Rack
 UN  192.168.2.104  940.73 GB  256 ?
 c13e0858-091c-47c4-8773-6d6262723435  rack1
 UN  192.168.2.100  1.07 TB256 ?
 c32a9357-e37e-452e-8eb1-57d86314b419  rack1
 UN  192.168.2.101  189.03 GB  256 ?
 9af90dea-90b3-4a8a-b88a-0aeabe3cea79  rack1
 UN  192.168.2.102  951.28 GB  256 ?
 8eb7a5bb-6903-4ae1-a372-5436d0cc170c  rack1
 UN  192.168.2.103  196.54 GB  256 ?
 9efc6f13-2b02-4400-8cde-ae831feb86e9  rack1

 The nodes 192.168.2.101 and 103 are about what they were last week, but
 now the three other nodes have a load which is about 5 times bigger!

 1) Is this normal?
 2) What is the meaning of the column Load?
 3) Is there anything to fix? Can I leave it like that?

 Strange I’m asking to fix after I did a *repair*.

 Thanks a lot for your help.

 Kind regards

 Jean

Re: Nodetool repair with Load times 5

[Jean Tremblay]

No. I did not try.
I would like to understand what is going on before I make my problem, maybe 
even worse.

I really would like to understand:

1) Is this normal?
2) What is the meaning of the column Load?
3) Is there anything to fix? Can I leave it like that?
  4) Did I do something wrong? When you use -par you only need to run 
repair from one node right? E.g.  nodetool -h 192.168.2.100 repair -pr -par -inc

Thanks for your feedback.

Jean

On 18 Aug 2015, at 14:33 , Mark Greene 
green...@gmail.commailto:green...@gmail.com wrote:

Hey Jean,

Did you try running a nodetool cleanup on all your nodes, perhaps one at a time?

On Tue, Aug 18, 2015 at 3:59 AM, Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
wrote:
Hi,

I have a phenomena I cannot explain, and I would like to understand.

I’m running Cassandra 2.1.8 on a cluster of 5 nodes.
I’m using replication factor 3, with most default settings.

Last week I done a nodetool status which gave me on each node a load of about 
200 GB.
Since then there was no deletes no inserts.

This weekend I did a nodetool -h 192.168.2.100 repair -pr -par -inc

And now when I make a nodetool status I see completely a new picture!!

nodetool -h zennode0 status
Datacenter: datacenter1
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  AddressLoad   Tokens  OwnsHost ID   
Rack
UN  192.168.2.104  940.73 GB  256 ?   
c13e0858-091c-47c4-8773-6d6262723435  rack1
UN  192.168.2.100  1.07 TB256 ?   
c32a9357-e37e-452e-8eb1-57d86314b419  rack1
UN  192.168.2.101  189.03 GB  256 ?   
9af90dea-90b3-4a8a-b88a-0aeabe3cea79  rack1
UN  192.168.2.102  951.28 GB  256 ?   
8eb7a5bb-6903-4ae1-a372-5436d0cc170c  rack1
UN  192.168.2.103  196.54 GB  256 ?   
9efc6f13-2b02-4400-8cde-ae831feb86e9  rack1

The nodes 192.168.2.101 and 103 are about what they were last week, but now the 
three other nodes have a load which is about 5 times bigger!

1) Is this normal?
2) What is the meaning of the column Load?
3) Is there anything to fix? Can I leave it like that?

Strange I’m asking to fix after I did a *repair*.

Thanks a lot for your help.

Kind regards

Jean

Re: Nodetool repair with Load times 5

[Alain RODRIGUEZ]

Hi Jean,

I might help, but I need to know what you have done recently (change the
RF, Add remove node, cleanups, anything else as much as possible...)

Also, could you please do the nodetool status *myks* for your keyspace(s)
? We will then be able to know the theoretical ownership of each node on
your distinct (or unique) keyspace(s) ?

Some ideas:

You repaired only a primary range (-pr) of one node, with a RF of 3 and
have 3 big nodes, if not using vnodes, this would be almost normal
(excepted for the gap 200 GB -- 1 TB, this is huge, unless you messed up
with RF). So are you using them ?

Answers:

1/ It depends on what happen to this cluster (see my questions above)
2/ Load is barely the size of the data on each node
3/ No, this is not a normal nor stable situation.
4/ No, pr means you repaired only the partition that node is responsible
for (depends on token), you have to run this on all nodes. But I would wait
to find out first what's happening to avoid hitting the threshold on disk
space or whatever.

Anyway, see if you can give us more info related to this.

C*heers,

Alain



2015-08-18 14:40 GMT+02:00 Jean Tremblay jean.tremb...@zen-innovations.com
:

 No. I did not try.
 I would like to understand what is going on before I make my problem,
 maybe even worse.

 I really would like to understand:

 1) Is this normal?
 2) What is the meaning of the column Load?
 3) Is there anything to fix? Can I leave it like that?

   4) Did I do something wrong? When you use -par you only need to run
 repair from one node right? E.g.  nodetool -h 192.168.2.100 repair -pr -par
 -inc

 Thanks for your feedback.

 Jean

 On 18 Aug 2015, at 14:33 , Mark Greene green...@gmail.com wrote:

 Hey Jean,

 Did you try running a nodetool cleanup on all your nodes, perhaps one at a
 time?

 On Tue, Aug 18, 2015 at 3:59 AM, Jean Tremblay 
 jean.tremb...@zen-innovations.com wrote:

 Hi,

 I have a phenomena I cannot explain, and I would like to understand.

 I’m running Cassandra 2.1.8 on a cluster of 5 nodes.
 I’m using replication factor 3, with most default settings.

 Last week I done a nodetool status which gave me on each node a load of
 about 200 GB.
 Since then there was no deletes no inserts.

 This weekend I did a nodetool -h 192.168.2.100 repair -pr -par -inc

 And now when I make a nodetool status I see completely a new picture!!

 nodetool -h zennode0 status
 Datacenter: datacenter1
 ===
 Status=Up/Down
 |/ State=Normal/Leaving/Joining/Moving
 --  AddressLoad   Tokens  OwnsHost ID
   Rack
 UN  192.168.2.104  940.73 GB  256 ?
 c13e0858-091c-47c4-8773-6d6262723435  rack1
 UN  192.168.2.100  1.07 TB256 ?
 c32a9357-e37e-452e-8eb1-57d86314b419  rack1
 UN  192.168.2.101  189.03 GB  256 ?
 9af90dea-90b3-4a8a-b88a-0aeabe3cea79  rack1
 UN  192.168.2.102  951.28 GB  256 ?
 8eb7a5bb-6903-4ae1-a372-5436d0cc170c  rack1
 UN  192.168.2.103  196.54 GB  256 ?
 9efc6f13-2b02-4400-8cde-ae831feb86e9  rack1

 The nodes 192.168.2.101 and 103 are about what they were last week, but
 now the three other nodes have a load which is about 5 times bigger!

 1) Is this normal?
 2) What is the meaning of the column Load?
 3) Is there anything to fix? Can I leave it like that?

 Strange I’m asking to fix after I did a *repair*.

 Thanks a lot for your help.

 Kind regards

 Jean

RE: nodetool repair

[Jens Rantil]

Hi,


For the record I've succesfully used 
https://github.com/BrianGallew/cassandra_range_repair to make smooth repairing. 
Could maybe also be of interest don't know...




Cheers,

Jens





–
Skickat från Mailbox

On Fri, Jun 19, 2015 at 8:36 PM, null sean_r_dur...@homedepot.com wrote:

 It seems to me that running repair on any given node may also induce repairs 
 to related replica nodes. For example, if I run repair on node A and node B 
 has some replicas, data might stream from A to B (assuming A has newer/more 
 data). Now, that does NOT mean that node B will be fully repaired. You still 
 need to run repair -pr on all nodes before gc_grace_seconds.
 You can run repairs on multiple nodes at the same time. However, you might 
 end up with a large amount of streaming, if many repairs are needed. So, you 
 should be aware of a performance impact.
 I run weekly repairs on one node at a time, if possible. On, larger rings, 
 though, I run repairs on multiple nodes staggered by a few hours. Once your 
 routine maintenance is established, repairs will not run for very long. But, 
 if you have a large ring that hasn’t been repaired, those first repairs may 
 take days (but should get faster as you get further through the ring).
 Sean Durity
 From: Alain RODRIGUEZ [mailto:arodr...@gmail.com]
 Sent: Friday, June 19, 2015 3:56 AM
 To: user@cassandra.apache.org
 Subject: Re: nodetool repair
 Hi,
 This is not necessarily true. Repair will induce compactions only if you have 
 entropy in your cluster. If not it will just read your data to compare all 
 the replica of each piece of data (using indeed cpu and disk IO).
 If there is some data missing it will repair it. Though, due to merkle tree 
 size, you will generally stream more data than just the data needed. To limit 
 this downside and the compactions amount, use range repairs -- 
 http://www.datastax.com/dev/blog/advanced-repair-techniques.
 About tombstones, they will be evicted only after gc_grace_period and only if 
 all the parts of the row are part of the compaction.
 C*heers,
 Alain
 2015-06-19 9:08 GMT+02:00 arun sirimalla 
 arunsi...@gmail.commailto:arunsi...@gmail.com:
 Yes compactions will remove tombstones
 On Thu, Jun 18, 2015 at 11:46 PM, Jean Tremblay 
 jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
 wrote:
 Perfect thank you.
 So making a weekly nodetool repair -pr”  on all nodes one after the other 
 will repair my cluster. That is great.
 If it does a compaction, does it mean that it would also clean up my 
 tombstone from my LeveledCompactionStrategy tables at the same time?
 Thanks for your help.
 On 19 Jun 2015, at 07:56 , arun sirimalla 
 arunsi...@gmail.commailto:arunsi...@gmail.com wrote:
 Hi Jean,
 Running nodetool repair on a node will repair only that node in the cluster. 
 It is recommended to run nodetool repair on one node at a time.
 Few things to keep in mind while running repair
1. Running repair will trigger compactions
2. Increase in CPU utilization.
 Run node tool repair with -pr option, so that it will repair only the range 
 that node is responsible for.
 On Thu, Jun 18, 2015 at 10:50 PM, Jean Tremblay 
 jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
 wrote:
 Thanks Jonathan.
 But I need to know the following:
 If you issue a “nodetool repair” on one node will it repair all the nodes in 
 the cluster or only the one on which we issue the command?
 If it repairs only one node, do I have to wait that the nodetool repair ends, 
 and only then issue another “nodetool repair” on the next node?
 Kind regards
 On 18 Jun 2015, at 19:19 , Jonathan Haddad 
 j...@jonhaddad.commailto:j...@jonhaddad.com wrote:
 If you're using DSE, you can schedule it automatically using the repair 
 service.  If you're open source, check out Spotify cassandra reaper, it'll 
 manage it for you.
 https://github.com/spotify/cassandra-reaper
 On Thu, Jun 18, 2015 at 12:36 PM Jean Tremblay 
 jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
 wrote:
 Hi,
 I want to make on a regular base repairs on my cluster as suggested by the 
 documentation.
 I want to do this in a way that the cluster is still responding to read 
 requests.
 So I understand that I should not use the -par switch for that as it will do 
 the repair in parallel and consume all available resources.
 If you issue a “nodetool repair” on one node will it repair all the nodes in 
 the cluster or only the one on which we issue the command?
 If it repairs only one node, do I have to wait that the nodetool repair ends, 
 and only then issue another “nodetool repair” on the next node?
 If we had down time periods I would issue a nodetool -par, but we don’t have 
 down time periods.
 Sorry for the stupid questions.
 Thanks for your help.
 --
 Arun
 Senior Hadoop/Cassandra Engineer
 Cloudwick
 2014 Data Impact Award Winner (Cloudera)
 http://www.cloudera.com/content/cloudera/en/campaign/data-impact

Re: nodetool repair

[Jean Tremblay]

Perfect thank you.
So making a weekly nodetool repair -pr”  on all nodes one after the other will 
repair my cluster. That is great.

If it does a compaction, does it mean that it would also clean up my tombstone 
from my LeveledCompactionStrategy tables at the same time?

Thanks for your help.

On 19 Jun 2015, at 07:56 , arun sirimalla 
arunsi...@gmail.commailto:arunsi...@gmail.com wrote:

Hi Jean,

Running nodetool repair on a node will repair only that node in the cluster. It 
is recommended to run nodetool repair on one node at a time.

Few things to keep in mind while running repair
   1. Running repair will trigger compactions
   2. Increase in CPU utilization.


Run node tool repair with -pr option, so that it will repair only the range 
that node is responsible for.

On Thu, Jun 18, 2015 at 10:50 PM, Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
wrote:
Thanks Jonathan.

But I need to know the following:

If you issue a “nodetool repair” on one node will it repair all the nodes in 
the cluster or only the one on which we issue the command?

If it repairs only one node, do I have to wait that the nodetool repair ends, 
and only then issue another “nodetool repair” on the next node?

Kind regards

On 18 Jun 2015, at 19:19 , Jonathan Haddad 
j...@jonhaddad.commailto:j...@jonhaddad.com wrote:

If you're using DSE, you can schedule it automatically using the repair 
service.  If you're open source, check out Spotify cassandra reaper, it'll 
manage it for you.

https://github.com/spotify/cassandra-reaper



On Thu, Jun 18, 2015 at 12:36 PM Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
wrote:
Hi,

I want to make on a regular base repairs on my cluster as suggested by the 
documentation.
I want to do this in a way that the cluster is still responding to read 
requests.
So I understand that I should not use the -par switch for that as it will do 
the repair in parallel and consume all available resources.

If you issue a “nodetool repair” on one node will it repair all the nodes in 
the cluster or only the one on which we issue the command?

If it repairs only one node, do I have to wait that the nodetool repair ends, 
and only then issue another “nodetool repair” on the next node?

If we had down time periods I would issue a nodetool -par, but we don’t have 
down time periods.

Sorry for the stupid questions.
Thanks for your help.




--
Arun
Senior Hadoop/Cassandra Engineer
Cloudwick


2014 Data Impact Award Winner (Cloudera)
http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html

Re: nodetool repair

[arun sirimalla]

Yes compactions will remove tombstones

On Thu, Jun 18, 2015 at 11:46 PM, Jean Tremblay 
jean.tremb...@zen-innovations.com wrote:

  Perfect thank you.
 So making a weekly nodetool repair -pr”  on all nodes one after the other
 will repair my cluster. That is great.

  If it does a compaction, does it mean that it would also clean up my
 tombstone from my LeveledCompactionStrategy tables at the same time?

  Thanks for your help.

  On 19 Jun 2015, at 07:56 , arun sirimalla arunsi...@gmail.com wrote:

  Hi Jean,

  Running nodetool repair on a node will repair only that node in the
 cluster. It is recommended to run nodetool repair on one node at a time.

  Few things to keep in mind while running repair
1. Running repair will trigger compactions
2. Increase in CPU utilization.


  Run node tool repair with -pr option, so that it will repair only the
 range that node is responsible for.

 On Thu, Jun 18, 2015 at 10:50 PM, Jean Tremblay 
 jean.tremb...@zen-innovations.com wrote:

 Thanks Jonathan.

  But I need to know the following:

  If you issue a “nodetool repair” on one node will it repair all the
 nodes in the cluster or only the one on which we issue the command?

If it repairs only one node, do I have to wait that the nodetool
 repair ends, and only then issue another “nodetool repair” on the next node?

  Kind regards

  On 18 Jun 2015, at 19:19 , Jonathan Haddad j...@jonhaddad.com wrote:

  If you're using DSE, you can schedule it automatically using the repair
 service.  If you're open source, check out Spotify cassandra reaper, it'll
 manage it for you.

  https://github.com/spotify/cassandra-reaper



  On Thu, Jun 18, 2015 at 12:36 PM Jean Tremblay 
 jean.tremb...@zen-innovations.com wrote:

 Hi,

 I want to make on a regular base repairs on my cluster as suggested by
 the documentation.
 I want to do this in a way that the cluster is still responding to read
 requests.
 So I understand that I should not use the -par switch for that as it
 will do the repair in parallel and consume all available resources.

 If you issue a “nodetool repair” on one node will it repair all the
 nodes in the cluster or only the one on which we issue the command?

 If it repairs only one node, do I have to wait that the nodetool repair
 ends, and only then issue another “nodetool repair” on the next node?

 If we had down time periods I would issue a nodetool -par, but we don’t
 have down time periods.

 Sorry for the stupid questions.
 Thanks for your help.





  --
 Arun
 Senior Hadoop/Cassandra Engineer
 Cloudwick


  2014 Data Impact Award Winner (Cloudera)

 http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html





-- 
Arun
Senior Hadoop/Cassandra Engineer
Cloudwick


2014 Data Impact Award Winner (Cloudera)
http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html

Re: nodetool repair

[Alain RODRIGUEZ]

Hi,

This is not necessarily true. Repair will induce compactions only if you
have entropy in your cluster. If not it will just read your data to compare
all the replica of each piece of data (using indeed cpu and disk IO).

If there is some data missing it will repair it. Though, due to merkle
tree size, you will generally stream more data than just the data needed.
To limit this downside and the compactions amount, use range repairs --
http://www.datastax.com/dev/blog/advanced-repair-techniques.

About tombstones, they will be evicted only after gc_grace_period and only
if all the parts of the row are part of the compaction.

C*heers,

Alain

2015-06-19 9:08 GMT+02:00 arun sirimalla arunsi...@gmail.com:

 Yes compactions will remove tombstones

 On Thu, Jun 18, 2015 at 11:46 PM, Jean Tremblay 
 jean.tremb...@zen-innovations.com wrote:

  Perfect thank you.
 So making a weekly nodetool repair -pr”  on all nodes one after the
 other will repair my cluster. That is great.

  If it does a compaction, does it mean that it would also clean up my
 tombstone from my LeveledCompactionStrategy tables at the same time?

  Thanks for your help.

  On 19 Jun 2015, at 07:56 , arun sirimalla arunsi...@gmail.com wrote:

  Hi Jean,

  Running nodetool repair on a node will repair only that node in the
 cluster. It is recommended to run nodetool repair on one node at a time.

  Few things to keep in mind while running repair
1. Running repair will trigger compactions
2. Increase in CPU utilization.


  Run node tool repair with -pr option, so that it will repair only the
 range that node is responsible for.

 On Thu, Jun 18, 2015 at 10:50 PM, Jean Tremblay 
 jean.tremb...@zen-innovations.com wrote:

 Thanks Jonathan.

  But I need to know the following:

  If you issue a “nodetool repair” on one node will it repair all the
 nodes in the cluster or only the one on which we issue the command?

If it repairs only one node, do I have to wait that the nodetool
 repair ends, and only then issue another “nodetool repair” on the next node?

  Kind regards

  On 18 Jun 2015, at 19:19 , Jonathan Haddad j...@jonhaddad.com wrote:

  If you're using DSE, you can schedule it automatically using the
 repair service.  If you're open source, check out Spotify cassandra reaper,
 it'll manage it for you.

  https://github.com/spotify/cassandra-reaper



  On Thu, Jun 18, 2015 at 12:36 PM Jean Tremblay 
 jean.tremb...@zen-innovations.com wrote:

 Hi,

 I want to make on a regular base repairs on my cluster as suggested by
 the documentation.
 I want to do this in a way that the cluster is still responding to read
 requests.
 So I understand that I should not use the -par switch for that as it
 will do the repair in parallel and consume all available resources.

 If you issue a “nodetool repair” on one node will it repair all the
 nodes in the cluster or only the one on which we issue the command?

 If it repairs only one node, do I have to wait that the nodetool repair
 ends, and only then issue another “nodetool repair” on the next node?

 If we had down time periods I would issue a nodetool -par, but we don’t
 have down time periods.

 Sorry for the stupid questions.
 Thanks for your help.





  --
 Arun
 Senior Hadoop/Cassandra Engineer
 Cloudwick


  2014 Data Impact Award Winner (Cloudera)

 http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html





 --
 Arun
 Senior Hadoop/Cassandra Engineer
 Cloudwick


 2014 Data Impact Award Winner (Cloudera)

 http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html

Re: nodetool repair

[Jean Tremblay]

Thank you for your reply.



On 19 Jun 2015, at 20:36, 
sean_r_dur...@homedepot.commailto:sean_r_dur...@homedepot.com 
sean_r_dur...@homedepot.commailto:sean_r_dur...@homedepot.com wrote:

It seems to me that running repair on any given node may also induce repairs to 
related replica nodes. For example, if I run repair on node A and node B has 
some replicas, data might stream from A to B (assuming A has newer/more data). 
Now, that does NOT mean that node B will be fully repaired. You still need to 
run repair -pr on all nodes before gc_grace_seconds.

You can run repairs on multiple nodes at the same time. However, you might end 
up with a large amount of streaming, if many repairs are needed. So, you should 
be aware of a performance impact.

I run weekly repairs on one node at a time, if possible. On, larger rings, 
though, I run repairs on multiple nodes staggered by a few hours. Once your 
routine maintenance is established, repairs will not run for very long. But, if 
you have a large ring that hasn’t been repaired, those first repairs may take 
days (but should get faster as you get further through the ring).


Sean Durity

From: Alain RODRIGUEZ [mailto:arodr...@gmail.com]
Sent: Friday, June 19, 2015 3:56 AM
To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
Subject: Re: nodetool repair

Hi,

This is not necessarily true. Repair will induce compactions only if you have 
entropy in your cluster. If not it will just read your data to compare all the 
replica of each piece of data (using indeed cpu and disk IO).

If there is some data missing it will repair it. Though, due to merkle tree 
size, you will generally stream more data than just the data needed. To limit 
this downside and the compactions amount, use range repairs -- 
http://www.datastax.com/dev/blog/advanced-repair-techniques.

About tombstones, they will be evicted only after gc_grace_period and only if 
all the parts of the row are part of the compaction.

C*heers,

Alain

2015-06-19 9:08 GMT+02:00 arun sirimalla 
arunsi...@gmail.commailto:arunsi...@gmail.com:
Yes compactions will remove tombstones

On Thu, Jun 18, 2015 at 11:46 PM, Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
wrote:
Perfect thank you.
So making a weekly nodetool repair -pr”  on all nodes one after the other will 
repair my cluster. That is great.

If it does a compaction, does it mean that it would also clean up my tombstone 
from my LeveledCompactionStrategy tables at the same time?

Thanks for your help.

On 19 Jun 2015, at 07:56 , arun sirimalla 
arunsi...@gmail.commailto:arunsi...@gmail.com wrote:

Hi Jean,

Running nodetool repair on a node will repair only that node in the cluster. It 
is recommended to run nodetool repair on one node at a time.

Few things to keep in mind while running repair
   1. Running repair will trigger compactions
   2. Increase in CPU utilization.


Run node tool repair with -pr option, so that it will repair only the range 
that node is responsible for.

On Thu, Jun 18, 2015 at 10:50 PM, Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
wrote:
Thanks Jonathan.

But I need to know the following:

If you issue a “nodetool repair” on one node will it repair all the nodes in 
the cluster or only the one on which we issue the command?
If it repairs only one node, do I have to wait that the nodetool repair ends, 
and only then issue another “nodetool repair” on the next node?

Kind regards

On 18 Jun 2015, at 19:19 , Jonathan Haddad 
j...@jonhaddad.commailto:j...@jonhaddad.com wrote:

If you're using DSE, you can schedule it automatically using the repair 
service.  If you're open source, check out Spotify cassandra reaper, it'll 
manage it for you.

https://github.com/spotify/cassandra-reaper



On Thu, Jun 18, 2015 at 12:36 PM Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
wrote:
Hi,

I want to make on a regular base repairs on my cluster as suggested by the 
documentation.
I want to do this in a way that the cluster is still responding to read 
requests.
So I understand that I should not use the -par switch for that as it will do 
the repair in parallel and consume all available resources.

If you issue a “nodetool repair” on one node will it repair all the nodes in 
the cluster or only the one on which we issue the command?

If it repairs only one node, do I have to wait that the nodetool repair ends, 
and only then issue another “nodetool repair” on the next node?

If we had down time periods I would issue a nodetool -par, but we don’t have 
down time periods.

Sorry for the stupid questions.
Thanks for your help.




--
Arun
Senior Hadoop/Cassandra Engineer
Cloudwick


2014 Data Impact Award Winner (Cloudera)
http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html





--
Arun
Senior Hadoop/Cassandra Engineer
Cloudwick


2014 Data Impact Award Winner (Cloudera)
http

RE: nodetool repair

[SEAN_R_DURITY]

It seems to me that running repair on any given node may also induce repairs to 
related replica nodes. For example, if I run repair on node A and node B has 
some replicas, data might stream from A to B (assuming A has newer/more data). 
Now, that does NOT mean that node B will be fully repaired. You still need to 
run repair -pr on all nodes before gc_grace_seconds.

You can run repairs on multiple nodes at the same time. However, you might end 
up with a large amount of streaming, if many repairs are needed. So, you should 
be aware of a performance impact.

I run weekly repairs on one node at a time, if possible. On, larger rings, 
though, I run repairs on multiple nodes staggered by a few hours. Once your 
routine maintenance is established, repairs will not run for very long. But, if 
you have a large ring that hasn’t been repaired, those first repairs may take 
days (but should get faster as you get further through the ring).


Sean Durity

From: Alain RODRIGUEZ [mailto:arodr...@gmail.com]
Sent: Friday, June 19, 2015 3:56 AM
To: user@cassandra.apache.org
Subject: Re: nodetool repair

Hi,

This is not necessarily true. Repair will induce compactions only if you have 
entropy in your cluster. If not it will just read your data to compare all the 
replica of each piece of data (using indeed cpu and disk IO).

If there is some data missing it will repair it. Though, due to merkle tree 
size, you will generally stream more data than just the data needed. To limit 
this downside and the compactions amount, use range repairs -- 
http://www.datastax.com/dev/blog/advanced-repair-techniques.

About tombstones, they will be evicted only after gc_grace_period and only if 
all the parts of the row are part of the compaction.

C*heers,

Alain

2015-06-19 9:08 GMT+02:00 arun sirimalla 
arunsi...@gmail.commailto:arunsi...@gmail.com:
Yes compactions will remove tombstones

On Thu, Jun 18, 2015 at 11:46 PM, Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
wrote:
Perfect thank you.
So making a weekly nodetool repair -pr”  on all nodes one after the other will 
repair my cluster. That is great.

If it does a compaction, does it mean that it would also clean up my tombstone 
from my LeveledCompactionStrategy tables at the same time?

Thanks for your help.

On 19 Jun 2015, at 07:56 , arun sirimalla 
arunsi...@gmail.commailto:arunsi...@gmail.com wrote:

Hi Jean,

Running nodetool repair on a node will repair only that node in the cluster. It 
is recommended to run nodetool repair on one node at a time.

Few things to keep in mind while running repair
   1. Running repair will trigger compactions
   2. Increase in CPU utilization.


Run node tool repair with -pr option, so that it will repair only the range 
that node is responsible for.

On Thu, Jun 18, 2015 at 10:50 PM, Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
wrote:
Thanks Jonathan.

But I need to know the following:

If you issue a “nodetool repair” on one node will it repair all the nodes in 
the cluster or only the one on which we issue the command?
If it repairs only one node, do I have to wait that the nodetool repair ends, 
and only then issue another “nodetool repair” on the next node?

Kind regards

On 18 Jun 2015, at 19:19 , Jonathan Haddad 
j...@jonhaddad.commailto:j...@jonhaddad.com wrote:

If you're using DSE, you can schedule it automatically using the repair 
service.  If you're open source, check out Spotify cassandra reaper, it'll 
manage it for you.

https://github.com/spotify/cassandra-reaper



On Thu, Jun 18, 2015 at 12:36 PM Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
wrote:
Hi,

I want to make on a regular base repairs on my cluster as suggested by the 
documentation.
I want to do this in a way that the cluster is still responding to read 
requests.
So I understand that I should not use the -par switch for that as it will do 
the repair in parallel and consume all available resources.

If you issue a “nodetool repair” on one node will it repair all the nodes in 
the cluster or only the one on which we issue the command?

If it repairs only one node, do I have to wait that the nodetool repair ends, 
and only then issue another “nodetool repair” on the next node?

If we had down time periods I would issue a nodetool -par, but we don’t have 
down time periods.

Sorry for the stupid questions.
Thanks for your help.




--
Arun
Senior Hadoop/Cassandra Engineer
Cloudwick


2014 Data Impact Award Winner (Cloudera)
http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html





--
Arun
Senior Hadoop/Cassandra Engineer
Cloudwick


2014 Data Impact Award Winner (Cloudera)
http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html





The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely

Re: nodetool repair

[Jonathan Haddad]

If you're using DSE, you can schedule it automatically using the repair
service.  If you're open source, check out Spotify cassandra reaper, it'll
manage it for you.

https://github.com/spotify/cassandra-reaper



On Thu, Jun 18, 2015 at 12:36 PM Jean Tremblay 
jean.tremb...@zen-innovations.com wrote:

 Hi,

 I want to make on a regular base repairs on my cluster as suggested by the
 documentation.
 I want to do this in a way that the cluster is still responding to read
 requests.
 So I understand that I should not use the -par switch for that as it will
 do the repair in parallel and consume all available resources.

 If you issue a “nodetool repair” on one node will it repair all the nodes
 in the cluster or only the one on which we issue the command?

 If it repairs only one node, do I have to wait that the nodetool repair
 ends, and only then issue another “nodetool repair” on the next node?

 If we had down time periods I would issue a nodetool -par, but we don’t
 have down time periods.

 Sorry for the stupid questions.
 Thanks for your help.

Re: nodetool repair

[Jean Tremblay]

Thanks Jonathan.

But I need to know the following:

If you issue a “nodetool repair” on one node will it repair all the nodes in 
the cluster or only the one on which we issue the command?

If it repairs only one node, do I have to wait that the nodetool repair ends, 
and only then issue another “nodetool repair” on the next node?

Kind regards

On 18 Jun 2015, at 19:19 , Jonathan Haddad 
j...@jonhaddad.commailto:j...@jonhaddad.com wrote:

If you're using DSE, you can schedule it automatically using the repair 
service.  If you're open source, check out Spotify cassandra reaper, it'll 
manage it for you.

https://github.com/spotify/cassandra-reaper



On Thu, Jun 18, 2015 at 12:36 PM Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
wrote:
Hi,

I want to make on a regular base repairs on my cluster as suggested by the 
documentation.
I want to do this in a way that the cluster is still responding to read 
requests.
So I understand that I should not use the -par switch for that as it will do 
the repair in parallel and consume all available resources.

If you issue a “nodetool repair” on one node will it repair all the nodes in 
the cluster or only the one on which we issue the command?

If it repairs only one node, do I have to wait that the nodetool repair ends, 
and only then issue another “nodetool repair” on the next node?

If we had down time periods I would issue a nodetool -par, but we don’t have 
down time periods.

Sorry for the stupid questions.
Thanks for your help.

Re: nodetool repair

[arun sirimalla]

Hi Jean,

Running nodetool repair on a node will repair only that node in the
cluster. It is recommended to run nodetool repair on one node at a time.

Few things to keep in mind while running repair
   1. Running repair will trigger compactions
   2. Increase in CPU utilization.


Run node tool repair with -pr option, so that it will repair only the range
that node is responsible for.

On Thu, Jun 18, 2015 at 10:50 PM, Jean Tremblay 
jean.tremb...@zen-innovations.com wrote:

  Thanks Jonathan.

  But I need to know the following:

  If you issue a “nodetool repair” on one node will it repair all the
 nodes in the cluster or only the one on which we issue the command?

If it repairs only one node, do I have to wait that the nodetool
 repair ends, and only then issue another “nodetool repair” on the next node?

  Kind regards

  On 18 Jun 2015, at 19:19 , Jonathan Haddad j...@jonhaddad.com wrote:

  If you're using DSE, you can schedule it automatically using the repair
 service.  If you're open source, check out Spotify cassandra reaper, it'll
 manage it for you.

  https://github.com/spotify/cassandra-reaper



  On Thu, Jun 18, 2015 at 12:36 PM Jean Tremblay 
 jean.tremb...@zen-innovations.com wrote:

 Hi,

 I want to make on a regular base repairs on my cluster as suggested by
 the documentation.
 I want to do this in a way that the cluster is still responding to read
 requests.
 So I understand that I should not use the -par switch for that as it will
 do the repair in parallel and consume all available resources.

 If you issue a “nodetool repair” on one node will it repair all the nodes
 in the cluster or only the one on which we issue the command?

 If it repairs only one node, do I have to wait that the nodetool repair
 ends, and only then issue another “nodetool repair” on the next node?

 If we had down time periods I would issue a nodetool -par, but we don’t
 have down time periods.

 Sorry for the stupid questions.
 Thanks for your help.





-- 
Arun
Senior Hadoop/Cassandra Engineer
Cloudwick


2014 Data Impact Award Winner (Cloudera)
http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html

Re: nodetool repair options

[Robert Coli]

On Fri, Jan 23, 2015 at 10:03 AM, Robert Wille rwi...@fold3.com wrote:

 The docs say Use -pr to repair only the first range returned by the
 partitioner”. What does this mean? Why would I only want to repair the
 first range?


If you're repairing the whole cluster, repairing only the primary range on
each node avoids avoiding once per replication factor.


 What are the tradeoffs of a parallel versus serial repair?


Parallel repair affects all replicas simultaneously and can thereby degrade
latency for that replica set. Serial repair doesn't, but is serial and
intensely slower. Serial repair is probably not usable at all with RF5 or
so, unless you set an extremely long gc_grace_seconds.


 What are the recommended options for regular, periodic repair?


(Snapshot/incremental repair, default IIRC in newer Cassandra, changes many
of these assumptions. I refer to old-style nodetool repair with my
statements.)

The canonical response is repair the entire cluster with -pr once per
gc_grace_seconds.

Regarding frequent repair... consider your RF, CL and whether you actually
care about consistency and durability for any given colunfamily. If you
never do DELETE-like-operations (in CQL, this includes things other than
DELETE statements) in the CF, probably don't repair it just for consistency
purposes.

Then, consider how long you can tolerate DELETEd data sticking around. If
you can tolerate it because you don't DELETE much data, set
gc_grace_seconds to at least 34 days. With 34 days, you can begin a repair
on the first of the month and have between 3 and 7 days for it to complete.
You repair for up to a few days in order to repair a month's data. With
shorter repair cycles, you pay the relatively high cost of repair
repeatedly.

Last, consider your Cassandra version. Newer versions have had significant
focus on streaming and repair stability and performance. Upgrade to the
HEAD of 2.0.x if possible.

There's this thing I jokingly call the Coli Conjecture, which says that if
you're in a good case for Cassandra you probably don't actually don't care
about consistency or durability, even if you think you do. This comes from
years of observing consistency edge cases in Cassandra and noticing that
even very few people who detected them and reported them seemed to
experience very negative results from the perspective of their application.
I think it is an interesting observation and a different mindset for many
people coming from the non-distributed, normalized, relational world.

=Rob

Re: nodetool repair

[Robert Coli]

On Fri, Jan 9, 2015 at 8:01 AM, Adil adil.cha...@gmail.com wrote:

 We have two DC, we are planning to schedule running nodetool repair
 weekly, my question is : nodetool repair is cross cluster or not? it's
 sufficient to run it without options on a node or should be scheduled on
 every node with the host option.


The latter, generally. Use -pr when repairing ones whole cluster.

=Rob

Re: nodetool repair exception

[Eric Stevens]

The official recommendation is 100k:
http://www.datastax.com/documentation/cassandra/2.0/cassandra/install/installRecommendSettings.html

I wonder if there's an advantage to this over unlimited if you're running
servers which are dedicated to your Cassandra cluster (which you should be
for anything production).

On Fri Dec 05 2014 at 2:39:24 PM Robert Coli rc...@eventbrite.com wrote:

 On Wed, Dec 3, 2014 at 6:37 AM, Rafał Furmański rfurman...@opera.com
 wrote:

 I see “Too many open files” exception in logs, but I’m sure that my limit
 is now 150k.
 Should I increase it? What’s the reasonable limit of open files for
 cassandra?


 Why provide any limit? ulimit allows unlimited?

 =Rob

Re: nodetool repair exception

[Tim Heckman]

On Sat, Dec 6, 2014 at 8:05 AM, Eric Stevens migh...@gmail.com wrote:
 The official recommendation is 100k:
 http://www.datastax.com/documentation/cassandra/2.0/cassandra/install/installRecommendSettings.html

 I wonder if there's an advantage to this over unlimited if you're running
 servers which are dedicated to your Cassandra cluster (which you should be
 for anything production).

There is the potential to have monitoring systems, and other small
agents, running on systems in production. I could see this simply as a
stop-gap to prevent Cassandra from being able to starve the system of
free file descriptors. In theory, if there's not a proper watchdog on
your monitors this could prevent an issue from causing an alert.
However, just a potential advantage I could think of.

Cheers!
-Tim

 On Fri Dec 05 2014 at 2:39:24 PM Robert Coli rc...@eventbrite.com wrote:

 On Wed, Dec 3, 2014 at 6:37 AM, Rafał Furmański rfurman...@opera.com
 wrote:

 I see “Too many open files” exception in logs, but I’m sure that my limit
 is now 150k.
 Should I increase it? What’s the reasonable limit of open files for
 cassandra?


 Why provide any limit? ulimit allows unlimited?

 =Rob

Re: nodetool repair exception

[Robert Coli]

On Wed, Dec 3, 2014 at 6:37 AM, Rafał Furmański rfurman...@opera.com
wrote:

 I see “Too many open files” exception in logs, but I’m sure that my limit
 is now 150k.
 Should I increase it? What’s the reasonable limit of open files for
 cassandra?


Why provide any limit? ulimit allows unlimited?

=Rob

Re: nodetool repair exception

[Yuki Morishita]

As the exception indicates, nodetool just lost communication with the
Cassandra node and cannot print progress any further.
Check your system.log on the node, and see if your repair was
completed. If there is no error, then it should be fine.

On Wed, Dec 3, 2014 at 5:08 AM, Rafał Furmański rfurman...@opera.com wrote:
 Hi All!

 We have a 8 nodes cluster in 2 DC (4 per DC, RF=3) running Cassandra 2.1.2 on 
 Linux Debian Wheezy.
 I executed “nodetool repair” on one of the nodes, and this command returned 
 following error:

 Exception occurred during clean-up. 
 java.lang.reflect.UndeclaredThrowableException
 error: JMX connection closed. You should check server log for repair status 
 of keyspace sync(Subsequent keyspaces are not going to be repaired).
 -- StackTrace --
 java.io.IOException: JMX connection closed. You should check server log for 
 repair status of keyspace sync(Subsequent keyspaces are not going to be 
 repaired).
at 
 org.apache.cassandra.tools.RepairRunner.handleNotification(NodeProbe.java:1351)
at 
 javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:274)
at 
 javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:339)
at 
 javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:324)
at 
 javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:247)
at 
 javax.management.remote.rmi.RMIConnector.sendNotification(RMIConnector.java:441)
at 
 javax.management.remote.rmi.RMIConnector.access$1100(RMIConnector.java:121)
at 
 javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.gotIOException(RMIConnector.java:1505)
at 
 javax.management.remote.rmi.RMIConnector$RMINotifClient.fetchNotifs(RMIConnector.java:1350)
at 
 com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchNotifs(ClientNotifForwarder.java:587)
at 
 com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:470)
at 
 com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:451)
at 
 com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:107)

 This error was followed by lots of “Lost Notification” messages.
 Node became unusable and I had to restart it. Is this an issue?

 Rafal



-- 
Yuki Morishita
 t:yukim (http://twitter.com/yukim)

Re: nodetool repair exception

[Rafał Furmański]

I see “Too many open files” exception in logs, but I’m sure that my limit is 
now 150k.
Should I increase it? What’s the reasonable limit of open files for cassandra?

On 3 gru 2014, at 15:02, Yuki Morishita mor.y...@gmail.com wrote:

 As the exception indicates, nodetool just lost communication with the
 Cassandra node and cannot print progress any further.
 Check your system.log on the node, and see if your repair was
 completed. If there is no error, then it should be fine.
 
 On Wed, Dec 3, 2014 at 5:08 AM, Rafał Furmański rfurman...@opera.com wrote:
 Hi All!
 
 We have a 8 nodes cluster in 2 DC (4 per DC, RF=3) running Cassandra 2.1.2 
 on Linux Debian Wheezy.
 I executed “nodetool repair” on one of the nodes, and this command returned 
 following error:
 
 Exception occurred during clean-up. 
 java.lang.reflect.UndeclaredThrowableException
 error: JMX connection closed. You should check server log for repair status 
 of keyspace sync(Subsequent keyspaces are not going to be repaired).
 -- StackTrace --
 java.io.IOException: JMX connection closed. You should check server log for 
 repair status of keyspace sync(Subsequent keyspaces are not going to be 
 repaired).
   at 
 org.apache.cassandra.tools.RepairRunner.handleNotification(NodeProbe.java:1351)
   at 
 javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:274)
   at 
 javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:339)
   at 
 javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:324)
   at 
 javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:247)
   at 
 javax.management.remote.rmi.RMIConnector.sendNotification(RMIConnector.java:441)
   at 
 javax.management.remote.rmi.RMIConnector.access$1100(RMIConnector.java:121)
   at 
 javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.gotIOException(RMIConnector.java:1505)
   at 
 javax.management.remote.rmi.RMIConnector$RMINotifClient.fetchNotifs(RMIConnector.java:1350)
   at 
 com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchNotifs(ClientNotifForwarder.java:587)
   at 
 com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:470)
   at 
 com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:451)
   at 
 com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:107)
 
 This error was followed by lots of “Lost Notification” messages.
 Node became unusable and I had to restart it. Is this an issue?
 
 Rafal
 
 
 
 -- 
 Yuki Morishita
 t:yukim (http://twitter.com/yukim)



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: nodetool repair stalled

[Eric Stevens]

Wouldn't it be a better idea to issue removenode on the crashed node, wipe
the whole data directory (including system) and let it bootstrap cleanly so
that it's not part of the cluster while it gets back up to speed?

On Tue, Nov 11, 2014, 12:32 PM Robert Coli rc...@eventbrite.com wrote:

 On Tue, Nov 11, 2014 at 10:48 AM, venkat sam samvenkat...@outlook.com
 wrote:


 I have a 5 node cluster. In one node one of the data directory partition
 got crashed. After disk replacement I restarted the Cassandra daemon and
 gave nodetool repair to restore the missing replica’s. But nodetool repair
 is getting stuck after syncing one of the columnfamily


 Yes, nodetool repair often hangs. Search through the archives, but the
 summary is.

 1) try to repair CFs one at a time
 2) it's worse with vnodes
 3) try tuning the phi detector or network stream timeouts

 =Rob

Re: nodetool repair stalled

[Robert Coli]

On Wed, Nov 12, 2014 at 6:50 AM, Eric Stevens migh...@gmail.com wrote:

 Wouldn't it be a better idea to issue removenode on the crashed node, wipe
 the whole data directory (including system) and let it bootstrap cleanly so
 that it's not part of the cluster while it gets back up to

Yes, with replace_node. I missed that the entire data dir was lost in my
first response.

=Rob

Re: nodetool repair stalled

[venkat sam]

Hi Eric,

The data are stored in JBOD. Only one of the disk got crashed other 3 disk 
still holds the old data . That's why I didn't clean the whole node and issue a 
fresh restart


Thanks Rob. Will do try that way.






From: Eric Stevens
Sent: ‎Wednesday‎, ‎November‎ ‎12‎, ‎2014 ‎8‎:‎21‎ ‎PM
To: user@cassandra.apache.org





Wouldn't it be a better idea to issue removenode on the crashed node, wipe the 
whole data directory (including system) and let it bootstrap cleanly so that 
it's not part of the cluster while it gets back up to speed?



On Tue, Nov 11, 2014, 12:32 PM Robert Coli rc...@eventbrite.com wrote:




On Tue, Nov 11, 2014 at 10:48 AM, venkat sam samvenkat...@outlook.com wrote:







I have a 5 node cluster. In one node one of the data directory partition got 
crashed. After disk replacement I restarted the Cassandra daemon and gave 
nodetool repair to restore the missing replica’s. But nodetool repair is 
getting stuck after syncing one of the columnfamily




Yes, nodetool repair often hangs. Search through the archives, but the summary 
is.




1) try to repair CFs one at a time

2) it's worse with vnodes

3) try tuning the phi detector or network stream timeouts




=Rob

Re: Nodetool Repair questions

[Mark Reddy]

Hi Vish,

1. This tool repairs inconsistencies across replicas of the row. Since
 latest update always wins, I dont see inconsistencies other than ones
 resulting from the combination of deletes, tombstones, and crashed nodes.
 Technically, if data is never deleted from cassandra, then nodetool repair
 does not need to be run at all. Is this understanding correct? If wrong,
 can anyone provide other ways inconsistencies could occur?


Even if you never delete data you should run repairs occasionally to ensure
overall consistency. While hinted handoffs and read repairs do lead to
better consistency, they are only helpers/optimization and are not regarded
as operations that ensure consistency.

2. Want to understand the performance of 'nodetool repair' in a Cassandra
 multi data center setup. As we add nodes to the cluster in various data
 centers, does the performance of nodetool repair on each node increase
 linearly, or is it quadratic ?


Its difficult to calculate the performance of a repair, I've seen the time
to completion fluctuate between 4hrs to 10hrs+ on the same node. However in
theory adding more nodes would spread the data and free up machine
resources, thus resulting in more performant repairs.

The essence of this question is: If I have a keyspace with x number of
 replicas in each data center, do I have to deal with an upper limit on the
 number of data centers/nodes?


Could you expand on why you believe there would be an upper limit of
dc/nodes due to running repairs?


Mark


On Tue, Aug 12, 2014 at 10:06 PM, Viswanathan Ramachandran 
vish.ramachand...@gmail.com wrote:

 Some questions on nodetool repair.

 1. This tool repairs inconsistencies across replicas of the row. Since
 latest update always wins, I dont see inconsistencies other than ones
 resulting from the combination of deletes, tombstones, and crashed nodes.
 Technically, if data is never deleted from cassandra, then nodetool repair
 does not need to be run at all. Is this understanding correct? If wrong,
 can anyone provide other ways inconsistencies could occur?

 2. Want to understand the performance of 'nodetool repair' in a Cassandra
 multi data center setup. As we add nodes to the cluster in various data
 centers, does the performance of nodetool repair on each node increase
 linearly, or is it quadratic ? The essence of this question is: If I have a
 keyspace with x number of replicas in each data center, do I have to deal
 with an upper limit on the number of data centers/nodes?


 Thanks

 Vish

Re: Nodetool Repair questions

[Andrey Ilinykh]

1. You don't have to repair if you use QUORUM consistency and you don't
delete data.
2.Performance depends on size of data each node has. It's very difficult to
predict. It may take days.

Thank you,
  Andrey


On Tue, Aug 12, 2014 at 2:06 PM, Viswanathan Ramachandran 
vish.ramachand...@gmail.com wrote:

 Some questions on nodetool repair.

 1. This tool repairs inconsistencies across replicas of the row. Since
 latest update always wins, I dont see inconsistencies other than ones
 resulting from the combination of deletes, tombstones, and crashed nodes.
 Technically, if data is never deleted from cassandra, then nodetool repair
 does not need to be run at all. Is this understanding correct? If wrong,
 can anyone provide other ways inconsistencies could occur?

 2. Want to understand the performance of 'nodetool repair' in a Cassandra
 multi data center setup. As we add nodes to the cluster in various data
 centers, does the performance of nodetool repair on each node increase
 linearly, or is it quadratic ? The essence of this question is: If I have a
 keyspace with x number of replicas in each data center, do I have to deal
 with an upper limit on the number of data centers/nodes?


 Thanks

 Vish

Re: Nodetool Repair questions

[Viswanathan Ramachandran]

Thanks Mark,
Since we have replicas in each data center, addition of a new data center
(and new replicas) has a performance implication on nodetool repair.
I do understand that adding nodes without increasing number of replicas may
improve repair performance, but in this case we are adding new data center
and additional replicas which is an added overhead on nodetool repair.
Hence the thinking that we may reach an upper limit which could be the
point when the nodetool repair costs are way too high.


On Tue, Aug 12, 2014 at 2:59 PM, Mark Reddy mark.re...@boxever.com wrote:

 Hi Vish,

 1. This tool repairs inconsistencies across replicas of the row. Since
 latest update always wins, I dont see inconsistencies other than ones
 resulting from the combination of deletes, tombstones, and crashed nodes.
 Technically, if data is never deleted from cassandra, then nodetool repair
 does not need to be run at all. Is this understanding correct? If wrong,
 can anyone provide other ways inconsistencies could occur?


 Even if you never delete data you should run repairs occasionally to
 ensure overall consistency. While hinted handoffs and read repairs do lead
 to better consistency, they are only helpers/optimization and are not
 regarded as operations that ensure consistency.

 2. Want to understand the performance of 'nodetool repair' in a Cassandra
 multi data center setup. As we add nodes to the cluster in various data
 centers, does the performance of nodetool repair on each node increase
 linearly, or is it quadratic ?


 Its difficult to calculate the performance of a repair, I've seen the time
 to completion fluctuate between 4hrs to 10hrs+ on the same node. However in
 theory adding more nodes would spread the data and free up machine
 resources, thus resulting in more performant repairs.

 The essence of this question is: If I have a keyspace with x number of
 replicas in each data center, do I have to deal with an upper limit on the
 number of data centers/nodes?


 Could you expand on why you believe there would be an upper limit of
 dc/nodes due to running repairs?


 Mark


 On Tue, Aug 12, 2014 at 10:06 PM, Viswanathan Ramachandran 
 vish.ramachand...@gmail.com wrote:

  Some questions on nodetool repair.

 1. This tool repairs inconsistencies across replicas of the row. Since
 latest update always wins, I dont see inconsistencies other than ones
 resulting from the combination of deletes, tombstones, and crashed nodes.
 Technically, if data is never deleted from cassandra, then nodetool repair
 does not need to be run at all. Is this understanding correct? If wrong,
 can anyone provide other ways inconsistencies could occur?

 2. Want to understand the performance of 'nodetool repair' in a Cassandra
 multi data center setup. As we add nodes to the cluster in various data
 centers, does the performance of nodetool repair on each node increase
 linearly, or is it quadratic ? The essence of this question is: If I have a
 keyspace with x number of replicas in each data center, do I have to deal
 with an upper limit on the number of data centers/nodes?


 Thanks

 Vish

Re: Nodetool Repair questions

[Viswanathan Ramachandran]

Andrey, QUORUM consistency and no deletes makes perfect sense.
I believe we could modify that to EACH_QUORUM or QUORUM consistency and no
deletes - isnt that right ?

Thanks


On Tue, Aug 12, 2014 at 3:10 PM, Andrey Ilinykh ailin...@gmail.com wrote:

 1. You don't have to repair if you use QUORUM consistency and you don't
 delete data.
 2.Performance depends on size of data each node has. It's very difficult
 to predict. It may take days.

 Thank you,
   Andrey



 On Tue, Aug 12, 2014 at 2:06 PM, Viswanathan Ramachandran 
 vish.ramachand...@gmail.com wrote:

 Some questions on nodetool repair.

 1. This tool repairs inconsistencies across replicas of the row. Since
 latest update always wins, I dont see inconsistencies other than ones
 resulting from the combination of deletes, tombstones, and crashed nodes.
 Technically, if data is never deleted from cassandra, then nodetool repair
 does not need to be run at all. Is this understanding correct? If wrong,
 can anyone provide other ways inconsistencies could occur?

 2. Want to understand the performance of 'nodetool repair' in a Cassandra
 multi data center setup. As we add nodes to the cluster in various data
 centers, does the performance of nodetool repair on each node increase
 linearly, or is it quadratic ? The essence of this question is: If I have a
 keyspace with x number of replicas in each data center, do I have to deal
 with an upper limit on the number of data centers/nodes?


 Thanks

 Vish

Re: Nodetool Repair questions

[Andrey Ilinykh]

On Tue, Aug 12, 2014 at 4:46 PM, Viswanathan Ramachandran 
vish.ramachand...@gmail.com wrote:

 Andrey, QUORUM consistency and no deletes makes perfect sense.
 I believe we could modify that to EACH_QUORUM or QUORUM consistency and no
 deletes - isnt that right?


 yes.

Re: nodetool repair -snapshot option?

[Ken Hancock]

I also expanded on a script originally written by Matt Stump @ Datastax.
The readme has the reasoning behind requiring sub-range repairs.

https://github.com/hancockks/cassandra_range_repair




On Mon, Jun 30, 2014 at 10:20 PM, Phil Burress philburress...@gmail.com
wrote:

 @Paulo, this is very cool! Thanks very much for the link!


 On Mon, Jun 30, 2014 at 9:37 PM, Paulo Ricardo Motta Gomes 
 paulo.mo...@chaordicsystems.com wrote:

 If you find it useful, I created a tool where you input the node IP,
 keyspace, column family, and optionally the number of partitions (default:
 32K), and it outputs the list of subranges for that node, CF, partition
 size: https://github.com/pauloricardomg/cassandra-list-subranges

 So you can basically iterate over the output of that and do subrange
 repair for each node and cf, maybe in parallel. :)


 On Mon, Jun 30, 2014 at 10:26 PM, Phil Burress philburress...@gmail.com
 wrote:

 One last question. Any tips on scripting a subrange repair?


 On Mon, Jun 30, 2014 at 7:12 PM, Phil Burress philburress...@gmail.com
 wrote:

 We are running repair -pr. We've tried subrange manually and that seems
 to work ok. I guess we'll go with that going forward. Thanks for all the
 info!


 On Mon, Jun 30, 2014 at 6:52 PM, Jaydeep Chovatia 
 chovatia.jayd...@gmail.com wrote:

 Are you running full repair or on subset? If you are running full
 repair then try running on sub-set of ranges which means less data to 
 worry
 during repair and that would help JAVA heap in general. You will have to 
 do
 multiple iterations to complete entire range but at-least it will work.

 -jaydeep


 On Mon, Jun 30, 2014 at 3:22 PM, Robert Coli rc...@eventbrite.com
 wrote:

 On Mon, Jun 30, 2014 at 3:08 PM, Yuki Morishita mor.y...@gmail.com
 wrote:

 Repair uses snapshot option by default since 2.0.2 (see NEWS.txt).


 As a general meta comment, the process by which operationally
 important defaults change in Cassandra seems ad-hoc and sub-optimal.

 For to record, my view was that this change, which makes repair even
 slower than it previously was, was probably overly optimistic.

 It's also weird in that it changes default behavior which has been
 unchanged since the start of Cassandra time and is therefore probably
 automated against. Why was it so critically important to switch to 
 snapshot
 repair that it needed to be shotgunned as a new default in 2.0.2?

 =Rob








 --
 *Paulo Motta*

 Chaordic | *Platform*
 *www.chaordic.com.br http://www.chaordic.com.br/*
 +55 48 3232.3200





-- 
*Ken Hancock *| System Architect, Advanced Advertising
SeaChange International
50 Nagog Park
Acton, Massachusetts 01720
ken.hanc...@schange.com | www.schange.com | NASDAQ:SEAC
http://www.schange.com/en-US/Company/InvestorRelations.aspx
Office: +1 (978) 889-3329 | [image: Google Talk:]
ken.hanc...@schange.com | [image:
Skype:]hancockks | [image: Yahoo IM:]hancockks [image: LinkedIn]
http://www.linkedin.com/in/kenhancock

[image: SeaChange International]
 http://www.schange.com/This e-mail and any attachments may contain
information which is SeaChange International confidential. The information
enclosed is intended only for the addressees herein and may not be copied
or forwarded without permission from SeaChange International.

Re: nodetool repair saying starting and then nothing, and nothing in any of the server logs either

[Kevin Burton]

if the boxes are idle, you could use jstack and look at the stack… perhaps
it's locked somewhere.

Worth a shot.


On Tue, Jul 1, 2014 at 9:24 AM, Brian Tarbox tar...@cabotresearch.com
wrote:

 I have a six node cluster in AWS (repl:3) and recently noticed that repair
 was hanging.  I've run with the -pr switch.

 I see this output in the nodetool command line (and also in that node's
 system.log):
  Starting repair command #9, repairing 256 ranges for keyspace dev_a

 but then no other output.  And I see nothing in any of the other node's
 log files.

 Right now the application using C* is turned off so there is zero activity.
 I've let it be in this state for up to 24 hours with nothing more logged.

 Any suggestions?




-- 

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
https://plus.google.com/102718274791889610666/posts
http://spinn3r.com

Re: nodetool repair saying starting and then nothing, and nothing in any of the server logs either

[Robert Coli]

On Tue, Jul 1, 2014 at 9:24 AM, Brian Tarbox tar...@cabotresearch.com
wrote:

 I have a six node cluster in AWS (repl:3) and recently noticed that repair
 was hanging.  I've run with the -pr switch.


It'll do that.

What version of Cassandra?

=Rob

Re: nodetool repair saying starting and then nothing, and nothing in any of the server logs either

[Brian Tarbox]

We're running 1.2.13.

Any chance that doing a rolling-restart would help?

Would running without the -pr improve the odds?

Thanks.


On Tue, Jul 1, 2014 at 1:40 PM, Robert Coli rc...@eventbrite.com wrote:

 On Tue, Jul 1, 2014 at 9:24 AM, Brian Tarbox tar...@cabotresearch.com
 wrote:

 I have a six node cluster in AWS (repl:3) and recently noticed that
 repair was hanging.  I've run with the -pr switch.


 It'll do that.

 What version of Cassandra?

 =Rob