Re: non incremental repairs with cassandra 2.2+

[kurt Greaves]

probably because i was looking the wrong version of the codebase :p

Re: non incremental repairs with cassandra 2.2+

[kurt Greaves]

Welp, that's good but wasn't apparent in the codebase :S.

Kurt Greaves
k...@instaclustr.com
www.instaclustr.com

On 20 October 2016 at 05:02, Alexander Dejanovski 
wrote:

> Hi Kurt,
>
> we're not actually.
> Reaper performs full repair by subrange but does incremental repair on all
> ranges at once, node by node.
> Subrange is incompatible with incremental repair anyway.
>
> Cheers,
>
> On Thu, Oct 20, 2016 at 5:24 AM kurt Greaves  wrote:
>
>>
>> On 19 October 2016 at 17:13, Alexander Dejanovski > > wrote:
>>
>> There aren't that many tools I know to orchestrate repairs and we
>> maintain a fork of Reaper, that was made by Spotify, and handles
>> incremental repair : https://github.com/thelastpickle/cassandra-reaper
>>
>>
>> Looks like you're using subranges with incremental repairs. This will
>> generate a lot of anticompactions as you'll only repair a portion of the
>> SSTables. You should use forceRepairAsync for incremental repairs so that
>> it's possible for the repair to act on the whole SSTable, minimising
>> anticompactions.
>>
>> Kurt Greaves
>> k...@instaclustr.com
>> www.instaclustr.com
>>
> --
> -
> Alexander Dejanovski
> France
> @alexanderdeja
>
> Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>

Re: non incremental repairs with cassandra 2.2+

[Alexander Dejanovski]

Hi Kurt,

we're not actually.
Reaper performs full repair by subrange but does incremental repair on all
ranges at once, node by node.
Subrange is incompatible with incremental repair anyway.

Cheers,

On Thu, Oct 20, 2016 at 5:24 AM kurt Greaves  wrote:

>
> On 19 October 2016 at 17:13, Alexander Dejanovski 
> wrote:
>
> There aren't that many tools I know to orchestrate repairs and we maintain
> a fork of Reaper, that was made by Spotify, and handles incremental repair
> : https://github.com/thelastpickle/cassandra-reaper
>
>
> Looks like you're using subranges with incremental repairs. This will
> generate a lot of anticompactions as you'll only repair a portion of the
> SSTables. You should use forceRepairAsync for incremental repairs so that
> it's possible for the repair to act on the whole SSTable, minimising
> anticompactions.
>
> Kurt Greaves
> k...@instaclustr.com
> www.instaclustr.com
>
-- 
-
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

Re: non incremental repairs with cassandra 2.2+

[kurt Greaves]

On 19 October 2016 at 17:13, Alexander Dejanovski 
wrote:

> There aren't that many tools I know to orchestrate repairs and we maintain
> a fork of Reaper, that was made by Spotify, and handles incremental repair
> : https://github.com/thelastpickle/cassandra-reaper


Looks like you're using subranges with incremental repairs. This will
generate a lot of anticompactions as you'll only repair a portion of the
SSTables. You should use forceRepairAsync for incremental repairs so that
it's possible for the repair to act on the whole SSTable, minimising
anticompactions.

Kurt Greaves
k...@instaclustr.com
www.instaclustr.com

Re: non incremental repairs with cassandra 2.2+

[Kant Kodali]

Sorry I shouldn't have said adding a node. Sometimes data seems to be corrupted 
or inconsistent in which case would like to run a repair. 

Sent from my iPhone

> On Oct 19, 2016, at 10:10 AM, Sean Bridges  
> wrote:
> 
> Thanks, we will try that.
> 
> Sean
> 
>> On 16-10-19 09:34 AM, Alexander Dejanovski wrote:
>> Hi Sean,
>> 
>> you should be able to do that by running subrange repairs, which is the only 
>> type of repair that wouldn't trigger anticompaction AFAIK.
>> Beware that now you will have sstables marked as repaired and others marked 
>> as unrepaired, which will never be compacted   together.
>> You might want to flag all sstables as unrepaired before moving on, if you 
>> do not intend to switch to incremental repair for now.
>> 
>> Cheers,
>> 
>>> On Wed, Oct 19, 2016 at 6:31 PM Sean Bridges  
>>> wrote:
>>> Hey,
>>> 
>>> We are upgrading from cassandra 2.1 to cassandra 2.2.  
>>> 
>>> With cassandra 2.1 we would periodically repair all nodes, using the -pr 
>>> flag.  
>>> 
>>> With cassandra 2.2, the same repair takes a very long time, as cassandra 
>>> does an anti compaction after the repair.  This anti compaction causes most 
>>> (all?) the sstables to be rewritten.  Is there a way to do full repairs 
>>> without continually anti compacting?  If we do a full repair on 
>>> each node with the -pr flag, will subsequent full repairs also force 
>>> anti compacting most (all?) sstables?  
>>> 
>>> Thanks,
>>> 
>>> Sean
>> 
>> -- 
>> -
>> Alexander Dejanovski
>> France
>> @alexanderdeja
>> 
>> Consultant
>> Apache Cassandra Consulting
>> http://www.thelastpickle.com
> 
> 
> -- 
> Sean Bridges
> 
> senior systems architect
> Global Relay 
> 
> sean.brid...@globalrelay.net 
> 
> 866.484.6630 
> New York | Chicago | Vancouver | London (+44.0800.032.9829) | Singapore 
> (+65.3158.1301) 
> Global Relay Archive supports email, instant messaging, BlackBerry, 
> Bloomberg, Thomson Reuters, Pivot, YellowJacket, LinkedIn, Twitter, Facebook 
> and more. 
> 
> Ask about Global Relay Message - The Future of Collaboration in the Financial 
> Services World
> 
> All email sent to or from this address will be retained by Global Relay's 
> email archiving system. This message is intended only for the use of the 
> individual or entity to which it is addressed, and may contain information 
> that is privileged, confidential, and exempt from disclosure under applicable 
> law. Global Relay will not be liable for any compliance or technical 
> information provided herein. All trademarks are the property of their 
> respective owners.
> 

Re: non incremental repairs with cassandra 2.2+

[Alexander Dejanovski]

There aren't that many tools I know to orchestrate repairs and we maintain
a fork of Reaper, that was made by Spotify, and handles incremental repair
: https://github.com/thelastpickle/cassandra-reaper

We just added Cassandra as storage back end (only postgres currently) in
one of the branches, which should soon be merged to master.

Le mer. 19 oct. 2016 19:03, Kant Kodali  a écrit :

Also any suggestions on a tool to orchestrate the incremental repair? Like
say most commonly used

Sent from my iPhone

On Oct 19, 2016, at 9:54 AM, Alexander Dejanovski 
wrote:

Hi Kant,

subrange is a form of full repair, so it will just split the repair process
in smaller yet sequential pieces of work (repair is started giving a start
and end token). Overall, you should not expect improvements other than
having less overstreaming and better chances of success if your cluster is
dense.

You can try to use incremental repair if you know what the caveats are and
use a proper tool to orchestrate it, that would save you from repairing all
10TB each time.
CASSANDRA-12580 might help too as Romain showed us :
https://www.mail-archive.com/user@cassandra.apache.org/msg49344.html

Cheers,



On Wed, Oct 19, 2016 at 6:42 PM Kant Kodali  wrote:

Another question on a same note would be what would be the fastest way to
do repairs of size 10TB cluster ? Full repairs are taking days. So among
repair parallel or repair sub range which is faster in the case of say
adding a new node to the cluster?

Sent from my iPhone

On Oct 19, 2016, at 9:30 AM, Sean Bridges 
wrote:

Hey,

We are upgrading from cassandra 2.1 to cassandra 2.2.

With cassandra 2.1 we would periodically repair all nodes, using the -pr
flag.

With cassandra 2.2, the same repair takes a very long time, as cassandra
does an anti compaction after the repair.  This anti compaction causes most
(all?) the sstables to be rewritten.  Is there a way to do full repairs
without continually anti compacting?  If we do a full repair on each node
with the -pr flag, will subsequent full repairs also force anti compacting
most (all?) sstables?

Thanks,

Sean

-- 
-
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

-- 
-
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

Re: non incremental repairs with cassandra 2.2+

[Sean Bridges]

Thanks, we will try that.

Sean

On 16-10-19 09:34 AM, Alexander Dejanovski wrote:

Hi Sean,

you should be able to do that by running subrange repairs, which is 
the only type of repair that wouldn't trigger anticompaction AFAIK.
Beware that now you will have sstables marked as repaired and others 
marked as unrepaired, which will never be compacted together.
You might want to flag all sstables as unrepaired before moving on, if 
you do not intend to switch to incremental repair for now.


Cheers,

On Wed, Oct 19, 2016 at 6:31 PM Sean Bridges 
> 
wrote:


Hey,

We are upgrading from cassandra 2.1 to cassandra 2.2.

With cassandra 2.1 we would periodically repair all nodes, using
the -pr flag.

With cassandra 2.2, the same repair takes a very long time, as
cassandra does an anti compaction after the repair. This anti
compaction causes most (all?) the sstables to be rewritten.  Is
there a way to do full repairs without continually anti
compacting?  If we do a full repair on each node with the -pr
flag, will subsequent full repairs also force anti compacting most
(all?) sstables?

Thanks,

Sean

--
-
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com 



--

Sean Bridges

senior systems architect
Global Relay

_sean.bridges@globalrelay.net_ 

*866.484.6630 *
New York | Chicago | Vancouver | London (+44.0800.032.9829) | Singapore 
(+65.3158.1301)


Global Relay Archive supports email, instant messaging, BlackBerry, 
Bloomberg, Thomson Reuters, Pivot, YellowJacket, LinkedIn, Twitter, 
Facebook and more.


Ask about *_Global Relay Message_* 
 - The Future of 
Collaboration in the Financial Services World


All email sent to or from this address will be retained by Global 
Relay's email archiving system. This message is intended only for the 
use of the individual or entity to which it is addressed, and may 
contain information that is privileged, confidential, and exempt from 
disclosure under applicable law. Global Relay will not be liable for any 
compliance or technical information provided herein. All trademarks are 
the property of their respective owners.

Re: non incremental repairs with cassandra 2.2+

[Alexander Dejanovski]

Can you explain why you would want to run repair for new nodes?

Aren't you talking about bootstrap, which is not related to repair actually?

Le mer. 19 oct. 2016 18:57, Kant Kodali  a écrit :

> Thanks! How do I do an incremental repair when I add a new node?
>
> Sent from my iPhone
>
> On Oct 19, 2016, at 9:54 AM, Alexander Dejanovski 
> wrote:
>
> Hi Kant,
>
> subrange is a form of full repair, so it will just split the repair
> process in smaller yet sequential pieces of work (repair is started giving
> a start and end token). Overall, you should not expect improvements other
> than having less overstreaming and better chances of success if your
> cluster is dense.
>
> You can try to use incremental repair if you know what the caveats are and
> use a proper tool to orchestrate it, that would save you from repairing all
> 10TB each time.
> CASSANDRA-12580 might help too as Romain showed us :
> https://www.mail-archive.com/user@cassandra.apache.org/msg49344.html
>
> Cheers,
>
>
>
> On Wed, Oct 19, 2016 at 6:42 PM Kant Kodali  wrote:
>
> Another question on a same note would be what would be the fastest way to
> do repairs of size 10TB cluster ? Full repairs are taking days. So among
> repair parallel or repair sub range which is faster in the case of say
> adding a new node to the cluster?
>
> Sent from my iPhone
>
> On Oct 19, 2016, at 9:30 AM, Sean Bridges 
> wrote:
>
> Hey,
>
> We are upgrading from cassandra 2.1 to cassandra 2.2.
>
> With cassandra 2.1 we would periodically repair all nodes, using the -pr
> flag.
>
> With cassandra 2.2, the same repair takes a very long time, as cassandra
> does an anti compaction after the repair.  This anti compaction causes most
> (all?) the sstables to be rewritten.  Is there a way to do full repairs
> without continually anti compacting?  If we do a full repair on each node
> with the -pr flag, will subsequent full repairs also force anti compacting
> most (all?) sstables?
>
> Thanks,
>
> Sean
>
> --
> -
> Alexander Dejanovski
> France
> @alexanderdeja
>
> Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> --
-
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

Re: non incremental repairs with cassandra 2.2+

[Kant Kodali]

Also any suggestions on a tool to orchestrate the incremental repair? Like say 
most commonly used 

Sent from my iPhone

> On Oct 19, 2016, at 9:54 AM, Alexander Dejanovski  
> wrote:
> 
> Hi Kant,
> 
> subrange is a form of full repair, so it will just split the repair process 
> in smaller yet sequential pieces of work (repair is started giving a start 
> and end token). Overall, you should not expect improvements other than having 
> less overstreaming and better chances of success if your cluster is dense.
> 
> You can try to use incremental repair if you know what the caveats are and 
> use a proper tool to orchestrate it, that would save you from repairing all 
> 10TB each time.
> CASSANDRA-12580 might help too as Romain showed us : 
> https://www.mail-archive.com/user@cassandra.apache.org/msg49344.html
> 
> Cheers,
> 
> 
> 
> On Wed, Oct 19, 2016 at 6:42 PM Kant Kodali  wrote:
> Another question on a same note would be what would be the fastest way to do 
> repairs of size 10TB cluster ? Full repairs are taking days. So among repair 
> parallel or repair sub range which is faster in the case of say adding a new 
> node to the cluster?
> 
> Sent from my iPhone
> 
>> On Oct 19, 2016, at 9:30 AM, Sean Bridges  
>> wrote:
>> 
>> Hey,
>> 
>> We are upgrading from cassandra 2.1 to cassandra 2.2.  
>> 
>> With cassandra 2.1 we would periodically repair all nodes, using the -pr 
>> flag.  
>> 
>> With cassandra 2.2, the same repair takes a very long time, as cassandra 
>> does an anti compaction after the repair.  This anti compaction causes most 
>> (all?) the sstables to be rewritten.  Is there a way to do full repairs 
>> without continually anti compacting?  If we do a full repair on each node 
>> with the -pr flag, will subsequent full repairs also force anti 
>> compacting most (all?) sstables?  
>> 
>> Thanks,
>> 
>> Sean
> 
> -- 
> -
> Alexander Dejanovski
> France
> @alexanderdeja
> 
> Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com

Re: non incremental repairs with cassandra 2.2+

[Kant Kodali]

Thanks! How do I do an incremental repair when I add a new node?

Sent from my iPhone

> On Oct 19, 2016, at 9:54 AM, Alexander Dejanovski  
> wrote:
> 
> Hi Kant,
> 
> subrange is a form of full repair, so it will just split the repair process 
> in smaller yet sequential pieces of work (repair is started giving a start 
> and end token). Overall, you should not expect improvements other than having 
> less overstreaming and better chances of success if your cluster is dense.
> 
> You can try to use incremental repair if you know what the caveats are and 
> use a proper tool to orchestrate it, that would save you from repairing all 
> 10TB each time.
> CASSANDRA-12580 might help too as Romain showed us : 
> https://www.mail-archive.com/user@cassandra.apache.org/msg49344.html
> 
> Cheers,
> 
> 
> 
> On Wed, Oct 19, 2016 at 6:42 PM Kant Kodali  wrote:
> Another question on a same note would be what would be the fastest way to do 
> repairs of size 10TB cluster ? Full repairs are taking days. So among repair 
> parallel or repair sub range which is faster in the case of say adding a new 
> node to the cluster?
> 
> Sent from my iPhone
> 
>> On Oct 19, 2016, at 9:30 AM, Sean Bridges  
>> wrote:
>> 
>> Hey,
>> 
>> We are upgrading from cassandra 2.1 to cassandra 2.2.  
>> 
>> With cassandra 2.1 we would periodically repair all nodes, using the -pr 
>> flag.  
>> 
>> With cassandra 2.2, the same repair takes a very long time, as cassandra 
>> does an anti compaction after the repair.  This anti compaction causes most 
>> (all?) the sstables to be rewritten.  Is there a way to do full repairs 
>> without continually anti compacting?  If we do a full repair on each node 
>> with the -pr flag, will subsequent full repairs also force anti 
>> compacting most (all?) sstables?  
>> 
>> Thanks,
>> 
>> Sean
> 
> -- 
> -
> Alexander Dejanovski
> France
> @alexanderdeja
> 
> Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com

Re: non incremental repairs with cassandra 2.2+

[Alexander Dejanovski]

Hi Kant,

subrange is a form of full repair, so it will just split the repair process
in smaller yet sequential pieces of work (repair is started giving a start
and end token). Overall, you should not expect improvements other than
having less overstreaming and better chances of success if your cluster is
dense.

You can try to use incremental repair if you know what the caveats are and
use a proper tool to orchestrate it, that would save you from repairing all
10TB each time.
CASSANDRA-12580 might help too as Romain showed us :
https://www.mail-archive.com/user@cassandra.apache.org/msg49344.html

Cheers,



On Wed, Oct 19, 2016 at 6:42 PM Kant Kodali  wrote:

Another question on a same note would be what would be the fastest way to
do repairs of size 10TB cluster ? Full repairs are taking days. So among
repair parallel or repair sub range which is faster in the case of say
adding a new node to the cluster?

Sent from my iPhone

On Oct 19, 2016, at 9:30 AM, Sean Bridges 
wrote:

Hey,

We are upgrading from cassandra 2.1 to cassandra 2.2.

With cassandra 2.1 we would periodically repair all nodes, using the -pr
flag.

With cassandra 2.2, the same repair takes a very long time, as cassandra
does an anti compaction after the repair.  This anti compaction causes most
(all?) the sstables to be rewritten.  Is there a way to do full repairs
without continually anti compacting?  If we do a full repair on each node
with the -pr flag, will subsequent full repairs also force anti compacting
most (all?) sstables?

Thanks,

Sean

-- 
-
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

Re: non incremental repairs with cassandra 2.2+

[Kant Kodali]

Another question on a same note would be what would be the fastest way to do 
repairs of size 10TB cluster ? Full repairs are taking days. So among repair 
parallel or repair sub range which is faster in the case of say adding a new 
node to the cluster?

Sent from my iPhone

> On Oct 19, 2016, at 9:30 AM, Sean Bridges  
> wrote:
> 
> Hey,
> 
> We are upgrading from cassandra 2.1 to cassandra 2.2.  
> 
> With cassandra 2.1 we would periodically repair all nodes, using the -pr 
> flag.  
> 
> With cassandra 2.2, the same repair takes a very long time, as cassandra does 
> an anti compaction after the repair.  This anti compaction causes most (all?) 
> the sstables to be rewritten.  Is there a way to do full repairs without 
> continually anti compacting?  If we do a full repair on each node with the 
> -pr flag, will subsequent full repairs also force anti compacting most (all?) 
> sstables?  
> 
> Thanks,
> 
> Sean

Re: non incremental repairs with cassandra 2.2+

[Alexander Dejanovski]

Hi Sean,

you should be able to do that by running subrange repairs, which is the
only type of repair that wouldn't trigger anticompaction AFAIK.
Beware that now you will have sstables marked as repaired and others marked
as unrepaired, which will never be compacted together.
You might want to flag all sstables as unrepaired before moving on, if you
do not intend to switch to incremental repair for now.

Cheers,

On Wed, Oct 19, 2016 at 6:31 PM Sean Bridges 
wrote:

> Hey,
>
> We are upgrading from cassandra 2.1 to cassandra 2.2.
>
> With cassandra 2.1 we would periodically repair all nodes, using the -pr
> flag.
>
> With cassandra 2.2, the same repair takes a very long time, as cassandra
> does an anti compaction after the repair.  This anti compaction causes most
> (all?) the sstables to be rewritten.  Is there a way to do full repairs
> without continually anti compacting?  If we do a full repair on each node
> with the -pr flag, will subsequent full repairs also force anti compacting
> most (all?) sstables?
>
> Thanks,
>
> Sean
>
-- 
-
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com