Re: Anticompaction causing significant increase in disk usage

[Martin Mačura]

Hi Alain,
thank you for your response.

I'm using incremental repair. I'm afraid subrange repair is not a
viable alternative, because it's very slow - takes over a week to
complete.

I've found at least a partial solution - specifying '-local' or '-dc'
parameter will also disable anticompaction, but the repair will skip
SSTables that are already marked as repaired. Our data is about 50%
repaired, so this significantly reduces repair time.

What if I ran 'sstablerepairedset --really-set --is-repaired' on every
table that was repaired by a subrange repair?  Would it prevent these
tables from being anticompacted, and allow us to use incremental
repair again?


Regards,

Martin
On Wed, Sep 12, 2018 at 1:31 PM Alain RODRIGUEZ  wrote:
>
> Hello Martin,
>
> How do you perform the repairs?
>
> Are you using incremental repairs or full repairs but without subranges? Alex 
> described issues related to these repairs here: 
> http://thelastpickle.com/blog/2017/12/14/should-you-use-incremental-repair.html.
>
> tl;dr:
>
>> The only way to perform repair without anticompaction in “modern” versions 
>> of Apache Cassandra is subrange repair, which fully skips anticompaction. To 
>> perform a subrange repair correctly, you have three options :
>> - Compute valid token subranges yourself and script repairs accordingly
>> - Use the Cassandra range repair script which performs subrange repair
>> - Use Cassandra Reaper, which also performs subrange repair
>
>
> If you can prevent anti-compaction, disk space growth should be more 
> predictable.
>
> There might be more solutions now out there, C* should also soon be shipped 
> with a side-car it's being actively discussed. Finally, Incremental repairs 
> will receive important fixes in Cassandra 4.0, Alex also wrote about this too 
> (yes, this guy loves repairs ¯\_(ツ)_/¯) 
> http://thelastpickle.com/blog/2018/09/10/incremental-repair-improvements-in-cassandra-4.html
>
> I believe (and hope) this information is relevant to help you fix this issue.
>
> C*heers,
> ---
> Alain Rodriguez - @arodream - al...@thelastpickle.com
> France / Spain
>
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> Le mer. 12 sept. 2018 à 10:14, Martin Mačura  a écrit :
>>
>> Hi,
>> we're on cassandra 3.11.2 . During an anticompaction after repair,
>> TotalDiskSpaceUsed value of one table gradually went from 700GB to
>> 1180GB, and then suddenly jumped back to 700GB.  This happened on all
>> nodes involved in the repair. There was no change in PercentRepaired
>> during or after this process. SSTable count is currently 857, with a
>> peak of 2460 during the repair.
>>
>> Table is using TWCS with 1-day time window.  Most daily SSTables are
>> around 1 GB but the oldest one is 156 GB - caused by a major
>> compaction.
>>
>> system.log.6.zip:INFO  [CompactionExecutor:9923] 2018-09-10
>> 15:29:54,238 CompactionManager.java:649 - [repair
>> #88c36e30-b4cb-11e8-bebe-cd3efd73ed33] Starting anticompaction for ...
>> on 519 [...]  SSTables
>> ...
>> system.log:INFO  [CompactionExecutor:9923] 2018-09-12 00:29:39,262
>> CompactionManager.java:1524 - Anticompaction completed successfully,
>> anticompacted from 0 to 518 sstable(s).
>>
>> What could be the cause of the temporary increase, and how can we
>> prevent it?  We are concerned about running out of disk space soon.
>>
>> Thanks for any help
>>
>> Martin
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Anticompaction causing significant increase in disk usage

[Alain RODRIGUEZ]

Hello Martin,

How do you perform the repairs?

Are you using incremental repairs or full repairs but without subranges?
Alex described issues related to these repairs here:
http://thelastpickle.com/blog/2017/12/14/should-you-use-incremental-repair.html
.

*tl;dr: *

The only way to perform repair without anticompaction in “modern” versions
> of Apache Cassandra is subrange repair, which fully skips anticompaction.
> To perform a subrange repair correctly, you have three options :
> - Compute valid token subranges yourself and script repairs accordingly
> - Use the Cassandra range repair script which performs subrange repair
> - Use Cassandra Reaper, which also performs subrange repair


If you can prevent anti-compaction, disk space growth should be more
predictable.

There might be more solutions now out there, C* should also soon be shipped
with a side-car it's being actively discussed. Finally, Incremental repairs
will receive important fixes in Cassandra 4.0, Alex also wrote about this
too (yes, this guy loves repairs ¯\_(ツ)_/¯)
http://thelastpickle.com/blog/2018/09/10/incremental-repair-improvements-in-cassandra-4.html

I believe (and hope) this information is relevant to help you fix this
issue.

C*heers,
---
Alain Rodriguez - @arodream - al...@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

Le mer. 12 sept. 2018 à 10:14, Martin Mačura  a écrit :

> Hi,
> we're on cassandra 3.11.2 . During an anticompaction after repair,
> TotalDiskSpaceUsed value of one table gradually went from 700GB to
> 1180GB, and then suddenly jumped back to 700GB.  This happened on all
> nodes involved in the repair. There was no change in PercentRepaired
> during or after this process. SSTable count is currently 857, with a
> peak of 2460 during the repair.
>
> Table is using TWCS with 1-day time window.  Most daily SSTables are
> around 1 GB but the oldest one is 156 GB - caused by a major
> compaction.
>
> system.log.6.zip:INFO  [CompactionExecutor:9923] 2018-09-10
> 15:29:54,238 CompactionManager.java:649 - [repair
> #88c36e30-b4cb-11e8-bebe-cd3efd73ed33] Starting anticompaction for ...
> on 519 [...]  SSTables
> ...
> system.log:INFO  [CompactionExecutor:9923] 2018-09-12 00:29:39,262
> CompactionManager.java:1524 - Anticompaction completed successfully,
> anticompacted from 0 to 518 sstable(s).
>
> What could be the cause of the temporary increase, and how can we
> prevent it?  We are concerned about running out of disk space soon.
>
> Thanks for any help
>
> Martin
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Anticompaction causing significant increase in disk usage

[Martin Mačura]

Hi,
we're on cassandra 3.11.2 . During an anticompaction after repair,
TotalDiskSpaceUsed value of one table gradually went from 700GB to
1180GB, and then suddenly jumped back to 700GB.  This happened on all
nodes involved in the repair. There was no change in PercentRepaired
during or after this process. SSTable count is currently 857, with a
peak of 2460 during the repair.

Table is using TWCS with 1-day time window.  Most daily SSTables are
around 1 GB but the oldest one is 156 GB - caused by a major
compaction.

system.log.6.zip:INFO  [CompactionExecutor:9923] 2018-09-10
15:29:54,238 CompactionManager.java:649 - [repair
#88c36e30-b4cb-11e8-bebe-cd3efd73ed33] Starting anticompaction for ...
on 519 [...]  SSTables
...
system.log:INFO  [CompactionExecutor:9923] 2018-09-12 00:29:39,262
CompactionManager.java:1524 - Anticompaction completed successfully,
anticompacted from 0 to 518 sstable(s).

What could be the cause of the temporary increase, and how can we
prevent it?  We are concerned about running out of disk space soon.

Thanks for any help

Martin

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Anticompaction

[Vlad]

Hi,
I run repair, then I see that anticompaction started on all nodes.Does it mean 
that all data is already repaired. Actually I increased RF, so can I already 
use database?
Thanks.

Re: Anticompaction Question

[Rajath Subramanyam]

Hi Anubhav,

According to the Datastax documentation here
, after the
anti-compaction process splits the ranges to repaired and un-repaired
SSTables, they are compacted in their own separate pools.

Regards,
Rajath




Rajath Subramanyam


On Tue, Oct 25, 2016 at 4:56 PM, Anubhav Kale 
wrote:

> Hello,
>
>
>
> If incremental repairs are enabled, there is logic in every compaction
> strategy to make sure not to mix repaired and unrepaired SS Tables.
>
>
>
> Does this mean if some SS Table files are repaired and some aren’t and
> incremental repairs don’t work reliably, the unrepaired tables will
> *never* get compacted into repaired ones ?
>
>
>
> Thanks !
>

Anticompaction Question

[Anubhav Kale]

Hello,

If incremental repairs are enabled, there is logic in every compaction strategy 
to make sure not to mix repaired and unrepaired SS Tables.

Does this mean if some SS Table files are repaired and some aren't and 
incremental repairs don't work reliably, the unrepaired tables will never get 
compacted into repaired ones ?

Thanks !

What are the repercussions of a restart during anticompaction?

[Bryan Cheng]

Hey list,

Tried to find an answer to this elsewhere, but turned up nothing.

We ran our first incremental repair after a large dc migration two days
ago; the cluster had been running full repairs prior to this during the
migration. Our nodes are currently going through anticompaction, as
expected.

However, two days later, there is little to no apparent progress on this
process. The compaction count does increase, in bursts, but compactionstats
hangs with no response. We're seeing our disk space footprint grow steadily
as well. The number of sstables on disk is reaching high levels.

In the past, when our compactions seem to hang, a restart seems to move
things along; at the very least, it seems to allow JMX to respond. However,
I'm not sure of the repercussions of a restart during anticompaction.

Given my understanding of anticompaction, my expectation would be that the
sstables that had been split and marked repaired would remain that way, the
ones that had not yet been split would be left as unrepaired and some
ranges would probably be re-repaired on the next incremental repair, and
the machine would do standard compaction among the two sets (repaired vs
unrepaired). In other words, we wouldn't lose any progress in incremental
repair + anticompaction, but some repaired data would get re-repaired. Does
this seem reasonable?

Should I just let this anticompaction run its course? We did the migration
procedure (marking sstables as repaired) awhile ago, but did a full repair
again after that before we decommissioned our old dc.

Any guidance would be appreciated! Thanks,

Bryan