Re: User Defined Compaction Issue

[shalom sagges]

Awesome explanation :-)

Thanks a lot!

On Tue, Sep 26, 2017 at 3:40 PM, Jeff Jirsa  wrote:

> Write row A, flush into sstable 1
> Delete row A, flush the tombstone into sstable 2
>
> The tombstone in sstable 2 can’t be removed until row A in sstable 1 gets
> removed. If you just keep recompacting sstable 2 by itself, the row in
> sstable A remains on disk.
>
>
>
> --
> Jeff Jirsa
>
>
> On Sep 26, 2017, at 2:01 AM, shalom sagges  wrote:
>
> Thanks Jeff!
>
> I'll try that.
> I'm not sure I understand how the tombstones are covering data in another
> file. Do you have a small example perhaps?
>
> Thanks again!
>
> On Tue, Sep 26, 2017 at 1:38 AM, Jeff Jirsa  wrote:
>
>> The problem is likely that your sstables overlap - your 91% droppable
>> tombstones are probably covering data in another file. Until that data is
>> removed, those tombstones cant be dropped.
>>
>> This is why a full major compaction often works better for simply
>> reclaiming disk space (though it takes a lot more disk as it runs and has
>> other drawbacks) - it joins the shadowed data with the tombstones into a
>> single file. You can sorta fake that by using user-defined compaction with
>> multiple sstables to try to compact away the shadowed data - pick 2 or more
>> -Data files and compact them together. Works best if you can be sure which
>> data is overlapping, but short of that you'll probably want to pick data
>> with approximately the same (or older) calendar timestamps.
>>
>>
>>
>> On Mon, Sep 25, 2017 at 11:10 AM, shalom sagges 
>> wrote:
>>
>>> Hi Everyone,
>>>
>>> I'm running into an issue I can't seem to Solve.
>>>
>>> I execute force compaction in order to reclaim back storage.
>>> Everything was working fine for a time, but after a while I found that
>>> tombstones aren't being removed any longer.
>>>
>>> For example, I've compacted the following SSTable:
>>> *21G *Sep 19 10:18 file-jb-69103-Data.db
>>> Estimated droppable tombstones:
>>>
>>> *0.9115601492568154*
>>> I ran it with jmxterm and the compaction started and completed
>>> successfully.
>>>  $>run -b org.apache.cassandra.db:type=CompactionManager
>>> forceUserDefinedCompaction file-jb-69103-Data.db
>>>  #calling operation forceUserDefinedCompaction of mbean
>>> org.apache.cassandra.db:type=CompactionManager
>>>  #operation returns:
>>> null
>>>
>>>
>>> A new SStable was created, but no tombstones were removed.
>>>  *21G *Sep 25 12:16 file-jb-71319-Data.db
>>>  Estimated droppable tombstones:
>>>
>>> * 0.9115601492568154*
>>> I've tried running it from jconsole as well, but the outcome was the
>>> same.
>>>
>>> Is anyone familiar with this issue?
>>> Btw, this cluster is on version 2.0.14 .
>>>
>>>
>>> Thanks!
>>> Shalom
>>>
>>>
>>
>

Re: User Defined Compaction Issue

[Jeff Jirsa]

Write row A, flush into sstable 1
Delete row A, flush the tombstone into sstable 2

The tombstone in sstable 2 can’t be removed until row A in sstable 1 gets 
removed. If you just keep recompacting sstable 2 by itself, the row in sstable 
A remains on disk.



-- 
Jeff Jirsa


> On Sep 26, 2017, at 2:01 AM, shalom sagges  wrote:
> 
> Thanks Jeff!
> 
> I'll try that. 
> I'm not sure I understand how the tombstones are covering data in another 
> file. Do you have a small example perhaps?
> 
> Thanks again!
> 
>> On Tue, Sep 26, 2017 at 1:38 AM, Jeff Jirsa  wrote:
>> The problem is likely that your sstables overlap - your 91% droppable 
>> tombstones are probably covering data in another file. Until that data is 
>> removed, those tombstones cant be dropped.
>> 
>> This is why a full major compaction often works better for simply reclaiming 
>> disk space (though it takes a lot more disk as it runs and has other 
>> drawbacks) - it joins the shadowed data with the tombstones into a single 
>> file. You can sorta fake that by using user-defined compaction with multiple 
>> sstables to try to compact away the shadowed data - pick 2 or more -Data 
>> files and compact them together. Works best if you can be sure which data is 
>> overlapping, but short of that you'll probably want to pick data with 
>> approximately the same (or older) calendar timestamps.
>> 
>> 
>> 
>>> On Mon, Sep 25, 2017 at 11:10 AM, shalom sagges  
>>> wrote:
>>> Hi Everyone, 
>>> 
>>> I'm running into an issue I can't seem to Solve. 
>>> 
>>> I execute force compaction in order to reclaim back storage. 
>>> Everything was working fine for a time, but after a while I found that 
>>> tombstones aren't being removed any longer. 
>>> 
>>> For example, I've compacted the following SSTable:
>>> 21G Sep 19 10:18 file-jb-69103-Data.db
>>> Estimated droppable tombstones: 0.9115601492568154
>>> 
>>> I ran it with jmxterm and the compaction started and completed 
>>> successfully. 
>>>  $>run -b org.apache.cassandra.db:type=CompactionManager 
>>> forceUserDefinedCompaction file-jb-69103-Data.db
>>>  #calling operation forceUserDefinedCompaction of mbean 
>>> org.apache.cassandra.db:type=CompactionManager
>>>  #operation returns: 
>>> null
>>> 
>>> 
>>> A new SStable was created, but no tombstones were removed. 
>>>  21G Sep 25 12:16 file-jb-71319-Data.db
>>>  Estimated droppable tombstones: 0.9115601492568154
>>> 
>>> I've tried running it from jconsole as well, but the outcome was the same. 
>>> 
>>> Is anyone familiar with this issue? 
>>> Btw, this cluster is on version 2.0.14 . 
>>> 
>>> 
>>> Thanks!
>>> Shalom
>>> 
>> 
> 

Re: User Defined Compaction Issue

[shalom sagges]

Thanks Jeff!

I'll try that.
I'm not sure I understand how the tombstones are covering data in another
file. Do you have a small example perhaps?

Thanks again!

On Tue, Sep 26, 2017 at 1:38 AM, Jeff Jirsa  wrote:

> The problem is likely that your sstables overlap - your 91% droppable
> tombstones are probably covering data in another file. Until that data is
> removed, those tombstones cant be dropped.
>
> This is why a full major compaction often works better for simply
> reclaiming disk space (though it takes a lot more disk as it runs and has
> other drawbacks) - it joins the shadowed data with the tombstones into a
> single file. You can sorta fake that by using user-defined compaction with
> multiple sstables to try to compact away the shadowed data - pick 2 or more
> -Data files and compact them together. Works best if you can be sure which
> data is overlapping, but short of that you'll probably want to pick data
> with approximately the same (or older) calendar timestamps.
>
>
>
> On Mon, Sep 25, 2017 at 11:10 AM, shalom sagges 
> wrote:
>
>> Hi Everyone,
>>
>> I'm running into an issue I can't seem to Solve.
>>
>> I execute force compaction in order to reclaim back storage.
>> Everything was working fine for a time, but after a while I found that
>> tombstones aren't being removed any longer.
>>
>> For example, I've compacted the following SSTable:
>> *21G *Sep 19 10:18 file-jb-69103-Data.db
>> Estimated droppable tombstones:
>>
>> *0.9115601492568154*
>> I ran it with jmxterm and the compaction started and completed
>> successfully.
>>  $>run -b org.apache.cassandra.db:type=CompactionManager
>> forceUserDefinedCompaction file-jb-69103-Data.db
>>  #calling operation forceUserDefinedCompaction of mbean
>> org.apache.cassandra.db:type=CompactionManager
>>  #operation returns:
>> null
>>
>>
>> A new SStable was created, but no tombstones were removed.
>>  *21G *Sep 25 12:16 file-jb-71319-Data.db
>>  Estimated droppable tombstones:
>>
>> * 0.9115601492568154*
>> I've tried running it from jconsole as well, but the outcome was the
>> same.
>>
>> Is anyone familiar with this issue?
>> Btw, this cluster is on version 2.0.14 .
>>
>>
>> Thanks!
>> Shalom
>>
>>
>

Re: User Defined Compaction Issue

[Jeff Jirsa]

The problem is likely that your sstables overlap - your 91% droppable
tombstones are probably covering data in another file. Until that data is
removed, those tombstones cant be dropped.

This is why a full major compaction often works better for simply
reclaiming disk space (though it takes a lot more disk as it runs and has
other drawbacks) - it joins the shadowed data with the tombstones into a
single file. You can sorta fake that by using user-defined compaction with
multiple sstables to try to compact away the shadowed data - pick 2 or more
-Data files and compact them together. Works best if you can be sure which
data is overlapping, but short of that you'll probably want to pick data
with approximately the same (or older) calendar timestamps.



On Mon, Sep 25, 2017 at 11:10 AM, shalom sagges 
wrote:

> Hi Everyone,
>
> I'm running into an issue I can't seem to Solve.
>
> I execute force compaction in order to reclaim back storage.
> Everything was working fine for a time, but after a while I found that
> tombstones aren't being removed any longer.
>
> For example, I've compacted the following SSTable:
> *21G *Sep 19 10:18 file-jb-69103-Data.db
> Estimated droppable tombstones:
>
> *0.9115601492568154*
> I ran it with jmxterm and the compaction started and completed
> successfully.
>  $>run -b org.apache.cassandra.db:type=CompactionManager
> forceUserDefinedCompaction file-jb-69103-Data.db
>  #calling operation forceUserDefinedCompaction of mbean
> org.apache.cassandra.db:type=CompactionManager
>  #operation returns:
> null
>
>
> A new SStable was created, but no tombstones were removed.
>  *21G *Sep 25 12:16 file-jb-71319-Data.db
>  Estimated droppable tombstones:
>
> * 0.9115601492568154*
> I've tried running it from jconsole as well, but the outcome was the same.
>
> Is anyone familiar with this issue?
> Btw, this cluster is on version 2.0.14 .
>
>
> Thanks!
> Shalom
>
>