Re: old big tombstone data file occupy much disk space

[Shalom Sagges]

Thanks! :-)



On Mon, Sep 4, 2017 at 2:56 PM, Nicolas Guyomar 
wrote:

> Wrong copy/paste !
>
> Looking at the code, it should do nothing :
>
>  // look up the sstables now that we're on the compaction executor, so we
> don't try to re-compact
>  // something that was already being compacted earlier.
>
> On 4 September 2017 at 13:54, Nicolas Guyomar 
> wrote:
>
>> You'll get the WARN "Will not compact {}: it is not an active sstable"
>>  :)
>>
>> On 4 September 2017 at 12:07, Shalom Sagges 
>> wrote:
>>
>>> By the way, does anyone know what happens if I run a user defined
>>> compaction on an sstable that's already in compaction?
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Sun, Sep 3, 2017 at 2:55 PM, Shalom Sagges 
>>> wrote:
>>>
 Try this blog by The Last Pickle:

 http://thelastpickle.com/blog/2016/10/18/user-defined-compaction.html






 Shalom Sagges
 DBA
 
   We
 Create Meaningful Connections



 On Sat, Sep 2, 2017 at 8:34 PM, Jeff Jirsa  wrote:

> If you're on 3.0 (3.0.6 or 3.0.8 or newer I don't remember which),
> TWCS was designed for ttl-only time series use cases
>
> Alternatively, if you have IO to spare, you may find LCS works as well
> (it'll cause quite a bit more compaction, but a much higher chance to
> compact away tombstones)
>
> There are also tombstone focused sub properties to more aggressively
> compact sstables that have a lot of tombstones - check the docs for
> "unchecked tombstone compaction" and "tombstone threshold" - enabling 
> those
> will enable more aggressive automatic single-sstable compactions
>
> --
> Jeff Jirsa
>
>
> On Sep 2, 2017, at 7:10 AM, qf zhou  wrote:
>
>
> Yes, your are right. I am using STCS compaction strategy with some
> kind of timeseries model. Too much disk space has been occupied.
>
> What should I  do to stop  the  disk full ?
>
>  I only want to keep 100 days data most recently,  so I set 
> default_time_to_live
> = 864(100 days ).
>
> I know I need to do something to stop the disk space cost, but I
> really don’t know how to do it.
>
>
> Here is the strategy of the big data table :
>
> AND compaction = {'class': 'org.apache.cassandra.db.compa
> ction.SizeTieredCompactionStrategy', 'max_threshold': '32',
> 'min_threshold': '12', 'tombstone_threshold': '0.1',
> 'unchecked_tombstone_compaction': 'true'}
> AND compression = {'chunk_length_in_kb': '64', 'class': '
> org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 864
> AND gc_grace_seconds = 432000
>
>
>
> 在 2017年9月2日，下午7:34，Nicolas Guyomar  写道：
>
> your are using STCS compaction strategy with some kind of timeseries
> model, and you are going to end up with yor disk full!
>
>
>

>>>
>>> This message may contain confidential and/or privileged information.
>>> If you are not the addressee or authorized to receive this on behalf of
>>> the addressee you must not use, copy, disclose or take action based on this
>>> message or any information herein.
>>> If you have received this message in error, please advise the sender
>>> immediately by reply email and delete this message. Thank you.
>>>
>>
>>
>

-- 
This message may contain confidential and/or privileged information. 
If you are not the addressee or authorized to receive this on behalf of the 
addressee you must not use, copy, disclose or take action based on this 
message or any information herein. 
If you have received this message in error, please advise the sender 
immediately by reply email and delete this message. Thank you.

Re: old big tombstone data file occupy much disk space

[Nicolas Guyomar]

Wrong copy/paste !

Looking at the code, it should do nothing :

 // look up the sstables now that we're on the compaction executor, so we
don't try to re-compact
 // something that was already being compacted earlier.

On 4 September 2017 at 13:54, Nicolas Guyomar 
wrote:

> You'll get the WARN "Will not compact {}: it is not an active sstable"  :)
>
> On 4 September 2017 at 12:07, Shalom Sagges 
> wrote:
>
>> By the way, does anyone know what happens if I run a user defined
>> compaction on an sstable that's already in compaction?
>>
>>
>>
>>
>>
>>
>> On Sun, Sep 3, 2017 at 2:55 PM, Shalom Sagges 
>> wrote:
>>
>>> Try this blog by The Last Pickle:
>>>
>>> http://thelastpickle.com/blog/2016/10/18/user-defined-compaction.html
>>>
>>>
>>>
>>>
>>>
>>>
>>> Shalom Sagges
>>> DBA
>>>  
>>>  We Create Meaningful Connections
>>>
>>>
>>>
>>> On Sat, Sep 2, 2017 at 8:34 PM, Jeff Jirsa  wrote:
>>>
 If you're on 3.0 (3.0.6 or 3.0.8 or newer I don't remember which), TWCS
 was designed for ttl-only time series use cases

 Alternatively, if you have IO to spare, you may find LCS works as well
 (it'll cause quite a bit more compaction, but a much higher chance to
 compact away tombstones)

 There are also tombstone focused sub properties to more aggressively
 compact sstables that have a lot of tombstones - check the docs for
 "unchecked tombstone compaction" and "tombstone threshold" - enabling those
 will enable more aggressive automatic single-sstable compactions

 --
 Jeff Jirsa


 On Sep 2, 2017, at 7:10 AM, qf zhou  wrote:


 Yes, your are right. I am using STCS compaction strategy with some kind
 of timeseries model. Too much disk space has been occupied.

 What should I  do to stop  the  disk full ?

  I only want to keep 100 days data most recently,  so I set 
 default_time_to_live
 = 864(100 days ).

 I know I need to do something to stop the disk space cost, but I really
 don’t know how to do it.


 Here is the strategy of the big data table :

 AND compaction = {'class': 'org.apache.cassandra.db.compa
 ction.SizeTieredCompactionStrategy', 'max_threshold': '32',
 'min_threshold': '12', 'tombstone_threshold': '0.1',
 'unchecked_tombstone_compaction': 'true'}
 AND compression = {'chunk_length_in_kb': '64', 'class': '
 org.apache.cassandra.io.compress.LZ4Compressor'}
 AND crc_check_chance = 1.0
 AND dclocal_read_repair_chance = 0.1
 AND default_time_to_live = 864
 AND gc_grace_seconds = 432000



 在 2017年9月2日，下午7:34，Nicolas Guyomar  写道：

 your are using STCS compaction strategy with some kind of timeseries
 model, and you are going to end up with yor disk full!



>>>
>>
>> This message may contain confidential and/or privileged information.
>> If you are not the addressee or authorized to receive this on behalf of
>> the addressee you must not use, copy, disclose or take action based on this
>> message or any information herein.
>> If you have received this message in error, please advise the sender
>> immediately by reply email and delete this message. Thank you.
>>
>
>

Re: old big tombstone data file occupy much disk space

[Nicolas Guyomar]

You'll get the WARN "Will not compact {}: it is not an active sstable"  :)

On 4 September 2017 at 12:07, Shalom Sagges  wrote:

> By the way, does anyone know what happens if I run a user defined
> compaction on an sstable that's already in compaction?
>
>
>
>
>
>
> On Sun, Sep 3, 2017 at 2:55 PM, Shalom Sagges 
> wrote:
>
>> Try this blog by The Last Pickle:
>>
>> http://thelastpickle.com/blog/2016/10/18/user-defined-compaction.html
>>
>>
>>
>>
>>
>>
>> Shalom Sagges
>> DBA
>>  
>>  We Create Meaningful Connections
>>
>>
>>
>> On Sat, Sep 2, 2017 at 8:34 PM, Jeff Jirsa  wrote:
>>
>>> If you're on 3.0 (3.0.6 or 3.0.8 or newer I don't remember which), TWCS
>>> was designed for ttl-only time series use cases
>>>
>>> Alternatively, if you have IO to spare, you may find LCS works as well
>>> (it'll cause quite a bit more compaction, but a much higher chance to
>>> compact away tombstones)
>>>
>>> There are also tombstone focused sub properties to more aggressively
>>> compact sstables that have a lot of tombstones - check the docs for
>>> "unchecked tombstone compaction" and "tombstone threshold" - enabling those
>>> will enable more aggressive automatic single-sstable compactions
>>>
>>> --
>>> Jeff Jirsa
>>>
>>>
>>> On Sep 2, 2017, at 7:10 AM, qf zhou  wrote:
>>>
>>>
>>> Yes, your are right. I am using STCS compaction strategy with some kind
>>> of timeseries model. Too much disk space has been occupied.
>>>
>>> What should I  do to stop  the  disk full ?
>>>
>>>  I only want to keep 100 days data most recently,  so I set 
>>> default_time_to_live
>>> = 864(100 days ).
>>>
>>> I know I need to do something to stop the disk space cost, but I really
>>> don’t know how to do it.
>>>
>>>
>>> Here is the strategy of the big data table :
>>>
>>> AND compaction = {'class': 'org.apache.cassandra.db.compa
>>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32',
>>> 'min_threshold': '12', 'tombstone_threshold': '0.1',
>>> 'unchecked_tombstone_compaction': 'true'}
>>> AND compression = {'chunk_length_in_kb': '64', 'class': '
>>> org.apache.cassandra.io.compress.LZ4Compressor'}
>>> AND crc_check_chance = 1.0
>>> AND dclocal_read_repair_chance = 0.1
>>> AND default_time_to_live = 864
>>> AND gc_grace_seconds = 432000
>>>
>>>
>>>
>>> 在 2017年9月2日，下午7:34，Nicolas Guyomar  写道：
>>>
>>> your are using STCS compaction strategy with some kind of timeseries
>>> model, and you are going to end up with yor disk full!
>>>
>>>
>>>
>>
>
> This message may contain confidential and/or privileged information.
> If you are not the addressee or authorized to receive this on behalf of
> the addressee you must not use, copy, disclose or take action based on this
> message or any information herein.
> If you have received this message in error, please advise the sender
> immediately by reply email and delete this message. Thank you.
>

Re: old big tombstone data file occupy much disk space

[Shalom Sagges]

By the way, does anyone know what happens if I run a user defined
compaction on an sstable that's already in compaction?





On Sun, Sep 3, 2017 at 2:55 PM, Shalom Sagges 
wrote:

> Try this blog by The Last Pickle:
>
> http://thelastpickle.com/blog/2016/10/18/user-defined-compaction.html
>
>
>
>
>
>
> Shalom Sagges
> DBA
>  
>  We Create Meaningful Connections
>
>
>
> On Sat, Sep 2, 2017 at 8:34 PM, Jeff Jirsa  wrote:
>
>> If you're on 3.0 (3.0.6 or 3.0.8 or newer I don't remember which), TWCS
>> was designed for ttl-only time series use cases
>>
>> Alternatively, if you have IO to spare, you may find LCS works as well
>> (it'll cause quite a bit more compaction, but a much higher chance to
>> compact away tombstones)
>>
>> There are also tombstone focused sub properties to more aggressively
>> compact sstables that have a lot of tombstones - check the docs for
>> "unchecked tombstone compaction" and "tombstone threshold" - enabling those
>> will enable more aggressive automatic single-sstable compactions
>>
>> --
>> Jeff Jirsa
>>
>>
>> On Sep 2, 2017, at 7:10 AM, qf zhou  wrote:
>>
>>
>> Yes, your are right. I am using STCS compaction strategy with some kind
>> of timeseries model. Too much disk space has been occupied.
>>
>> What should I  do to stop  the  disk full ?
>>
>>  I only want to keep 100 days data most recently,  so I set 
>> default_time_to_live
>> = 864(100 days ).
>>
>> I know I need to do something to stop the disk space cost, but I really
>> don’t know how to do it.
>>
>>
>> Here is the strategy of the big data table :
>>
>> AND compaction = {'class': 'org.apache.cassandra.db.compa
>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32',
>> 'min_threshold': '12', 'tombstone_threshold': '0.1',
>> 'unchecked_tombstone_compaction': 'true'}
>> AND compression = {'chunk_length_in_kb': '64', 'class': '
>> org.apache.cassandra.io.compress.LZ4Compressor'}
>> AND crc_check_chance = 1.0
>> AND dclocal_read_repair_chance = 0.1
>> AND default_time_to_live = 864
>> AND gc_grace_seconds = 432000
>>
>>
>>
>> 在 2017年9月2日，下午7:34，Nicolas Guyomar  写道：
>>
>> your are using STCS compaction strategy with some kind of timeseries
>> model, and you are going to end up with yor disk full!
>>
>>
>>
>

-- 
This message may contain confidential and/or privileged information. 
If you are not the addressee or authorized to receive this on behalf of the 
addressee you must not use, copy, disclose or take action based on this 
message or any information herein. 
If you have received this message in error, please advise the sender 
immediately by reply email and delete this message. Thank you.

Re: old big tombstone data file occupy much disk space

[Shalom Sagges]

Try this blog by The Last Pickle:

http://thelastpickle.com/blog/2016/10/18/user-defined-compaction.html






Shalom Sagges
DBA
 
 We Create Meaningful Connections



On Sat, Sep 2, 2017 at 8:34 PM, Jeff Jirsa  wrote:

> If you're on 3.0 (3.0.6 or 3.0.8 or newer I don't remember which), TWCS
> was designed for ttl-only time series use cases
>
> Alternatively, if you have IO to spare, you may find LCS works as well
> (it'll cause quite a bit more compaction, but a much higher chance to
> compact away tombstones)
>
> There are also tombstone focused sub properties to more aggressively
> compact sstables that have a lot of tombstones - check the docs for
> "unchecked tombstone compaction" and "tombstone threshold" - enabling those
> will enable more aggressive automatic single-sstable compactions
>
> --
> Jeff Jirsa
>
>
> On Sep 2, 2017, at 7:10 AM, qf zhou  wrote:
>
>
> Yes, your are right. I am using STCS compaction strategy with some kind of
> timeseries model. Too much disk space has been occupied.
>
> What should I  do to stop  the  disk full ?
>
>  I only want to keep 100 days data most recently,  so I set 
> default_time_to_live
> = 864(100 days ).
>
> I know I need to do something to stop the disk space cost, but I really
> don’t know how to do it.
>
>
> Here is the strategy of the big data table :
>
> AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '12', 'tombstone_threshold': '0.1',
> 'unchecked_tombstone_compaction': 'true'}
> AND compression = {'chunk_length_in_kb': '64', 'class': '
> org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 864
> AND gc_grace_seconds = 432000
>
>
>
> 在 2017年9月2日，下午7:34，Nicolas Guyomar  写道：
>
> your are using STCS compaction strategy with some kind of timeseries
> model, and you are going to end up with yor disk full!
>
>
>

-- 
This message may contain confidential and/or privileged information. 
If you are not the addressee or authorized to receive this on behalf of the 
addressee you must not use, copy, disclose or take action based on this 
message or any information herein. 
If you have received this message in error, please advise the sender 
immediately by reply email and delete this message. Thank you.

Re: old big tombstone data file occupy much disk space

[Jeff Jirsa]

If you're on 3.0 (3.0.6 or 3.0.8 or newer I don't remember which), TWCS was 
designed for ttl-only time series use cases

Alternatively, if you have IO to spare, you may find LCS works as well (it'll 
cause quite a bit more compaction, but a much higher chance to compact away 
tombstones)

There are also tombstone focused sub properties to more aggressively compact 
sstables that have a lot of tombstones - check the docs for "unchecked 
tombstone compaction" and "tombstone threshold" - enabling those will enable 
more aggressive automatic single-sstable compactions 

-- 
Jeff Jirsa


> On Sep 2, 2017, at 7:10 AM, qf zhou  wrote:
> 
> 
> Yes, your are right. I am using STCS compaction strategy with some kind of 
> timeseries model. Too much disk space has been occupied.
> 
> What should I  do to stop  the  disk full ? 
> 
>  I only want to keep 100 days data most recently,  so I set 
> default_time_to_live = 864(100 days ).
> 
> I know I need to do something to stop the disk space cost, but I really don’t 
> know how to do it.
> 
> 
> Here is the strategy of the big data table :
> 
> AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 
> 'max_threshold': '32', 'min_threshold': '12', 'tombstone_threshold': '0.1', 
> 'unchecked_tombstone_compaction': 'true'}
> AND compression = {'chunk_length_in_kb': '64', 'class': 
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 864
> AND gc_grace_seconds = 432000
> 
> 
> 
>> 在 2017年9月2日，下午7:34，Nicolas Guyomar  写道：
>> 
>> your are using STCS compaction strategy with some kind of timeseries model, 
>> and you are going to end up with yor disk full!
> 

Re: old big tombstone data file occupy much disk space

[qf zhou]

Yes, your are right. I am using STCS compaction strategy with some kind of 
timeseries model. Too much disk space has been occupied.

What should I  do to stop  the  disk full ? 

 I only want to keep 100 days data most recently,  so I set 
default_time_to_live = 864(100 days ).

I know I need to do something to stop the disk space cost, but I really don’t 
know how to do it.


Here is the strategy of the big data table :

AND compaction = {'class': 
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 
'max_threshold': '32', 'min_threshold': '12', 'tombstone_threshold': '0.1', 
'unchecked_tombstone_compaction': 'true'}
AND compression = {'chunk_length_in_kb': '64', 'class': 
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 864
AND gc_grace_seconds = 432000



> 在 2017年9月2日，下午7:34，Nicolas Guyomar  写道：
> 
> your are using STCS compaction strategy with some kind of timeseries model, 
> and you are going to end up with yor disk full!

Re: old big tombstone data file occupy much disk space

[Nicolas Guyomar]

Hi,

The nodetool command only shows what's going on on this particular node.
Validation Compaction means that Cassandra is computing Merkel Tree so that
this node can participate in an ongoing repair.

What kind of disk hardware do you have ? Node with To of data seems a lot
in regards to your first email "some few people know cassandra in my
company." !  Usually people tends to keep their node around 1Tb of data and
start adding new node in an horizontal scaling manner

The running compation 3fc33340-8e4e-11e7-9754-c981af5d39a9 Compaction
 gps  gpsfullwithstate 451.73 GiB 817.51 GiB bytes 55.26%   is really
going to end up creating a 817.51 GiB sstable, which seems huge.

At first sight I'd say that your are using STCS compaction strategy with
some kind of timeseries model, and you are going to end up with yor disk
full!



On 2 September 2017 at 03:46, qf zhou  wrote:

> After  I run  nodetool compactionstats -H,  it says that:
>
> pending tasks: 6
> - gps.gpsfullwithstate: 6
>
> id   compaction type keyspace table
>  completed  total  unit  progress
> 56ebd730-8ede-11e7-9754-c981af5d39a9 Validation  gps
>  gpsfullwithstate 478.67 GiB 4.59 TiB   bytes 10.19%
> 3fc33340-8e4e-11e7-9754-c981af5d39a9 Compaction  gps
>  gpsfullwithstate 451.73 GiB 817.51 GiB bytes 55.26%
> f9acc4b0-8edf-11e7-9754-c981af5d39a9 Validation  gps
>  gpsfullwithstate 472.36 GiB 5.32 TiB   bytes 8.67%
> 4af0b300-8f7a-11e7-9754-c981af5d39a9 Compaction  gps
>  gpsfullwithstate 3.76 GiB   75.37 GiB  bytes 5.00%
> f1282280-8edf-11e7-9754-c981af5d39a9 Validation  gps
>  gpsfullwithstate 474.95 GiB 4.59 TiB   bytes 10.11%
> 0ccb7b90-8ee0-11e7-9754-c981af5d39a9 Validation  gps
>  gpsfullwithstate 472.4 GiB  5.32 TiB   bytes 8.67%
>
> what does it mean? the difference between Validation and Compaction
>
>
> 在 2017年9月1日，下午8:36，Nicolas Guyomar  写道：
>
> Hi,
>
> Well, the command you are using works for me on 3.0.9, I do not have any
> logs in INFO level when I force a compaction and everything works fine for
> me.
>
> Are you sure there is nothing happening behind the scene ? What dies
> 'nodetool compactionstats -H' says ?
>
> On 1 September 2017 at 12:05, qf zhou  wrote:
>
>> When I trigger the compaction with the full path,  I found nothing in the
>> system.log.  Nothing happens in the  terminal and it just stops there.
>>
>> #calling operation forceUserDefinedCompaction of mbean
>> org.apache.cassandra.db:type=CompactionManager
>>
>>
>>
>>
>> 在 2017年9月1日，下午5:06，qf zhou  写道：
>>
>> I  found the  following log.  What does it mean ?
>>
>> INFO  [CompactionExecutor:11] 2017-09-01 16:55:47,909
>> NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot
>> allocate chunk of 1.000MiB
>> WARN  [RMI TCP Connection(1714)-127.0.0.1] 2017-09-01 17:02:42,516
>> CompactionManager.java:704 - Schema does not exist for file
>> mc-151276-big-Data.db. Skipping.
>>
>>
>> 在 2017年9月1日，下午4:54，Nicolas Guyomar  写道：
>>
>> You should have a log coming from the CompactionManager (in cassandra
>> system.log) when you try the command, what does it says  ?
>>
>> On 1 September 2017 at 10:07, qf zhou  wrote:
>>
>>> When I run the command,  the following occurs and  it returns null.
>>>
>>> Is it normal ?
>>>
>>> echo "run -b org.apache.cassandra.db:type=CompactionManager
>>> forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar
>>> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l
>>> localhost:7199
>>>
>>>
>>> Welcome to JMX terminal. Type "help" for available commands.
>>> $>run -b org.apache.cassandra.db:type=CompactionManager
>>> forceUserDefinedCompaction mc-100963-big-Data.db
>>> #calling operation forceUserDefinedCompaction of mbean
>>> org.apache.cassandra.db:type=CompactionManager
>>> #operation returns:
>>> null
>>>
>>>
>>>
>>>
>>> 在 2017年9月1日，下午3:49，Nicolas Guyomar  写道：
>>>
>>> Hi,
>>>
>>> Last time I used forceUserDefinedCompaction, I got myself a headache
>>> because I was trying to use a full path like you're doing, but in fact it
>>> just need the sstable as parameter
>>>
>>> Can you just try :
>>>
>>> echo "run -b org.apache.cassandra.db:type=CompactionManager
>>> forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar
>>> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l
>>> localhost:7199
>>>
>>>
>>>
>>> On 1 September 2017 at 08:29, qf zhou  wrote:
>>>

 dataPath=/hdd3/cassandra/data/gps/gpsfullwithstate-073e51a0c
 db811e68dce511be6a305f6/mc-100963-big-Data.db
 echo "run -b org.apache.cassandra.db:type=CompactionManager
 forceUserDefinedCompaction $dataPath" | java -jar
 /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l
 localhost:7199

 In the above, I am using a jmx method. But it seems that the file 

Re: old big tombstone data file occupy much disk space

[qf zhou]

After  I run  nodetool compactionstats -H,  it says that:

pending tasks: 6
- gps.gpsfullwithstate: 6

id   compaction type keyspace table
completed  total  unit  progress
56ebd730-8ede-11e7-9754-c981af5d39a9 Validation  gps  gpsfullwithstate 
478.67 GiB 4.59 TiB   bytes 10.19%  
3fc33340-8e4e-11e7-9754-c981af5d39a9 Compaction  gps  gpsfullwithstate 
451.73 GiB 817.51 GiB bytes 55.26%  
f9acc4b0-8edf-11e7-9754-c981af5d39a9 Validation  gps  gpsfullwithstate 
472.36 GiB 5.32 TiB   bytes 8.67%   
4af0b300-8f7a-11e7-9754-c981af5d39a9 Compaction  gps  gpsfullwithstate 
3.76 GiB   75.37 GiB  bytes 5.00%   
f1282280-8edf-11e7-9754-c981af5d39a9 Validation  gps  gpsfullwithstate 
474.95 GiB 4.59 TiB   bytes 10.11%  
0ccb7b90-8ee0-11e7-9754-c981af5d39a9 Validation  gps  gpsfullwithstate 
472.4 GiB  5.32 TiB   bytes 8.67%  

what does it mean? the difference between Validation and Compaction


> 在 2017年9月1日，下午8:36，Nicolas Guyomar  写道：
> 
> Hi,
> 
> Well, the command you are using works for me on 3.0.9, I do not have any logs 
> in INFO level when I force a compaction and everything works fine for me.
> 
> Are you sure there is nothing happening behind the scene ? What dies 
> 'nodetool compactionstats -H' says ? 
> 
> On 1 September 2017 at 12:05, qf zhou  > wrote:
> When I trigger the compaction with the full path,  I found nothing in the 
> system.log.  Nothing happens in the  terminal and it just stops there.
> 
> #calling operation forceUserDefinedCompaction of mbean 
> org.apache.cassandra.db:type=CompactionManager
> 
> 
> 
> 
>> 在 2017年9月1日，下午5:06，qf zhou > > 写道：
>> 
>> I  found the  following log.  What does it mean ?
>> 
>> INFO  [CompactionExecutor:11] 2017-09-01 16:55:47,909 NoSpamLogger.java:91 - 
>> Maximum memory usage reached (512.000MiB), cannot allocate chunk of 1.000MiB
>> WARN  [RMI TCP Connection(1714)-127.0.0.1] 2017-09-01 17:02:42,516 
>> CompactionManager.java:704 - Schema does not exist for file 
>> mc-151276-big-Data.db. Skipping.
>> 
>> 
>>> 在 2017年9月1日，下午4:54，Nicolas Guyomar >> > 写道：
>>> 
>>> You should have a log coming from the CompactionManager (in cassandra 
>>> system.log) when you try the command, what does it says  ?
>>> 
>>> On 1 September 2017 at 10:07, qf zhou >> > wrote:
>>> When I run the command,  the following occurs and  it returns null.
>>> 
>>> Is it normal ?
>>> 
>>> echo "run -b org.apache.cassandra.db:type=CompactionManager 
>>> forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar 
>>> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l localhost:7199
>>> 
>>> 
>>> Welcome to JMX terminal. Type "help" for available commands.
>>> $>run -b org.apache.cassandra.db:type=CompactionManager 
>>> forceUserDefinedCompaction mc-100963-big-Data.db
>>> #calling operation forceUserDefinedCompaction of mbean 
>>> org.apache.cassandra.db:type=CompactionManager
>>> #operation returns: 
>>> null
>>> 
>>> 
>>> 
>>> 
 在 2017年9月1日，下午3:49，Nicolas Guyomar > 写道：
 
 Hi,
 
 Last time I used forceUserDefinedCompaction, I got myself a headache 
 because I was trying to use a full path like you're doing, but in fact it 
 just need the sstable as parameter
 
 Can you just try : 
 
 echo "run -b org.apache.cassandra.db:type=CompactionManager 
 forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar 
 /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l localhost:7199
 
 
 
 On 1 September 2017 at 08:29, qf zhou > wrote:
 
 dataPath=/hdd3/cassandra/data/gps/gpsfullwithstate-073e51a0cdb811e68dce511be6a305f6/mc-100963-big-Data.db
 echo "run -b org.apache.cassandra.db:type=CompactionManager 
 forceUserDefinedCompaction $dataPath" | java -jar 
 /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l localhost:7199
 
 In the above, I am using a jmx method. But it seems that the file size 
 doesn’t change. My command is wrong ?
 
 > 在 2017年9月1日，下午2:17，Jeff Jirsa  > 写道：
 >
 > User defined compaction to do a single sstable compaction on just that 
 > sstable
 >
 > It's a nodetool command in very recent versions, or a jmx method in 
 > older versions
 >
 >
 > --
 > Jeff Jirsa
 >
 >
 >> On Aug 31, 2017, at 11:04 PM, qf zhou > > wrote:
 >>
 >> I am using  a cluster with  3 nodes and  the cassandra version is 
 >> 3.0.9. I have used it about 6 

Re: old big tombstone data file occupy much disk space

[Nicolas Guyomar]

Hi,

Well, the command you are using works for me on 3.0.9, I do not have any
logs in INFO level when I force a compaction and everything works fine for
me.

Are you sure there is nothing happening behind the scene ? What dies
'nodetool compactionstats -H' says ?

On 1 September 2017 at 12:05, qf zhou  wrote:

> When I trigger the compaction with the full path,  I found nothing in the
> system.log.  Nothing happens in the  terminal and it just stops there.
>
> #calling operation forceUserDefinedCompaction of mbean
> org.apache.cassandra.db:type=CompactionManager
>
>
>
>
> 在 2017年9月1日，下午5:06，qf zhou  写道：
>
> I  found the  following log.  What does it mean ?
>
> INFO  [CompactionExecutor:11] 2017-09-01 16:55:47,909 NoSpamLogger.java:91
> - Maximum memory usage reached (512.000MiB), cannot allocate chunk of
> 1.000MiB
> WARN  [RMI TCP Connection(1714)-127.0.0.1] 2017-09-01 17:02:42,516
> CompactionManager.java:704 - Schema does not exist for file
> mc-151276-big-Data.db. Skipping.
>
>
> 在 2017年9月1日，下午4:54，Nicolas Guyomar  写道：
>
> You should have a log coming from the CompactionManager (in cassandra
> system.log) when you try the command, what does it says  ?
>
> On 1 September 2017 at 10:07, qf zhou  wrote:
>
>> When I run the command,  the following occurs and  it returns null.
>>
>> Is it normal ?
>>
>> echo "run -b org.apache.cassandra.db:type=CompactionManager
>> forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar
>> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l localhost:7199
>>
>>
>> Welcome to JMX terminal. Type "help" for available commands.
>> $>run -b org.apache.cassandra.db:type=CompactionManager
>> forceUserDefinedCompaction mc-100963-big-Data.db
>> #calling operation forceUserDefinedCompaction of mbean
>> org.apache.cassandra.db:type=CompactionManager
>> #operation returns:
>> null
>>
>>
>>
>>
>> 在 2017年9月1日，下午3:49，Nicolas Guyomar  写道：
>>
>> Hi,
>>
>> Last time I used forceUserDefinedCompaction, I got myself a headache
>> because I was trying to use a full path like you're doing, but in fact it
>> just need the sstable as parameter
>>
>> Can you just try :
>>
>> echo "run -b org.apache.cassandra.db:type=CompactionManager
>> forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar
>> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l localhost:7199
>>
>>
>>
>> On 1 September 2017 at 08:29, qf zhou  wrote:
>>
>>>
>>> dataPath=/hdd3/cassandra/data/gps/gpsfullwithstate-073e51a0c
>>> db811e68dce511be6a305f6/mc-100963-big-Data.db
>>> echo "run -b org.apache.cassandra.db:type=CompactionManager
>>> forceUserDefinedCompaction $dataPath" | java -jar
>>> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l
>>> localhost:7199
>>>
>>> In the above, I am using a jmx method. But it seems that the file size
>>> doesn’t change. My command is wrong ?
>>>
>>> > 在 2017年9月1日，下午2:17，Jeff Jirsa  写道：
>>> >
>>> > User defined compaction to do a single sstable compaction on just that
>>> sstable
>>> >
>>> > It's a nodetool command in very recent versions, or a jmx method in
>>> older versions
>>> >
>>> >
>>> > --
>>> > Jeff Jirsa
>>> >
>>> >
>>> >> On Aug 31, 2017, at 11:04 PM, qf zhou  wrote:
>>> >>
>>> >> I am using  a cluster with  3 nodes and  the cassandra version is
>>> 3.0.9. I have used it about 6 months. Now each node has about 1.5T data in
>>> the disk.
>>> >> I found some sstables file are over 300G. Using the  sstablemetadata
>>> command,  I found it:  Estimated droppable tombstones: 0.9622972799707109.
>>> >> It is obvious that too much tombstone data exists.
>>> >> The default_time_to_live = 864(100 days) and   gc_grace_seconds =
>>> 432000(5 days).  Using nodetool  compactionstats, I found the some
>>> compaction processes exists.
>>> >> So I really  want to know how to clear tombstone data ?  otherwise
>>> the disk space will cost too much.
>>> >> I really need some help, because some few people know cassandra in my
>>> company.
>>> >> Thank you very much!
>>> >>
>>> >>
>>> >> -
>>> >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>>> >> For additional commands, e-mail: user-h...@cassandra.apache.org
>>> >>
>>> >
>>> > -
>>> > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>>> > For additional commands, e-mail: user-h...@cassandra.apache.org
>>> >
>>>
>>>
>>> -
>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>>
>>>
>>
>>
>
>
> - To
> unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For 

Re: old big tombstone data file occupy much disk space

[qf zhou]

When I trigger the compaction with the full path,  I found nothing in the system.log.  Nothing happens in the  terminal and it just stops there.#calling operation forceUserDefinedCompaction of mbean org.apache.cassandra.db:type=CompactionManager在 2017年9月1日，下午5:06，qf zhou  写道：I  found the  following log.  What does it mean ?INFO  [CompactionExecutor:11] 2017-09-01 16:55:47,909 NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot allocate chunk of 1.000MiBWARN  [RMI TCP Connection(1714)-127.0.0.1] 2017-09-01 17:02:42,516 CompactionManager.java:704 - Schema does not exist for file mc-151276-big-Data.db. Skipping.在 2017年9月1日，下午4:54，Nicolas Guyomar  写道：You should have a log coming from the CompactionManager (in cassandra system.log) when you try the command, what does it says  ?On 1 September 2017 at 10:07, qf zhou  wrote:When I run the command,  the following occurs and  it returns null.Is it normal ?echo "run -b org.apache.cassandra.db:type=CompactionManager forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l localhost:7199Welcome to JMX terminal. Type "help" for available commands.$>run -b org.apache.cassandra.db:type=CompactionManager forceUserDefinedCompaction mc-100963-big-Data.db#calling operation forceUserDefinedCompaction of mbean org.apache.cassandra.db:type=CompactionManager#operation returns: null在 2017年9月1日，下午3:49，Nicolas Guyomar  写道：Hi,Last time I used forceUserDefinedCompaction, I got myself a headache because I was trying to use a full path like you're doing, but in fact it just need the sstable as parameterCan you just try : echo "run -b org.apache.cassandra.db:type=CompactionManager forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l localhost:7199On 1 September 2017 at 08:29, qf zhou  wrote:
dataPath=/hdd3/cassandra/data/gps/gpsfullwithstate-073e51a0cdb811e68dce511be6a305f6/mc-100963-big-Data.db
echo "run -b org.apache.cassandra.db:type=CompactionManager forceUserDefinedCompaction $dataPath" | java -jar /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l localhost:7199

In the above, I am using a jmx method. But it seems that the file size doesn’t change. My command is wrong ?

> 在 2017年9月1日，下午2:17，Jeff Jirsa  写道：
>
> User defined compaction to do a single sstable compaction on just that sstable
>
> It's a nodetool command in very recent versions, or a jmx method in older versions
>
>
> --
> Jeff Jirsa
>
>
>> On Aug 31, 2017, at 11:04 PM, qf zhou  wrote:
>>
>> I am using  a cluster with  3 nodes and  the cassandra version is 3.0.9. I have used it about 6 months. Now each node has about 1.5T data in the disk.
>> I found some sstables file are over 300G. Using the  sstablemetadata command,  I found it:  Estimated droppable tombstones: 0.9622972799707109.
>> It is obvious that too much tombstone data exists.
>> The default_time_to_live = 864(100 days) and   gc_grace_seconds = 432000(5 days).  Using nodetool  compactionstats, I found the some compaction processes exists.
>> So I really  want to know how to clear tombstone data ?  otherwise the disk space will cost too much.
>> I really need some help, because some few people know cassandra in my company.
>> Thank you very much!
>>
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org




-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: old big tombstone data file occupy much disk space

[Nicolas Guyomar]

Whoops sorry I mislead you with cassandra 2.1 behavior, you were right
giving your sstable full path , what kind of log do you have when you
trigger the compaction with the full path ?

On 1 September 2017 at 11:30, Nicolas Guyomar 
wrote:

> Well, not sure why you reached a memory usage limit, but according to the
> 3.0 branche's code : https://github.com/apache/
> cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/compaction/
> CompactionManager.java#L632 you just need to give the sstable filename,
> and Cassandra manage to find it based on cassandra version, sstable
> filename convention and so on
>
> Are you sure those sstables you are trying to get rid off are really in an
> active schema, and not some leftover from an old keyspace/table? This is
> what "schema does not exist" means to me.
>
> On 1 September 2017 at 11:06, qf zhou  wrote:
>
>> I  found the  following log.  What does it mean ?
>>
>> INFO  [CompactionExecutor:11] 2017-09-01 16:55:47,909
>> NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot
>> allocate chunk of 1.000MiB
>> WARN  [RMI TCP Connection(1714)-127.0.0.1] 2017-09-01 17:02:42,516
>> CompactionManager.java:704 - Schema does not exist for file
>> mc-151276-big-Data.db. Skipping.
>>
>>
>> 在 2017年9月1日，下午4:54，Nicolas Guyomar  写道：
>>
>> You should have a log coming from the CompactionManager (in cassandra
>> system.log) when you try the command, what does it says  ?
>>
>> On 1 September 2017 at 10:07, qf zhou  wrote:
>>
>>> When I run the command,  the following occurs and  it returns null.
>>>
>>> Is it normal ?
>>>
>>> echo "run -b org.apache.cassandra.db:type=CompactionManager
>>> forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar
>>> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l
>>> localhost:7199
>>>
>>>
>>> Welcome to JMX terminal. Type "help" for available commands.
>>> $>run -b org.apache.cassandra.db:type=CompactionManager
>>> forceUserDefinedCompaction mc-100963-big-Data.db
>>> #calling operation forceUserDefinedCompaction of mbean
>>> org.apache.cassandra.db:type=CompactionManager
>>> #operation returns:
>>> null
>>>
>>>
>>>
>>>
>>> 在 2017年9月1日，下午3:49，Nicolas Guyomar  写道：
>>>
>>> Hi,
>>>
>>> Last time I used forceUserDefinedCompaction, I got myself a headache
>>> because I was trying to use a full path like you're doing, but in fact it
>>> just need the sstable as parameter
>>>
>>> Can you just try :
>>>
>>> echo "run -b org.apache.cassandra.db:type=CompactionManager
>>> forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar
>>> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l
>>> localhost:7199
>>>
>>>
>>>
>>> On 1 September 2017 at 08:29, qf zhou  wrote:
>>>

 dataPath=/hdd3/cassandra/data/gps/gpsfullwithstate-073e51a0c
 db811e68dce511be6a305f6/mc-100963-big-Data.db
 echo "run -b org.apache.cassandra.db:type=CompactionManager
 forceUserDefinedCompaction $dataPath" | java -jar
 /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l
 localhost:7199

 In the above, I am using a jmx method. But it seems that the file size
 doesn’t change. My command is wrong ?

 > 在 2017年9月1日，下午2:17，Jeff Jirsa  写道：
 >
 > User defined compaction to do a single sstable compaction on just
 that sstable
 >
 > It's a nodetool command in very recent versions, or a jmx method in
 older versions
 >
 >
 > --
 > Jeff Jirsa
 >
 >
 >> On Aug 31, 2017, at 11:04 PM, qf zhou  wrote:
 >>
 >> I am using  a cluster with  3 nodes and  the cassandra version is
 3.0.9. I have used it about 6 months. Now each node has about 1.5T data in
 the disk.
 >> I found some sstables file are over 300G. Using the  sstablemetadata
 command,  I found it:  Estimated droppable tombstones: 0.9622972799707109.
 >> It is obvious that too much tombstone data exists.
 >> The default_time_to_live = 864(100 days) and   gc_grace_seconds
 = 432000(5 days).  Using nodetool  compactionstats, I found the some
 compaction processes exists.
 >> So I really  want to know how to clear tombstone data ?  otherwise
 the disk space will cost too much.
 >> I really need some help, because some few people know cassandra in
 my company.
 >> Thank you very much!
 >>
 >>
 >> 
 -
 >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
 >> For additional commands, e-mail: user-h...@cassandra.apache.org
 >>
 >
 > -
 > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
 > For additional commands, e-mail: user-h...@cassandra.apache.org

Re: old big tombstone data file occupy much disk space

[Nicolas Guyomar]

Well, not sure why you reached a memory usage limit, but according to the
3.0 branche's code :
https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L632
you just need to give the sstable filename, and Cassandra manage to find it
based on cassandra version, sstable filename convention and so on

Are you sure those sstables you are trying to get rid off are really in an
active schema, and not some leftover from an old keyspace/table? This is
what "schema does not exist" means to me.

On 1 September 2017 at 11:06, qf zhou  wrote:

> I  found the  following log.  What does it mean ?
>
> INFO  [CompactionExecutor:11] 2017-09-01 16:55:47,909 NoSpamLogger.java:91
> - Maximum memory usage reached (512.000MiB), cannot allocate chunk of
> 1.000MiB
> WARN  [RMI TCP Connection(1714)-127.0.0.1] 2017-09-01 17:02:42,516
> CompactionManager.java:704 - Schema does not exist for file
> mc-151276-big-Data.db. Skipping.
>
>
> 在 2017年9月1日，下午4:54，Nicolas Guyomar  写道：
>
> You should have a log coming from the CompactionManager (in cassandra
> system.log) when you try the command, what does it says  ?
>
> On 1 September 2017 at 10:07, qf zhou  wrote:
>
>> When I run the command,  the following occurs and  it returns null.
>>
>> Is it normal ?
>>
>> echo "run -b org.apache.cassandra.db:type=CompactionManager
>> forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar
>> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l localhost:7199
>>
>>
>> Welcome to JMX terminal. Type "help" for available commands.
>> $>run -b org.apache.cassandra.db:type=CompactionManager
>> forceUserDefinedCompaction mc-100963-big-Data.db
>> #calling operation forceUserDefinedCompaction of mbean
>> org.apache.cassandra.db:type=CompactionManager
>> #operation returns:
>> null
>>
>>
>>
>>
>> 在 2017年9月1日，下午3:49，Nicolas Guyomar  写道：
>>
>> Hi,
>>
>> Last time I used forceUserDefinedCompaction, I got myself a headache
>> because I was trying to use a full path like you're doing, but in fact it
>> just need the sstable as parameter
>>
>> Can you just try :
>>
>> echo "run -b org.apache.cassandra.db:type=CompactionManager
>> forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar
>> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l localhost:7199
>>
>>
>>
>> On 1 September 2017 at 08:29, qf zhou  wrote:
>>
>>>
>>> dataPath=/hdd3/cassandra/data/gps/gpsfullwithstate-073e51a0c
>>> db811e68dce511be6a305f6/mc-100963-big-Data.db
>>> echo "run -b org.apache.cassandra.db:type=CompactionManager
>>> forceUserDefinedCompaction $dataPath" | java -jar
>>> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l
>>> localhost:7199
>>>
>>> In the above, I am using a jmx method. But it seems that the file size
>>> doesn’t change. My command is wrong ?
>>>
>>> > 在 2017年9月1日，下午2:17，Jeff Jirsa  写道：
>>> >
>>> > User defined compaction to do a single sstable compaction on just that
>>> sstable
>>> >
>>> > It's a nodetool command in very recent versions, or a jmx method in
>>> older versions
>>> >
>>> >
>>> > --
>>> > Jeff Jirsa
>>> >
>>> >
>>> >> On Aug 31, 2017, at 11:04 PM, qf zhou  wrote:
>>> >>
>>> >> I am using  a cluster with  3 nodes and  the cassandra version is
>>> 3.0.9. I have used it about 6 months. Now each node has about 1.5T data in
>>> the disk.
>>> >> I found some sstables file are over 300G. Using the  sstablemetadata
>>> command,  I found it:  Estimated droppable tombstones: 0.9622972799707109.
>>> >> It is obvious that too much tombstone data exists.
>>> >> The default_time_to_live = 864(100 days) and   gc_grace_seconds =
>>> 432000(5 days).  Using nodetool  compactionstats, I found the some
>>> compaction processes exists.
>>> >> So I really  want to know how to clear tombstone data ?  otherwise
>>> the disk space will cost too much.
>>> >> I really need some help, because some few people know cassandra in my
>>> company.
>>> >> Thank you very much!
>>> >>
>>> >>
>>> >> -
>>> >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>>> >> For additional commands, e-mail: user-h...@cassandra.apache.org
>>> >>
>>> >
>>> > -
>>> > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>>> > For additional commands, e-mail: user-h...@cassandra.apache.org
>>> >
>>>
>>>
>>> -
>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>>
>>>
>>
>>
>
>

Re: old big tombstone data file occupy much disk space

[qf zhou]

I  found the  following log.  What does it mean ?

INFO  [CompactionExecutor:11] 2017-09-01 16:55:47,909 NoSpamLogger.java:91 - 
Maximum memory usage reached (512.000MiB), cannot allocate chunk of 1.000MiB
WARN  [RMI TCP Connection(1714)-127.0.0.1] 2017-09-01 17:02:42,516 
CompactionManager.java:704 - Schema does not exist for file 
mc-151276-big-Data.db. Skipping.


> 在 2017年9月1日，下午4:54，Nicolas Guyomar  写道：
> 
> You should have a log coming from the CompactionManager (in cassandra 
> system.log) when you try the command, what does it says  ?
> 
> On 1 September 2017 at 10:07, qf zhou  > wrote:
> When I run the command,  the following occurs and  it returns null.
> 
> Is it normal ?
> 
> echo "run -b org.apache.cassandra.db:type=CompactionManager 
> forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar 
> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l localhost:7199
> 
> 
> Welcome to JMX terminal. Type "help" for available commands.
> $>run -b org.apache.cassandra.db:type=CompactionManager 
> forceUserDefinedCompaction mc-100963-big-Data.db
> #calling operation forceUserDefinedCompaction of mbean 
> org.apache.cassandra.db:type=CompactionManager
> #operation returns: 
> null
> 
> 
> 
> 
>> 在 2017年9月1日，下午3:49，Nicolas Guyomar > > 写道：
>> 
>> Hi,
>> 
>> Last time I used forceUserDefinedCompaction, I got myself a headache because 
>> I was trying to use a full path like you're doing, but in fact it just need 
>> the sstable as parameter
>> 
>> Can you just try : 
>> 
>> echo "run -b org.apache.cassandra.db:type=CompactionManager 
>> forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar 
>> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l localhost:7199
>> 
>> 
>> 
>> On 1 September 2017 at 08:29, qf zhou > > wrote:
>> 
>> dataPath=/hdd3/cassandra/data/gps/gpsfullwithstate-073e51a0cdb811e68dce511be6a305f6/mc-100963-big-Data.db
>> echo "run -b org.apache.cassandra.db:type=CompactionManager 
>> forceUserDefinedCompaction $dataPath" | java -jar 
>> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l localhost:7199
>> 
>> In the above, I am using a jmx method. But it seems that the file size 
>> doesn’t change. My command is wrong ?
>> 
>> > 在 2017年9月1日，下午2:17，Jeff Jirsa > 
>> > 写道：
>> >
>> > User defined compaction to do a single sstable compaction on just that 
>> > sstable
>> >
>> > It's a nodetool command in very recent versions, or a jmx method in older 
>> > versions
>> >
>> >
>> > --
>> > Jeff Jirsa
>> >
>> >
>> >> On Aug 31, 2017, at 11:04 PM, qf zhou > >> > wrote:
>> >>
>> >> I am using  a cluster with  3 nodes and  the cassandra version is 3.0.9. 
>> >> I have used it about 6 months. Now each node has about 1.5T data in the 
>> >> disk.
>> >> I found some sstables file are over 300G. Using the  sstablemetadata 
>> >> command,  I found it:  Estimated droppable tombstones: 0.9622972799707109.
>> >> It is obvious that too much tombstone data exists.
>> >> The default_time_to_live = 864(100 days) and   gc_grace_seconds = 
>> >> 432000(5 days).  Using nodetool  compactionstats, I found the some 
>> >> compaction processes exists.
>> >> So I really  want to know how to clear tombstone data ?  otherwise the 
>> >> disk space will cost too much.
>> >> I really need some help, because some few people know cassandra in my 
>> >> company.
>> >> Thank you very much!
>> >>
>> >>
>> >> -
>> >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
>> >> 
>> >> For additional commands, e-mail: user-h...@cassandra.apache.org 
>> >> 
>> >>
>> >
>> > -
>> > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
>> > 
>> > For additional commands, e-mail: user-h...@cassandra.apache.org 
>> > 
>> >
>> 
>> 
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
>> 
>> For additional commands, e-mail: user-h...@cassandra.apache.org 
>> 
>> 
>> 
> 
> 

Re: old big tombstone data file occupy much disk space

[Nicolas Guyomar]

You should have a log coming from the CompactionManager (in cassandra
system.log) when you try the command, what does it says  ?

On 1 September 2017 at 10:07, qf zhou  wrote:

> When I run the command,  the following occurs and  it returns null.
>
> Is it normal ?
>
> echo "run -b org.apache.cassandra.db:type=CompactionManager
> forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar
> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l localhost:7199
>
>
> Welcome to JMX terminal. Type "help" for available commands.
> $>run -b org.apache.cassandra.db:type=CompactionManager
> forceUserDefinedCompaction mc-100963-big-Data.db
> #calling operation forceUserDefinedCompaction of mbean
> org.apache.cassandra.db:type=CompactionManager
> #operation returns:
> null
>
>
>
>
> 在 2017年9月1日，下午3:49，Nicolas Guyomar  写道：
>
> Hi,
>
> Last time I used forceUserDefinedCompaction, I got myself a headache
> because I was trying to use a full path like you're doing, but in fact it
> just need the sstable as parameter
>
> Can you just try :
>
> echo "run -b org.apache.cassandra.db:type=CompactionManager
> forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar
> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l localhost:7199
>
>
>
> On 1 September 2017 at 08:29, qf zhou  wrote:
>
>>
>> dataPath=/hdd3/cassandra/data/gps/gpsfullwithstate-073e51a0c
>> db811e68dce511be6a305f6/mc-100963-big-Data.db
>> echo "run -b org.apache.cassandra.db:type=CompactionManager
>> forceUserDefinedCompaction $dataPath" | java -jar
>> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l localhost:7199
>>
>> In the above, I am using a jmx method. But it seems that the file size
>> doesn’t change. My command is wrong ?
>>
>> > 在 2017年9月1日，下午2:17，Jeff Jirsa  写道：
>> >
>> > User defined compaction to do a single sstable compaction on just that
>> sstable
>> >
>> > It's a nodetool command in very recent versions, or a jmx method in
>> older versions
>> >
>> >
>> > --
>> > Jeff Jirsa
>> >
>> >
>> >> On Aug 31, 2017, at 11:04 PM, qf zhou  wrote:
>> >>
>> >> I am using  a cluster with  3 nodes and  the cassandra version is
>> 3.0.9. I have used it about 6 months. Now each node has about 1.5T data in
>> the disk.
>> >> I found some sstables file are over 300G. Using the  sstablemetadata
>> command,  I found it:  Estimated droppable tombstones: 0.9622972799707109.
>> >> It is obvious that too much tombstone data exists.
>> >> The default_time_to_live = 864(100 days) and   gc_grace_seconds =
>> 432000(5 days).  Using nodetool  compactionstats, I found the some
>> compaction processes exists.
>> >> So I really  want to know how to clear tombstone data ?  otherwise the
>> disk space will cost too much.
>> >> I really need some help, because some few people know cassandra in my
>> company.
>> >> Thank you very much!
>> >>
>> >>
>> >> -
>> >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> >> For additional commands, e-mail: user-h...@cassandra.apache.org
>> >>
>> >
>> > -
>> > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> > For additional commands, e-mail: user-h...@cassandra.apache.org
>> >
>>
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>>
>
>

Re: old big tombstone data file occupy much disk space

[qf zhou]

When I run the command,  the following occurs and  it returns null.

Is it normal ?

echo "run -b org.apache.cassandra.db:type=CompactionManager 
forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar 
/opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l localhost:7199


Welcome to JMX terminal. Type "help" for available commands.
$>run -b org.apache.cassandra.db:type=CompactionManager 
forceUserDefinedCompaction mc-100963-big-Data.db
#calling operation forceUserDefinedCompaction of mbean 
org.apache.cassandra.db:type=CompactionManager
#operation returns: 
null




> 在 2017年9月1日，下午3:49，Nicolas Guyomar  写道：
> 
> Hi,
> 
> Last time I used forceUserDefinedCompaction, I got myself a headache because 
> I was trying to use a full path like you're doing, but in fact it just need 
> the sstable as parameter
> 
> Can you just try : 
> 
> echo "run -b org.apache.cassandra.db:type=CompactionManager 
> forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar 
> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l localhost:7199
> 
> 
> 
> On 1 September 2017 at 08:29, qf zhou  > wrote:
> 
> dataPath=/hdd3/cassandra/data/gps/gpsfullwithstate-073e51a0cdb811e68dce511be6a305f6/mc-100963-big-Data.db
> echo "run -b org.apache.cassandra.db:type=CompactionManager 
> forceUserDefinedCompaction $dataPath" | java -jar 
> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l localhost:7199
> 
> In the above, I am using a jmx method. But it seems that the file size 
> doesn’t change. My command is wrong ?
> 
> > 在 2017年9月1日，下午2:17，Jeff Jirsa > 
> > 写道：
> >
> > User defined compaction to do a single sstable compaction on just that 
> > sstable
> >
> > It's a nodetool command in very recent versions, or a jmx method in older 
> > versions
> >
> >
> > --
> > Jeff Jirsa
> >
> >
> >> On Aug 31, 2017, at 11:04 PM, qf zhou  >> > wrote:
> >>
> >> I am using  a cluster with  3 nodes and  the cassandra version is 3.0.9. I 
> >> have used it about 6 months. Now each node has about 1.5T data in the disk.
> >> I found some sstables file are over 300G. Using the  sstablemetadata 
> >> command,  I found it:  Estimated droppable tombstones: 0.9622972799707109.
> >> It is obvious that too much tombstone data exists.
> >> The default_time_to_live = 864(100 days) and   gc_grace_seconds = 
> >> 432000(5 days).  Using nodetool  compactionstats, I found the some 
> >> compaction processes exists.
> >> So I really  want to know how to clear tombstone data ?  otherwise the 
> >> disk space will cost too much.
> >> I really need some help, because some few people know cassandra in my 
> >> company.
> >> Thank you very much!
> >>
> >>
> >> -
> >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
> >> 
> >> For additional commands, e-mail: user-h...@cassandra.apache.org 
> >> 
> >>
> >
> > -
> > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
> > 
> > For additional commands, e-mail: user-h...@cassandra.apache.org 
> > 
> >
> 
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
> 
> For additional commands, e-mail: user-h...@cassandra.apache.org 
> 
> 
> 

Re: old big tombstone data file occupy much disk space

[Nicolas Guyomar]

Hi,

Last time I used forceUserDefinedCompaction, I got myself a headache
because I was trying to use a full path like you're doing, but in fact it
just need the sstable as parameter

Can you just try :

echo "run -b org.apache.cassandra.db:type=CompactionManager
forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar
/opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l localhost:7199



On 1 September 2017 at 08:29, qf zhou  wrote:

>
> dataPath=/hdd3/cassandra/data/gps/gpsfullwithstate-
> 073e51a0cdb811e68dce511be6a305f6/mc-100963-big-Data.db
> echo "run -b org.apache.cassandra.db:type=CompactionManager
> forceUserDefinedCompaction $dataPath" | java -jar 
> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar
>  -l localhost:7199
>
> In the above, I am using a jmx method. But it seems that the file size
> doesn’t change. My command is wrong ?
>
> > 在 2017年9月1日，下午2:17，Jeff Jirsa  写道：
> >
> > User defined compaction to do a single sstable compaction on just that
> sstable
> >
> > It's a nodetool command in very recent versions, or a jmx method in
> older versions
> >
> >
> > --
> > Jeff Jirsa
> >
> >
> >> On Aug 31, 2017, at 11:04 PM, qf zhou  wrote:
> >>
> >> I am using  a cluster with  3 nodes and  the cassandra version is
> 3.0.9. I have used it about 6 months. Now each node has about 1.5T data in
> the disk.
> >> I found some sstables file are over 300G. Using the  sstablemetadata
> command,  I found it:  Estimated droppable tombstones: 0.9622972799707109.
> >> It is obvious that too much tombstone data exists.
> >> The default_time_to_live = 864(100 days) and   gc_grace_seconds =
> 432000(5 days).  Using nodetool  compactionstats, I found the some
> compaction processes exists.
> >> So I really  want to know how to clear tombstone data ?  otherwise the
> disk space will cost too much.
> >> I really need some help, because some few people know cassandra in my
> company.
> >> Thank you very much!
> >>
> >>
> >> -
> >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> >> For additional commands, e-mail: user-h...@cassandra.apache.org
> >>
> >
> > -
> > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: user-h...@cassandra.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Re: old big tombstone data file occupy much disk space

[qf zhou]

dataPath=/hdd3/cassandra/data/gps/gpsfullwithstate-073e51a0cdb811e68dce511be6a305f6/mc-100963-big-Data.db
echo "run -b org.apache.cassandra.db:type=CompactionManager 
forceUserDefinedCompaction $dataPath" | java -jar 
/opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l localhost:7199

In the above, I am using a jmx method. But it seems that the file size doesn’t 
change. My command is wrong ?

> 在 2017年9月1日，下午2:17，Jeff Jirsa  写道：
> 
> User defined compaction to do a single sstable compaction on just that sstable
> 
> It's a nodetool command in very recent versions, or a jmx method in older 
> versions
> 
> 
> -- 
> Jeff Jirsa
> 
> 
>> On Aug 31, 2017, at 11:04 PM, qf zhou  wrote:
>> 
>> I am using  a cluster with  3 nodes and  the cassandra version is 3.0.9. I 
>> have used it about 6 months. Now each node has about 1.5T data in the disk.
>> I found some sstables file are over 300G. Using the  sstablemetadata 
>> command,  I found it:  Estimated droppable tombstones: 0.9622972799707109.
>> It is obvious that too much tombstone data exists.
>> The default_time_to_live = 864(100 days) and   gc_grace_seconds = 
>> 432000(5 days).  Using nodetool  compactionstats, I found the some 
>> compaction processes exists.
>> So I really  want to know how to clear tombstone data ?  otherwise the disk 
>> space will cost too much.
>> I really need some help, because some few people know cassandra in my 
>> company.  
>> Thank you very much!
>> 
>> 
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>> 
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
> 


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: old big tombstone data file occupy much disk space

[Jeff Jirsa]

User defined compaction to do a single sstable compaction on just that sstable

It's a nodetool command in very recent versions, or a jmx method in older 
versions


-- 
Jeff Jirsa


> On Aug 31, 2017, at 11:04 PM, qf zhou  wrote:
> 
> I am using  a cluster with  3 nodes and  the cassandra version is 3.0.9. I 
> have used it about 6 months. Now each node has about 1.5T data in the disk.
> I found some sstables file are over 300G. Using the  sstablemetadata command, 
>  I found it:  Estimated droppable tombstones: 0.9622972799707109.
> It is obvious that too much tombstone data exists.
> The default_time_to_live = 864(100 days) and   gc_grace_seconds = 
> 432000(5 days).  Using nodetool  compactionstats, I found the some compaction 
> processes exists.
> So I really  want to know how to clear tombstone data ?  otherwise the disk 
> space will cost too much.
> I really need some help, because some few people know cassandra in my 
> company.  
> Thank you very much!
> 
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org