Re: Files not removed after compaction

2015-06-13 Thread Spencer Brown 
You can't delete by year unless tables are organized by year.  For deleting
tables, cassandra keeeps a copy of the file history which you can delete.
Also, you could delete all the extra rows using CQL.

On Wed, Jun 10, 2015 at 7:15 PM, Jeff Ferland  wrote:

> Compaction finished yesterday, but I still have this going on:
>
> Space used (live), bytes:   878681716067
> Space used (total), bytes: 2227857083852
>
> jbf@ip-10-0-2-98:/ebs/cassandra/data/trends/trends$ sudo lsof *-Data.db
> COMMAND  PID  USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
> java4473 cassandra  446r   REG   0,26  17582559172 39241
> trends-trends-jb-144864-Data.db
> java4473 cassandra  448r   REG   0,26 62040962 37431
> trends-trends-jb-144731-Data.db
> java4473 cassandra  449r   REG   0,26 829935047545 21150
> trends-trends-jb-143581-Data.db
> java4473 cassandra  452r   REG   0,26  8980406 39503
> trends-trends-jb-144882-Data.db
> java4473 cassandra  454r   REG   0,26  8980406 39503
> trends-trends-jb-144882-Data.db
> java4473 cassandra  462r   REG   0,26  9487703 39542
> trends-trends-jb-144883-Data.db
> java4473 cassandra  463r   REG   0,26 36158226 39629
> trends-trends-jb-144889-Data.db
> java4473 cassandra  468r   REG   0,26105693505 39447
> trends-trends-jb-144881-Data.db
> java4473 cassandra  530r   REG   0,26  17582559172 39241
> trends-trends-jb-144864-Data.db
> java4473 cassandra  535r   REG   0,26105693505 39447
> trends-trends-jb-144881-Data.db
> java4473 cassandra  542r   REG   0,26  9487703 39542
> trends-trends-jb-144883-Data.db
> java4473 cassandra  553u   REG   0,26   6431729821 39556
> trends-trends-tmp-jb-144884-Data.db
> jbf@ip-10-0-2-98:/ebs/cassandra/data/trends/trends$ ls *-Data.db
> trends-trends-jb-142631-Data.db  trends-trends-jb-143562-Data.db
> trends-trends-jb-143581-Data.db  trends-trends-jb-144731-Data.db
> trends-trends-jb-144883-Data.db
> trends-trends-jb-142633-Data.db  trends-trends-jb-143563-Data.db
> trends-trends-jb-144530-Data.db  trends-trends-jb-144864-Data.db
> trends-trends-jb-144889-Data.db
> trends-trends-jb-143026-Data.db  trends-trends-jb-143564-Data.db
> trends-trends-jb-144551-Data.db  trends-trends-jb-144881-Data.db
> trends-trends-tmp-jb-144884-Data.db
> trends-trends-jb-143533-Data.db  trends-trends-jb-143578-Data.db
> trends-trends-jb-144552-Data.db  trends-trends-jb-144882-Data.db
> jbf@ip-10-0-2-98:/ebs/cassandra/data/trends/trends$ cd -
> /mnt/cassandra/data/trends/trends
> jbf@ip-10-0-2-98:/mnt/cassandra/data/trends/trends$ sudo lsof *
> jbf@ip-10-0-2-98:/mnt/cassandra/data/trends/trends$ ls *-Data.db
> trends-trends-jb-124502-Data.db  trends-trends-jb-141113-Data.db
> trends-trends-jb-141377-Data.db  trends-trends-jb-141846-Data.db
> trends-trends-jb-144890-Data.db
> trends-trends-jb-125457-Data.db  trends-trends-jb-141123-Data.db
> trends-trends-jb-141391-Data.db  trends-trends-jb-141871-Data.db
> trends-trends-jb-41121-Data.db
> trends-trends-jb-130016-Data.db  trends-trends-jb-141137-Data.db
> trends-trends-jb-141538-Data.db  trends-trends-jb-141883-Data.db
> trends-trends.trends_date_idx-jb-2100-Data.db
> trends-trends-jb-139563-Data.db  trends-trends-jb-141358-Data.db
> trends-trends-jb-141806-Data.db  trends-trends-jb-142033-Data.db
> trends-trends-jb-141102-Data.db  trends-trends-jb-141363-Data.db
> trends-trends-jb-141829-Data.db  trends-trends-jb-144553-Data.db
>
> Essentially, everything was brought down in a major compaction but
> Cassandra won’t let go of the files and I can’t figure out as to why
> they’re not being cleaned up even when they’re not open. I’d really like to
> get those files cleared out safely and rsync away from the EBS volume I
> used to survive a compaction.
>
> While I’m at it, is there any way to anti-compact a table by year? Since
> I’ve switched this recently to date tiered compaction, breaking up that big
> one would probably be helpful.
>
> -Jeff
>

Re: Files not removed after compaction

2015-06-10 Thread Jeff Ferland 
Cassandra 2.0.12-200 / DSE 4.6.1 
https://issues.apache.org/jira/browse/CASSANDRA-9577 


-Jeff

> On Jun 10, 2015, at 4:36 PM, Robert Coli  wrote:
> 
> On Wed, Jun 10, 2015 at 4:15 PM, Jeff Ferland  > wrote:
> Compaction finished yesterday, but I still have this going on:
> 
> 1) What version?
> 2) Probably file a JIRA, esp. if you can repro on another node?
> 3) if you do 2, reply to thread so we can track!
> 
> =Rob

Re: Files not removed after compaction

2015-06-10 Thread Robert Coli 
On Wed, Jun 10, 2015 at 4:15 PM, Jeff Ferland  wrote:

> Compaction finished yesterday, but I still have this going on:
>

1) What version?
2) Probably file a JIRA, esp. if you can repro on another node?
3) if you do 2, reply to thread so we can track!

=Rob

Files not removed after compaction

2015-06-10 Thread Jeff Ferland 
Compaction finished yesterday, but I still have this going on:

Space used (live), bytes:   878681716067
Space used (total), bytes: 2227857083852

jbf@ip-10-0-2-98:/ebs/cassandra/data/trends/trends$ sudo lsof *-Data.db 
COMMAND  PID  USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
java4473 cassandra  446r   REG   0,26  17582559172 39241 
trends-trends-jb-144864-Data.db
java4473 cassandra  448r   REG   0,26 62040962 37431 
trends-trends-jb-144731-Data.db
java4473 cassandra  449r   REG   0,26 829935047545 21150 
trends-trends-jb-143581-Data.db
java4473 cassandra  452r   REG   0,26  8980406 39503 
trends-trends-jb-144882-Data.db
java4473 cassandra  454r   REG   0,26  8980406 39503 
trends-trends-jb-144882-Data.db
java4473 cassandra  462r   REG   0,26  9487703 39542 
trends-trends-jb-144883-Data.db
java4473 cassandra  463r   REG   0,26 36158226 39629 
trends-trends-jb-144889-Data.db
java4473 cassandra  468r   REG   0,26105693505 39447 
trends-trends-jb-144881-Data.db
java4473 cassandra  530r   REG   0,26  17582559172 39241 
trends-trends-jb-144864-Data.db
java4473 cassandra  535r   REG   0,26105693505 39447 
trends-trends-jb-144881-Data.db
java4473 cassandra  542r   REG   0,26  9487703 39542 
trends-trends-jb-144883-Data.db
java4473 cassandra  553u   REG   0,26   6431729821 39556 
trends-trends-tmp-jb-144884-Data.db
jbf@ip-10-0-2-98:/ebs/cassandra/data/trends/trends$ ls *-Data.db
trends-trends-jb-142631-Data.db  trends-trends-jb-143562-Data.db  
trends-trends-jb-143581-Data.db  trends-trends-jb-144731-Data.db  
trends-trends-jb-144883-Data.db
trends-trends-jb-142633-Data.db  trends-trends-jb-143563-Data.db  
trends-trends-jb-144530-Data.db  trends-trends-jb-144864-Data.db  
trends-trends-jb-144889-Data.db
trends-trends-jb-143026-Data.db  trends-trends-jb-143564-Data.db  
trends-trends-jb-144551-Data.db  trends-trends-jb-144881-Data.db  
trends-trends-tmp-jb-144884-Data.db
trends-trends-jb-143533-Data.db  trends-trends-jb-143578-Data.db  
trends-trends-jb-144552-Data.db  trends-trends-jb-144882-Data.db
jbf@ip-10-0-2-98:/ebs/cassandra/data/trends/trends$ cd -
/mnt/cassandra/data/trends/trends
jbf@ip-10-0-2-98:/mnt/cassandra/data/trends/trends$ sudo lsof * 
jbf@ip-10-0-2-98:/mnt/cassandra/data/trends/trends$ ls *-Data.db
trends-trends-jb-124502-Data.db  trends-trends-jb-141113-Data.db  
trends-trends-jb-141377-Data.db  trends-trends-jb-141846-Data.db  
trends-trends-jb-144890-Data.db
trends-trends-jb-125457-Data.db  trends-trends-jb-141123-Data.db  
trends-trends-jb-141391-Data.db  trends-trends-jb-141871-Data.db  
trends-trends-jb-41121-Data.db
trends-trends-jb-130016-Data.db  trends-trends-jb-141137-Data.db  
trends-trends-jb-141538-Data.db  trends-trends-jb-141883-Data.db  
trends-trends.trends_date_idx-jb-2100-Data.db
trends-trends-jb-139563-Data.db  trends-trends-jb-141358-Data.db  
trends-trends-jb-141806-Data.db  trends-trends-jb-142033-Data.db
trends-trends-jb-141102-Data.db  trends-trends-jb-141363-Data.db  
trends-trends-jb-141829-Data.db  trends-trends-jb-144553-Data.db

Essentially, everything was brought down in a major compaction but Cassandra 
won’t let go of the files and I can’t figure out as to why they’re not being 
cleaned up even when they’re not open. I’d really like to get those files 
cleared out safely and rsync away from the EBS volume I used to survive a 
compaction.

While I’m at it, is there any way to anti-compact a table by year? Since I’ve 
switched this recently to date tiered compaction, breaking up that big one 
would probably be helpful.

-Jeff