Really old files in the data directory

2011-12-09 Thread Maxim Potekhin 

Hello,

a varied the GC grace a few times over the period of my cluster's 
lifetime, but I never went above
10 days. I did compactions, repairs etc. Now, I see that some files in 
the data directories of the nodes
that were there from day one carry timestamps back from July.  There are 
files containing secondary
indexes. But I have deleted a large portion of the data, one would 
expect that these files must have

been rebuilt one or many times. What's happening?

I run 0.8.6.

Thanks

Maxim

Re: Really old files in the data directory

2011-12-09 Thread Edward Capriolo 
Are you taking snapshots? If the file is referenced in a snapshot it will
not delete until it is also not part of any snapshot.

On Fri, Dec 9, 2011 at 1:45 PM, Maxim Potekhin potek...@bnl.gov wrote:

 Hello,

 a varied the GC grace a few times over the period of my cluster's
 lifetime, but I never went above
 10 days. I did compactions, repairs etc. Now, I see that some files in the
 data directories of the nodes
 that were there from day one carry timestamps back from July.  There are
 files containing secondary
 indexes. But I have deleted a large portion of the data, one would expect
 that these files must have
 been rebuilt one or many times. What's happening?

 I run 0.8.6.

 Thanks

 Maxim

Re: Really old files in the data directory

2011-12-09 Thread Brandon Williams 
On Fri, Dec 9, 2011 at 1:57 PM, Edward Capriolo edlinuxg...@gmail.com wrote:
 Are you taking snapshots? If the file is referenced in a snapshot it will
 not delete until it is also not part of any snapshot.

That isn't true.  The file will be removed from the data dir, but
still exist in the snapshot dir.

-Brandon

Re: Really old files in the data directory

2011-12-09 Thread Edward Capriolo 
Aha. I just tested that. Good point. It just seems that way because if I do
not clear my snapshots in production I run into disk space issues rather
quickly. That is what happens when you assume.

On Fri, Dec 9, 2011 at 4:09 PM, Brandon Williams dri...@gmail.com wrote:

 On Fri, Dec 9, 2011 at 1:57 PM, Edward Capriolo edlinuxg...@gmail.com
 wrote:
  Are you taking snapshots? If the file is referenced in a snapshot it will
  not delete until it is also not part of any snapshot.

 That isn't true.  The file will be removed from the data dir, but
 still exist in the snapshot dir.

 -Brandon