Re: Files not deleted after compaction and GCed
I think this might be what happening. Since you are using ScheduledThreadPoolExecutor.schedule(), the exception was swallowed by the FutureTask. You will have to perform a get() method on the ScheduledFuture, and you will get ExecutionException if there was any exception occured in run(). Regards, Chen On Wed, Jan 26, 2011 at 10:50 AM, Jonathan Ellis wrote: > Patch submitted. > > One thing I still don't understand is why > RetryingScheduledThreadPoolExecutor isn't firing the > DefaultUncaughtExceptionHandler, which should have logged that > exception. > > On Wed, Jan 26, 2011 at 9:41 AM, Jonathan Ellis wrote: > > Thanks for tracking that down! Created > > https://issues.apache.org/jira/browse/CASSANDRA-2059 to fix. > > > > On Wed, Jan 26, 2011 at 8:17 AM, Ching-Cheng Chen > > wrote: > >> It's a bug. > >> In SSTableDeletingReference, it try this operation > >> components.remove(Component.DATA); > >> before > >> STable.delete(desc, components); > >> However, the components was reference to the components object which was > >> created inside SSTable by > >> this.components = Collections.unmodifiableSet(dataComponents); > >> As you can see, you can't try the remove operation on that componets > object. > >> If I add a try block and output exception around the > >> components.remove(Component.DATA), I got this. > >> java.lang.UnsupportedOperationException > >> at java.util.Collections$UnmodifiableCollection.remove(Unknown > >> Source) > >> at > >> > org.apache.cassandra.io.sstable.SSTableDeletingReference$CleanupTask.run(SSTableDeletingReference.java:103) > >> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown > >> Source) > >> at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) > >> at java.util.concurrent.FutureTask.run(Unknown Source) > >> at > >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown > >> Source) > >> at > >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown > >> Source) > >> at > >> > org.apache.cassandra.concurrent.RetryingScheduledThreadPoolExecutor$LoggingScheduledFuture.run(RetryingScheduledThreadPoolExecutor.java:81) > >> at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown > >> Source) > >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > >> Source) > >> at java.lang.Thread.run(Unknown Source) > >> Regards, > >> Chen > >> On Tue, Jan 25, 2011 at 4:21 PM, Jonathan Ellis > wrote: > >>> > >>> the other component types are deleted by this line: > >>> > >>>SSTable.delete(desc, components); > >>> > >>> On Tue, Jan 25, 2011 at 3:11 PM, Ching-Cheng Chen > >>> wrote: > >>> > Nope, no exception at all. > >>> > But if the same class > >>> > (org.apache.cassandra.io.sstable.SSTableDeletingReference) is > >>> > responsible > >>> > for delete other files, then that's not right. > >>> > I checked the source code for SSTableDeletingReference, doesn't looks > >>> > like > >>> > it will delete other files type. > >>> > Regards, > >>> > Chen > >>> > > >>> > On Tue, Jan 25, 2011 at 4:05 PM, Jonathan Ellis > >>> > wrote: > >>> >> > >>> >> No, that is not expected. All the sstable components are removed in > >>> >> the same method; did you check the log for exceptions? > >>> >> > >>> >> On Tue, Jan 25, 2011 at 2:58 PM, Ching-Cheng Chen > >>> >> wrote: > >>> >> > Using cassandra 0.7.0 > >>> >> > The class org.apache.cassandra.io.sstable.SSTableDeletingReference > >>> >> > only > >>> >> > remove the -Data.db file, but leave the xxx-Compacted, > >>> >> > xxx-Filter.db, > >>> >> > xxx-Index.db and xxx-Statistics.db intact. > >>> >> > And that's the behavior I saw.I ran manual compact then > trigger a > >>> >> > GC > >>> >> > from jconsole. The Data.db file got removed but not the others. > >>> >> > Is this the expected behavior? > >>> >> > Regards, > >>> >> > Chen > >>> >> > >>> >> > >>> >> > >>> >> -- > >>> >> Jonathan Ellis > >>> >> Project Chair, Apache Cassandra > >>> >> co-founder of DataStax, the source for professional Cassandra > support > >>> >> http://www.datastax.com > >>> > > >>> > > >>> > >>> > >>> > >>> -- > >>> Jonathan Ellis > >>> Project Chair, Apache Cassandra > >>> co-founder of DataStax, the source for professional Cassandra support > >>> http://www.datastax.com > >> > >> > > > > > > > > -- > > Jonathan Ellis > > Project Chair, Apache Cassandra > > co-founder of DataStax, the source for professional Cassandra support > > http://www.datastax.com > > > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of DataStax, the source for professional Cassandra support > http://www.datastax.com >
Re: Files not deleted after compaction and GCed
Patch submitted. One thing I still don't understand is why RetryingScheduledThreadPoolExecutor isn't firing the DefaultUncaughtExceptionHandler, which should have logged that exception. On Wed, Jan 26, 2011 at 9:41 AM, Jonathan Ellis wrote: > Thanks for tracking that down! Created > https://issues.apache.org/jira/browse/CASSANDRA-2059 to fix. > > On Wed, Jan 26, 2011 at 8:17 AM, Ching-Cheng Chen > wrote: >> It's a bug. >> In SSTableDeletingReference, it try this operation >> components.remove(Component.DATA); >> before >> STable.delete(desc, components); >> However, the components was reference to the components object which was >> created inside SSTable by >> this.components = Collections.unmodifiableSet(dataComponents); >> As you can see, you can't try the remove operation on that componets object. >> If I add a try block and output exception around the >> components.remove(Component.DATA), I got this. >> java.lang.UnsupportedOperationException >> at java.util.Collections$UnmodifiableCollection.remove(Unknown >> Source) >> at >> org.apache.cassandra.io.sstable.SSTableDeletingReference$CleanupTask.run(SSTableDeletingReference.java:103) >> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown >> Source) >> at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) >> at java.util.concurrent.FutureTask.run(Unknown Source) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown >> Source) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown >> Source) >> at >> org.apache.cassandra.concurrent.RetryingScheduledThreadPoolExecutor$LoggingScheduledFuture.run(RetryingScheduledThreadPoolExecutor.java:81) >> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown >> Source) >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown >> Source) >> at java.lang.Thread.run(Unknown Source) >> Regards, >> Chen >> On Tue, Jan 25, 2011 at 4:21 PM, Jonathan Ellis wrote: >>> >>> the other component types are deleted by this line: >>> >>> SSTable.delete(desc, components); >>> >>> On Tue, Jan 25, 2011 at 3:11 PM, Ching-Cheng Chen >>> wrote: >>> > Nope, no exception at all. >>> > But if the same class >>> > (org.apache.cassandra.io.sstable.SSTableDeletingReference) is >>> > responsible >>> > for delete other files, then that's not right. >>> > I checked the source code for SSTableDeletingReference, doesn't looks >>> > like >>> > it will delete other files type. >>> > Regards, >>> > Chen >>> > >>> > On Tue, Jan 25, 2011 at 4:05 PM, Jonathan Ellis >>> > wrote: >>> >> >>> >> No, that is not expected. All the sstable components are removed in >>> >> the same method; did you check the log for exceptions? >>> >> >>> >> On Tue, Jan 25, 2011 at 2:58 PM, Ching-Cheng Chen >>> >> wrote: >>> >> > Using cassandra 0.7.0 >>> >> > The class org.apache.cassandra.io.sstable.SSTableDeletingReference >>> >> > only >>> >> > remove the -Data.db file, but leave the xxx-Compacted, >>> >> > xxx-Filter.db, >>> >> > xxx-Index.db and xxx-Statistics.db intact. >>> >> > And that's the behavior I saw. I ran manual compact then trigger a >>> >> > GC >>> >> > from jconsole. The Data.db file got removed but not the others. >>> >> > Is this the expected behavior? >>> >> > Regards, >>> >> > Chen >>> >> >>> >> >>> >> >>> >> -- >>> >> Jonathan Ellis >>> >> Project Chair, Apache Cassandra >>> >> co-founder of DataStax, the source for professional Cassandra support >>> >> http://www.datastax.com >>> > >>> > >>> >>> >>> >>> -- >>> Jonathan Ellis >>> Project Chair, Apache Cassandra >>> co-founder of DataStax, the source for professional Cassandra support >>> http://www.datastax.com >> >> > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of DataStax, the source for professional Cassandra support > http://www.datastax.com > -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Re: Files not deleted after compaction and GCed
Thanks for tracking that down! Created https://issues.apache.org/jira/browse/CASSANDRA-2059 to fix. On Wed, Jan 26, 2011 at 8:17 AM, Ching-Cheng Chen wrote: > It's a bug. > In SSTableDeletingReference, it try this operation > components.remove(Component.DATA); > before > STable.delete(desc, components); > However, the components was reference to the components object which was > created inside SSTable by > this.components = Collections.unmodifiableSet(dataComponents); > As you can see, you can't try the remove operation on that componets object. > If I add a try block and output exception around the > components.remove(Component.DATA), I got this. > java.lang.UnsupportedOperationException > at java.util.Collections$UnmodifiableCollection.remove(Unknown > Source) > at > org.apache.cassandra.io.sstable.SSTableDeletingReference$CleanupTask.run(SSTableDeletingReference.java:103) > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown > Source) > at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) > at java.util.concurrent.FutureTask.run(Unknown Source) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown > Source) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown > Source) > at > org.apache.cassandra.concurrent.RetryingScheduledThreadPoolExecutor$LoggingScheduledFuture.run(RetryingScheduledThreadPoolExecutor.java:81) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown > Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > Source) > at java.lang.Thread.run(Unknown Source) > Regards, > Chen > On Tue, Jan 25, 2011 at 4:21 PM, Jonathan Ellis wrote: >> >> the other component types are deleted by this line: >> >> SSTable.delete(desc, components); >> >> On Tue, Jan 25, 2011 at 3:11 PM, Ching-Cheng Chen >> wrote: >> > Nope, no exception at all. >> > But if the same class >> > (org.apache.cassandra.io.sstable.SSTableDeletingReference) is >> > responsible >> > for delete other files, then that's not right. >> > I checked the source code for SSTableDeletingReference, doesn't looks >> > like >> > it will delete other files type. >> > Regards, >> > Chen >> > >> > On Tue, Jan 25, 2011 at 4:05 PM, Jonathan Ellis >> > wrote: >> >> >> >> No, that is not expected. All the sstable components are removed in >> >> the same method; did you check the log for exceptions? >> >> >> >> On Tue, Jan 25, 2011 at 2:58 PM, Ching-Cheng Chen >> >> wrote: >> >> > Using cassandra 0.7.0 >> >> > The class org.apache.cassandra.io.sstable.SSTableDeletingReference >> >> > only >> >> > remove the -Data.db file, but leave the xxx-Compacted, >> >> > xxx-Filter.db, >> >> > xxx-Index.db and xxx-Statistics.db intact. >> >> > And that's the behavior I saw. I ran manual compact then trigger a >> >> > GC >> >> > from jconsole. The Data.db file got removed but not the others. >> >> > Is this the expected behavior? >> >> > Regards, >> >> > Chen >> >> >> >> >> >> >> >> -- >> >> Jonathan Ellis >> >> Project Chair, Apache Cassandra >> >> co-founder of DataStax, the source for professional Cassandra support >> >> http://www.datastax.com >> > >> > >> >> >> >> -- >> Jonathan Ellis >> Project Chair, Apache Cassandra >> co-founder of DataStax, the source for professional Cassandra support >> http://www.datastax.com > > -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Re: Files not deleted after compaction and GCed
It's a bug. In SSTableDeletingReference, it try this operation components.remove(Component.DATA); before STable.delete(desc, components); However, the components was reference to the components object which was created inside SSTable by this.components = Collections.unmodifiableSet(dataComponents); As you can see, you can't try the remove operation on that componets object. If I add a try block and output exception around the components.remove(Component.DATA), I got this. java.lang.UnsupportedOperationException at java.util.Collections$UnmodifiableCollection.remove(Unknown Source) at org.apache.cassandra.io.sstable.SSTableDeletingReference$CleanupTask.run(SSTableDeletingReference.java:103) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at org.apache.cassandra.concurrent.RetryingScheduledThreadPoolExecutor$LoggingScheduledFuture.run(RetryingScheduledThreadPoolExecutor.java:81) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Regards, Chen On Tue, Jan 25, 2011 at 4:21 PM, Jonathan Ellis wrote: > the other component types are deleted by this line: > >SSTable.delete(desc, components); > > On Tue, Jan 25, 2011 at 3:11 PM, Ching-Cheng Chen > wrote: > > Nope, no exception at all. > > But if the same class > > (org.apache.cassandra.io.sstable.SSTableDeletingReference) is responsible > > for delete other files, then that's not right. > > I checked the source code for SSTableDeletingReference, doesn't looks > like > > it will delete other files type. > > Regards, > > Chen > > > > On Tue, Jan 25, 2011 at 4:05 PM, Jonathan Ellis > wrote: > >> > >> No, that is not expected. All the sstable components are removed in > >> the same method; did you check the log for exceptions? > >> > >> On Tue, Jan 25, 2011 at 2:58 PM, Ching-Cheng Chen > >> wrote: > >> > Using cassandra 0.7.0 > >> > The class org.apache.cassandra.io.sstable.SSTableDeletingReference > only > >> > remove the -Data.db file, but leave the xxx-Compacted, > >> > xxx-Filter.db, > >> > xxx-Index.db and xxx-Statistics.db intact. > >> > And that's the behavior I saw.I ran manual compact then trigger a > GC > >> > from jconsole. The Data.db file got removed but not the others. > >> > Is this the expected behavior? > >> > Regards, > >> > Chen > >> > >> > >> > >> -- > >> Jonathan Ellis > >> Project Chair, Apache Cassandra > >> co-founder of DataStax, the source for professional Cassandra support > >> http://www.datastax.com > > > > > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of DataStax, the source for professional Cassandra support > http://www.datastax.com >
Re: Files not deleted after compaction and GCed
the other component types are deleted by this line: SSTable.delete(desc, components); On Tue, Jan 25, 2011 at 3:11 PM, Ching-Cheng Chen wrote: > Nope, no exception at all. > But if the same class > (org.apache.cassandra.io.sstable.SSTableDeletingReference) is responsible > for delete other files, then that's not right. > I checked the source code for SSTableDeletingReference, doesn't looks like > it will delete other files type. > Regards, > Chen > > On Tue, Jan 25, 2011 at 4:05 PM, Jonathan Ellis wrote: >> >> No, that is not expected. All the sstable components are removed in >> the same method; did you check the log for exceptions? >> >> On Tue, Jan 25, 2011 at 2:58 PM, Ching-Cheng Chen >> wrote: >> > Using cassandra 0.7.0 >> > The class org.apache.cassandra.io.sstable.SSTableDeletingReference only >> > remove the -Data.db file, but leave the xxx-Compacted, >> > xxx-Filter.db, >> > xxx-Index.db and xxx-Statistics.db intact. >> > And that's the behavior I saw. I ran manual compact then trigger a GC >> > from jconsole. The Data.db file got removed but not the others. >> > Is this the expected behavior? >> > Regards, >> > Chen >> >> >> >> -- >> Jonathan Ellis >> Project Chair, Apache Cassandra >> co-founder of DataStax, the source for professional Cassandra support >> http://www.datastax.com > > -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Re: Files not deleted after compaction and GCed
Nope, no exception at all. But if the same class (org.apache.cassandra.io.sstable.SSTableDeletingReference) is responsible for delete other files, then that's not right. I checked the source code for SSTableDeletingReference, doesn't looks like it will delete other files type. Regards, Chen On Tue, Jan 25, 2011 at 4:05 PM, Jonathan Ellis wrote: > No, that is not expected. All the sstable components are removed in > the same method; did you check the log for exceptions? > > On Tue, Jan 25, 2011 at 2:58 PM, Ching-Cheng Chen > wrote: > > Using cassandra 0.7.0 > > The class org.apache.cassandra.io.sstable.SSTableDeletingReference only > > remove the -Data.db file, but leave the xxx-Compacted, xxx-Filter.db, > > xxx-Index.db and xxx-Statistics.db intact. > > And that's the behavior I saw.I ran manual compact then trigger a GC > > from jconsole. The Data.db file got removed but not the others. > > Is this the expected behavior? > > Regards, > > Chen > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of DataStax, the source for professional Cassandra support > http://www.datastax.com >
Re: Files not deleted after compaction and GCed
No, that is not expected. All the sstable components are removed in the same method; did you check the log for exceptions? On Tue, Jan 25, 2011 at 2:58 PM, Ching-Cheng Chen wrote: > Using cassandra 0.7.0 > The class org.apache.cassandra.io.sstable.SSTableDeletingReference only > remove the -Data.db file, but leave the xxx-Compacted, xxx-Filter.db, > xxx-Index.db and xxx-Statistics.db intact. > And that's the behavior I saw. I ran manual compact then trigger a GC > from jconsole. The Data.db file got removed but not the others. > Is this the expected behavior? > Regards, > Chen -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Files not deleted after compaction and GCed
Using cassandra 0.7.0 The class org.apache.cassandra.io.sstable.SSTableDeletingReference only remove the -Data.db file, but leave the xxx-Compacted, xxx-Filter.db, xxx-Index.db and xxx-Statistics.db intact. And that's the behavior I saw.I ran manual compact then trigger a GC from jconsole. The Data.db file got removed but not the others. Is this the expected behavior? Regards, Chen
