as you noticed, "nodeprobe move" first unloads the data, then moves to the new position. so that won't help you here.
If you are using replicationfactor=1, scp the data to the previous node on the ring, then reduce the original node's token so it isn't responsible for so much, and run cleanup. (you can do this w/ higher RF too, you just have to scp the data more places.) Finally, you could work on https://issues.apache.org/jira/browse/CASSANDRA-579 so it doesn't need to anticompact to disk before moving data. -Jonathan On Wed, Feb 24, 2010 at 12:06 PM, shiv shivaji <shivaji...@yahoo.com> wrote: > According to the stack trace I get in the log, it makes it look like the > patch was for anti-compaction but I did not look at the source code in > detail yet. > > java.util.concurrent.ExecutionException: > java.lang.UnsupportedOperationException: disk full > at > java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) > at java.util.concurrent.FutureTask.get(FutureTask.java:83) > at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86) > at > org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:570) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > Caused by: java.lang.UnsupportedOperationException: disk full > at > org.apache.cassandra.db.CompactionManager.doAntiCompaction(CompactionManager.java:344) > at > org.apache.cassandra.db.CompactionManager.doCleanupCompaction(CompactionManager.java:405) > at > org.apache.cassandra.db.CompactionManager.access$400(CompactionManager.java:49) > at > org.apache.cassandra.db.CompactionManager$2.call(CompactionManager.java:130) > at > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > ... 2 more > > I tried "nodetool cleanup" before and it did not really stop the disk from > filling, is there a way to force move the data or some other way to solve > the issue? > > Thanks, Shiv > > ________________________________ > From: Jonathan Ellis <jbel...@gmail.com> > To: cassandra-user@incubator.apache.org > Sent: Wed, February 24, 2010 7:16:32 AM > Subject: Re: Anti-compaction Diskspace issue even when latest patch applied > > The patch you refer to was to help *compaction*, not *anticompaction*. > > If the space is mostly hints for other machines (is that what you > meant by "due to past problems with others?") you should run nodeprobe > cleanup on it to remove data that doesn't actually belong on that > node. > > -Jonathan > > On Wed, Feb 24, 2010 at 3:09 AM, shiv shivaji <shivaji...@yahoo.com> wrote: >> For about 6TB of total data size with a replication factor of 2 (6TB x 2) >> on a five node cluster, I see about 4.6 TB on one machine (due to >> potential >> past problems with other machines). The machine has a disk of 6TB. >> >> The data folder on this machine has 59,289 files totally 4.6 TB. The files >> are the data, filter and indexes. I see that anti-compaction is running. I >> applied a recent patch which does not do anti-compaction if disk space is >> limited. I still see it happening. I have also called nodetool loadbalance >> on this machine. Seems like it will run out of disk space anyway. >> >> The machine diskspace consumed are: (Each machine has a 6TB hard-drive on >> RAID). >> >> Machine Space Consumed >> M1 4.47 TB >> M2 2.93 TB >> M3 1.83 GB >> M4 56.19 GB >> M5 398.01 GB >> >> How can I force M1 to immediately move its load to M3 and M4 for instance >> (or any other machine). The nodetool move command moves all data, is there >> a >> way instead to force move 50% of data to M3 and the remaining 50% to M4 >> and >> resume anti-compaction after the move? >> >> Thanks, Shiv >> >> >> >
