Re: Removes increasing disk space usage in Cassandra?

2009-12-04 Thread Ramzi Rabah
Done https://issues.apache.org/jira/browse/CASSANDRA-604 On Fri, Dec 4, 2009 at 4:01 PM, Jonathan Ellis wrote: > Please do. > > On Fri, Dec 4, 2009 at 5:53 PM, Ramzi Rabah wrote: >> Thanks Jonathan. >> Should I open a bug for this? >> >> Ray >> >> On Fri, Dec 4, 2009 at 3:47 PM, Jonathan Ellis

Re: Removes increasing disk space usage in Cassandra?

2009-12-04 Thread Jonathan Ellis
Please do. On Fri, Dec 4, 2009 at 5:53 PM, Ramzi Rabah wrote: > Thanks Jonathan. > Should I open a bug for this? > > Ray > > On Fri, Dec 4, 2009 at 3:47 PM, Jonathan Ellis wrote: >> On Fri, Dec 4, 2009 at 5:32 PM, Ramzi Rabah wrote: >>> Starting with fresh directories with no data and trying to

Re: Removes increasing disk space usage in Cassandra?

2009-12-04 Thread Ramzi Rabah
Thanks Jonathan. Should I open a bug for this? Ray On Fri, Dec 4, 2009 at 3:47 PM, Jonathan Ellis wrote: > On Fri, Dec 4, 2009 at 5:32 PM, Ramzi Rabah wrote: >> Starting with fresh directories with no data and trying to do simple >> inserts, I could not reproduce it *sigh*. Nothing is simple :(

Re: Removes increasing disk space usage in Cassandra?

2009-12-04 Thread Jonathan Ellis
On Fri, Dec 4, 2009 at 5:32 PM, Ramzi Rabah wrote: > Starting with fresh directories with no data and trying to do simple > inserts, I could not reproduce it *sigh*. Nothing is simple :(, so I > decided to dig deeper into the code. > > I was looking at the code for compaction, and this is a very n

Re: Removes increasing disk space usage in Cassandra?

2009-12-04 Thread Ramzi Rabah
Starting with fresh directories with no data and trying to do simple inserts, I could not reproduce it *sigh*. Nothing is simple :(, so I decided to dig deeper into the code. I was looking at the code for compaction, and this is a very noob concern, so please bare with me if I'm way off, this code

Re: Removes increasing disk space usage in Cassandra?

2009-12-04 Thread Jonathan Ellis
Okay, in that case it doesn't hurt to update just in case but I think you're going to need that test case. :) On Fri, Dec 4, 2009 at 2:45 PM, Ramzi Rabah wrote: > I have a two week old version of trunk. Probably need to update it to > latest build. > > On Fri, Dec 4, 2009 at 12:34 PM, Jonathan El

Re: Removes increasing disk space usage in Cassandra?

2009-12-04 Thread Ramzi Rabah
I have a two week old version of trunk. Probably need to update it to latest build. On Fri, Dec 4, 2009 at 12:34 PM, Jonathan Ellis wrote: > Are you testing trunk?  If not, you should check that first to see if > it's already fixed. > > On Fri, Dec 4, 2009 at 1:55 PM, Ramzi Rabah wrote: >> Just

Re: Removes increasing disk space usage in Cassandra?

2009-12-04 Thread Jonathan Ellis
Are you testing trunk? If not, you should check that first to see if it's already fixed. On Fri, Dec 4, 2009 at 1:55 PM, Ramzi Rabah wrote: > Just to be clear what I meant is that I ran the deletions and > compaction with GCGraceSeconds set to 1 hour, so there was enough time > for the tombstone

Re: Removes increasing disk space usage in Cassandra?

2009-12-04 Thread Ramzi Rabah
Just to be clear what I meant is that I ran the deletions and compaction with GCGraceSeconds set to 1 hour, so there was enough time for the tombstones to expire. Anyway I will try to make a simpler test case to hopefully reproduce this, and I will share the code if I can reproduce. Ray On Fri, D

Re: Removes increasing disk space usage in Cassandra?

2009-12-04 Thread Ramzi Rabah
Hi Jonathan I have changed that to 3600(one hour) based on your recommendation before. On Fri, Dec 4, 2009 at 11:01 AM, Jonathan Ellis wrote: > this is what I was referring to by "the period specified in your config file": > >   >  864000 > > On Fri, Dec 4, 2009 at 12:51 PM, Ramzi Rabah wrote: >

Re: Removes increasing disk space usage in Cassandra?

2009-12-04 Thread Jonathan Ellis
this is what I was referring to by "the period specified in your config file": 864000 On Fri, Dec 4, 2009 at 12:51 PM, Ramzi Rabah wrote: > I think there might be a bug in the deletion logic. I removed all the > data on the cluster by running remove on every single key I entered, > and I ru

Re: Removes increasing disk space usage in Cassandra?

2009-12-04 Thread Ramzi Rabah
I think there might be a bug in the deletion logic. I removed all the data on the cluster by running remove on every single key I entered, and I run major compaction nodeprobe -host hostname compact on a certain node, and after the compaction is over, I am left with one data file/ one index file an

Re: Removes increasing disk space usage in Cassandra?

2009-12-03 Thread Jonathan Ellis
cassandra never modifies data in-place. so it writes tombstones to supress the older writes, and when compaction occurs the data and tombstones get GC'd (after the period specified in your config file). On Thu, Dec 3, 2009 at 8:07 PM, Ramzi Rabah wrote: > Looking at jconsole I see a high number

Re: Removes increasing disk space usage in Cassandra?

2009-12-03 Thread Ramzi Rabah
Looking at jconsole I see a high number of writes when I do removes, so I am guessing these are tombstones being written? If that's the case, is the data being removed and replaced by tombstones? and will they all be deleted eventually when compaction runs? On Thu, Dec 3, 2009 at 3:18 PM, Ramzi

Removes increasing disk space usage in Cassandra?

2009-12-03 Thread Ramzi Rabah
Hi all, I ran a test where I inserted about 1.2 Gigabytes worth of data into each node of a 4 node cluster. I ran a script that first calls a get on each column inserted followed by a remove. Since I was basically removing every entry I inserted before, I expected that the disk space occupied by t