Re: Uneven distribution using RP

2010-06-23 Thread Eric Evans
On Tue, 2010-06-22 at 17:47 -0400, James Golick wrote: It's also flushing memtables really quickly for a particular CF. Like, really quickly. Like, one every minute. I increased the thresholds by 10x and it's still going fast. What is MemtableFlushAfterMinutes set to? -- Eric Evans

Re: Uneven distribution using RP

2010-06-22 Thread James Golick
This node's load is now growing at a ridiculous rate. It is at 105GB, with the next most loaded node at 70.63GB. Given that RF=3, I would assume that the replicas' nodes would grow relatively quickly too? On Mon, Jun 21, 2010 at 6:44 AM, aaron morton aa...@thelastpickle.comwrote: According to

Re: Uneven distribution using RP

2010-06-22 Thread Robert Coli
On 6/22/10 10:07 AM, James Golick wrote: This node's load is now growing at a ridiculous rate. It is at 105GB, with the next most loaded node at 70.63GB. Given that RF=3, I would assume that the replicas' nodes would grow relatively quickly too? What Replica Placement Strategy are you using

Re: Uneven distribution using RP

2010-06-22 Thread James Golick
RackUnaware, currently On Tue, Jun 22, 2010 at 1:26 PM, Robert Coli rc...@digg.com wrote: On 6/22/10 10:07 AM, James Golick wrote: This node's load is now growing at a ridiculous rate. It is at 105GB, with the next most loaded node at 70.63GB. Given that RF=3, I would assume that the

Re: Uneven distribution using RP

2010-06-22 Thread James Golick
Turns out that this is due to a larger proportion of the wide rows in the system being located on that node. I moved its token over a little to compensate for it, but it doesn't seem to have helped at this point. What's confusing about this is that RF=3 and no other node's load is growing as

Re: Uneven distribution using RP

2010-06-22 Thread Jeremy Dunck
On Tue, Jun 22, 2010 at 4:08 PM, James Golick jamesgol...@gmail.com wrote: Turns out that this is due to a larger proportion of the wide rows in the system being located on that node. I moved its token over a little to compensate for it, but it doesn't seem to have helped at this point. What's

Re: Uneven distribution using RP

2010-06-22 Thread James Golick
It's compacting at a ridiculously fast rate. The pending compactions have been growing for a while. It's also flushing memtables really quickly for a particular CF. Like, really quickly. Like, one every minute. I increased the thresholds by 10x and it's still going fast. On Tue, Jun 22, 2010 at

Re: Uneven distribution using RP

2010-06-21 Thread aaron morton
According to http://wiki.apache.org/cassandra/Operations nodetool repair is used to perform a major compaction and compare data between the nodes, repairing any conflicts. Not sure that would improve the load balance, though it may reduce some wasted space on the nodes. nodetool loadbalance

Uneven distribution using RP

2010-06-20 Thread James Golick
I just increased my cluster from 2 to 4 nodes, and RF=2 to RF=3, using RP. The tokens seem pretty even on the ring, but two of the nodes are far more heavily loaded than the others. I understand that there are a variety of possible reasons for this, but I'm wondering whether anybody has

Re: Uneven distribution using RP

2010-06-20 Thread Jordan Pittier - Rezel
Hi, Have you tried nodetool repair (or cleanup) on your nodes ? On Sun, Jun 20, 2010 at 4:16 PM, James Golick jamesgol...@gmail.com wrote: I just increased my cluster from 2 to 4 nodes, and RF=2 to RF=3, using RP. The tokens seem pretty even on the ring, but two of the nodes are far more

Re: Uneven distribution using RP

2010-06-20 Thread James Golick
I ran cleanup on all of them and the distribution looked roughly even after that, but a couple of days later, it's looking pretty uneven. On Sun, Jun 20, 2010 at 10:21 AM, Jordan Pittier - Rezel jor...@rezel.netwrote: Hi, Have you tried nodetool repair (or cleanup) on your nodes ? On Sun,

Re: Uneven distribution using RP

2010-06-20 Thread Jordan Pittier - Rezel
Node 1 should have token 42535295865117307932921825928971026432 and node 3 127605887595351923798765477786913079296 according to the formula i * (2**127 / 4) for i=1..4 On Sun, Jun 20, 2010 at 4:31 PM, James Golick jamesgol...@gmail.com wrote: I ran cleanup on all of them and the distribution

Re: Uneven distribution using RP

2010-06-20 Thread James Golick
I know, but that's not a big enough difference to warrant the huge amount of difference in load. On Sun, Jun 20, 2010 at 10:53 AM, Jordan Pittier - Rezel jor...@rezel.netwrote: Node 1 should have token 42535295865117307932921825928971026432 and node 3 127605887595351923798765477786913079296