I've somehow now ended up in a very strange place...
If I ask '150' or '155' about the ring they report each other, but if I
ask the rest of the ring they have '155' but not '150' as members. All
of the storage-conf files are basically clones of each other with the
same ring masters.
$ nodetool -h 10.1.0.155 ring
Address Status Load
Range Ring
99811300882272441299595351868344045866
10.3.0.150 Up 3.08 TB
6436333895300580402214871779965756352 |<--|
10.1.0.155 Up 3.08 TB
99811300882272441299595351868344045866 |-->|
$ nodetool -h 10.2.0.174 ring
Address Status Load
Range Ring
144951579690133260853298391132870993575
10.2.0.115 Up 2.7 TB
55758122058160717108501182340054262660 |<--|
10.1.0.155 Up 3.08 TB
99811300882272441299595351868344045866 | ^
10.2.0.174 Up 1.32 TB
118283207506463595491596277948095451613 v |
10.3.0.151 Up 414.51 GB
127520031787005730998588483181387651399 | ^
10.3.0.152 Up 143.03 GB
132137258578258111824507171284723589567 v |
10.3.0.153 Up 245.51 GB
134446064220575108370358944111505967571 | ^
10.2.0.175 Up 3.16 TB
136754979922617117666448707835107404441 v |
10.2.0.114 Up 1.41 TB
144951579690133260853298391132870993575 |-->|
$ nodetool -h 10.3.0.150 streams
Mode: Normal
Not sending any streams.
Not receiving any streams.
$ nodetool -h 10.1.0.155 streams
Mode: Normal
Not sending any streams.
Not receiving any streams.
$ nodetool -h 10.2.0.115 streams
Mode: Normal
Not sending any streams.
Not receiving any streams.
On 5/11/10 6:30 AM, Jonathan Ellis wrote:
s/keyspace/token/ and you've got it.
On Mon, May 10, 2010 at 10:34 AM, David Koblas<kob...@extra.com> wrote:
Sounds great, will give it a go. However, just to make sure I understand
getting the keyspace correct.
Lets say I've got:
A -- Node before overfull node in keyspace order
O -- Overfull node
B -- Node after O in keyspace order
N -- New empty node
I'm going to assume that I should make the following assignment:
keyspace(N) = keyspace(A) + ( keyspace(O) - keyspace(A) ) / 2
Or did I miss something else about keyspace ranges?
Thanks
On 5/7/10 1:25 PM, Jonathan Ellis wrote:
If you're using RackUnawareStrategy (the default replication strategy)
then you can "bootstrap" manually fairly easily -- copy all the data
(not system) sstables from an overfull machine to a new machine,
assign the new one a token that gives it about half of the old node's
range, then start it with autobootstrap OFF. Then run cleanup on both
new and old nodes to remove the part of the data that belongs to the
other.
The downside vs real bootstrap is you can't do this safely while
writes are coming in to the original node. You can reduce your
read-only period by doing an intial scp, then doing a flush + rsync
when you're ready to take it read only.
(https://issues.apache.org/jira/browse/CASSANDRA-579 will make this
problem obsolete for 0.7 but that doesn't help you on 0.6, of course.)
On Fri, May 7, 2010 at 2:08 PM, David Koblas<kob...@extra.com> wrote:
I've got two (out of five) nodes on my cassandra ring that somehow got
too
full (e.g. over 60% disk space utilization). I've now gotten a few new
machines added to the ring, but evertime one of the overfull nodes
attempts
to stream its data it runs out of diskspace... I've tried half a dozen
different bad ideas of how to get things moving along a bit smoother, but
am
at a total loss at this point.
Is there any good tricks to get cassandra to not need 2x the disk space
to
stream out, or is something else potentially going on that's causing me
problems?
Thanks,
