grt thx. on another note, is there a way to know that a node has fully bootstrapped or resync'ed after a restart? meaning it has its slice of the ring, the data replicated from other nodes, etc?
i've glanced thru the JMX properties but didn't see anything. thx! grt work On Tue, 2009-11-24 at 11:21 -0600, Jonathan Ellis wrote: > Looks like this is another symptom of > https://issues.apache.org/jira/browse/CASSANDRA-150, which is on track > to be fixed soon > > On Tue, Nov 24, 2009 at 11:19 AM, B. Todd Burruss <[email protected]> wrote: > > they all were restarted at various times. > > > > for vmguest85 the other three are seed nodes. > > > > > > On Mon, 2009-11-23 at 19:21 -0600, Jonathan Ellis wrote: > >> So vmquest85 was restarted, but gen-app02 hasn't told it that there > >> are 2 other nodes that are down? > >> > >> Which one is the seed node? > >> > >> On Mon, Nov 23, 2009 at 6:38 PM, B. Todd Burruss <[email protected]> wrote: > >> > i'm observing the following on a cluster that started with 4 nodes. i > >> > have > >> > been killing and restarting the various nodes as i test cassandra and now > >> > i'm seeing a lot of NotFoundException exceptions in the client because > >> > what > >> > i believe is ring state out of sync between the two nodes that are still > >> > up > >> > and available. The first ring state shown below reflects the current > >> > state > >> > of the cluster. Also I have seen similar issues when one of the nodes > >> > thinks another node is still available when in fact it has been killed. > >> > it > >> > seems to be related to bringing up, killing nodes too fast and not > >> > letting > >> > them figure out when a node is "dead". in this case i see > >> > TimedOutException > >> > related to NIO SocketChannel class. > >> > > >> > thx! > >> > > >> > [cassandra.883477]$ bin/nodeprobe -host gen-app02.dev.real.com -port 8080 > >> > ring > >> > Address Status Load > >> > Range Ring > >> > > >> > 144038903974614862325597275257769797985 > >> > 172.27.128.186Down 22.17 MB > >> > 31124469348629903091013930339840898757 |<--| > >> > 172.27.128.23 Down 22.17 MB > >> > 64378740291415296162944450043143967518 | | > >> > 172.27.128.22 Up 22.17 MB > >> > 121134220722269938669001112695509564769 | | > >> > 172.27.128.185Up 14.69 MB > >> > 144038903974614862325597275257769797985 |-->| > >> > > >> > [cassandra.883477]$ bin/nodeprobe -host vmguest85.prognet.com -port 8080 > >> > ring > >> > Address Status Load > >> > Range Ring > >> > > >> > 144038903974614862325597275257769797985 > >> > 172.27.128.22 Up 22.17 MB > >> > 121134220722269938669001112695509564769 |<--| > >> > 172.27.128.185Up 14.69 MB > >> > 144038903974614862325597275257769797985 |-->| > >> > [cassandra.883477]$ > >> > > >> > > >> > > > > > > >
