Doh - that makes total sense. For whatever reason I thought with 2 servers you couldn't get a majority :P
On Tue, Jan 12, 2010 at 3:17 PM, Henry Robinson <he...@cloudera.com> wrote: > Hi Adam - > > As long as a quorum of servers is running, ZK will be live. With majority > quorums, 2/3 is enough to keep going. In general, if fewer than half your > nodes have failed, ZK will keep on keeping on. > > The main concern with a cluster of 2/3 machines is that a single further > failure will bring down the whole cluster. > > Henry > > 2010/1/12 Adam Rosien <a...@rosien.net> > >> I have a related question: what's the behavior of a cluster of 3 when >> one is down? I've tried it and a leader is elected, but are there any >> other caveats for this situation? >> >> .. Adam >> >> On Tue, Jan 12, 2010 at 2:40 PM, Patrick Hunt <ph...@apache.org> wrote: >> > 12 servers? That's alot, if you dont' mind my asking why so many? >> Typically >> > we recommend 5 - that way you can have one down for maintenance and still >> > have a failure that doesn't bring down the cluster. >> > >> > The "electing a leader" is probably the restarted machine attempting to >> > re-join the ensemble (it should join as a follower if you have a leader >> > already elected, given that it's xid is behind the existing leader.) Hard >> to >> > tell though without the logs. >> > >> > You might also be seeing the initLimit exceeded, is the data you are >> storing >> > in ZK large? Or perhaps network connectivity is slow? >> > >> http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html#sc_clusterOptions >> > again the logs would give some insight on this. >> > >> > >> > Patrick >> > >> > Nick Bailey wrote: >> >> >> >> We are running zookeeper 3.1.0 >> >> >> >> Recently we noticed the cpu usage on our machines becoming >> >> increasingly high and we believe the cause is >> >> >> >> https://issues.apache.org/jira/browse/ZOOKEEPER-427 >> >> >> >> However our solution when we noticed the problem was to kill the >> >> zookeeper process and restart it. >> >> >> >> After doing that though it looks like the newly restarted zookeeper >> >> server is continually attempting to elect a leader even though one >> >> already exists. >> >> >> >> The process responses with 'imok' when asked, but the stat command >> >> returns 'ZooKeeperServer not running'. >> >> >> >> I belive that killing the current leader should trigger all servers >> >> to do an election and solve the problem, but I'm not sure. Should >> >> that be the course of action in this situation? >> >> >> >> Also we have 12 servers, but 5 are currently not running according to >> >> stat. So I guess this isn't a problem unless we lose another one. >> >> We have plans to upgrade zookeeper to solve the cpu issue but haven't >> >> been able to do that yet. >> >> >> >> Any help appreciated, Nick Bailey >> >> >> > >> >
