On Thu, Jun 2, 2016 at 11:31 AM, Pete VanderGiessen <[email protected]> wrote: > Hi All, > > I'm a dev at Canonical working on a Bigtop-based Zookeeper charm, and I > have two questions about Zookeeper under Bigtop:
Hi! my name is Roman. I wrote the original Puppet code for ZK. And I cheated ;-) Ok... more inline > 1) What's the recommended way for an Ops person to increase quorum in a > Zookeeper cluster that has been deployed via Bigtop? The truth is -- there's no easy way. Zookeeper nodes are made aware of their identity from the start and that identity is very sticky. If in other systems all nodes are created equal in ZK they are all unique. Theoretically, you can add nodes to the ZK cluster (provided that you increment the identity ID) but if the node goes down you have to maintain the identity mapping somehow. Which leads to a situation where most ZK cluster are very static in nature. You only bring nodes back from the dead -- you don't really scale cluster up and down. > Right now, my charm overrides hadoop_zookeeper::server::ensemble with a list > of the IP > addresses that it knows about, including the IP address of the node that > we're running on, and then re-runs puppet. That gives me what looks like > the correct result in zoo.cfg. I wind up with a list of servers like this: > > server.0=<some ip address>:<port>:<port> > server.1=<some ip address>:<port>:<port> > ... > > This results in a failure to join quorum on each of the servers, with the > following errors in the logs: > > 2016-06-02 18:26:01,195 [myid:0] - INFO > [WorkerSender[myid=0]:QuorumCnxManager@193] - Have smaller server > identifier, so dropping the connection: (1, 0) > > Am I missing something subtle about the ordering, or the way that I need to > specify the server indexes in that list? Yes. For all those servers you also has to maintain a corresponding identiy ID. If those don't match -- you're out of luck. See above. The Puppet code is currently cheating by assigning the static ID to everything. This needs to be fixed, but it isn't easy. Especially for the dynamic case. > 2) What is the simplest way to start Zookeeper REST? Bigtop installs the > zookeeper-rest package via apt, and I wind up with a zookeeper_rest.svc > file which seems to specify a service (for runit, maybe?), but I don't wind > up with a command anywhere in my path, or init init.d that will start the > REST service. I haven't seen anybody use that in production. What do you need it for? Thanks, Roman.
