Ok, great. This has been really helpful, big thanks!
2014-06-25 22:46 GMT+03:00 Alexander Shraer <[email protected]>: > no problem. yes - the leader only counts acks from followers in the config. > > Alex > > > > On Wed, Jun 25, 2014 at 12:37 PM, Niko Vuokko <[email protected]> > wrote: > > > Thanks for the clarification, > > > > I've actually read that doc already. One more question then: how do I > know > > whether a follower is of voting or non-voting kind? Is it simply > equivalent > > to seeing or not seeing the follower as part of the quorum's > configuration? > > > > niko > > > > > > > > 2014-06-25 13:27 GMT+03:00 Alexander Shraer <[email protected]>: > > > > > Hi Niko, > > > > > > Please see documentation here: > > > https://issues.apache.org/jira/browse/ZOOKEEPER-1660 > > > > > > When adding servers to an ensemble, we use the "feature" where the > leader > > > allows any server to > > > connect. After a new server connects and syncs with the leader one can > > > invoke the reconfig command > > > to actually make it part of the ensemble (the actual requirement is > that > > a > > > quorum of the new config has to > > > be in sync with the leader when the reconfig is invoked, otherwise it > > will > > > fail). > > > > > > So yes - any server can connect as a non-voting follower. Its not part > of > > > the ensemble. This was the case before dynamic reconfig support, but > now > > > we're actually making sure it can't vote. > > > > > > > > > Alex > > > > > > > > > > > > > > > > > > > > > On Wed, Jun 25, 2014 at 12:40 PM, Niko Vuokko <[email protected]> > > > wrote: > > > > > > > Another reconfiguration weirdness question: > > > > Steps: Three servers in total, servers 1 and 2 know only about > > > themselves, > > > > server 3 knows everyone. I first start up 1 and 2. After quorum is > > > formed I > > > > start 3. > > > > > > > > Expected: server 3 cannot join or at most becomes an observer. > > > > Actually: server 3 is allowed in to the quorum as follower and I can > do > > > > writes through it. The configuration, however, doesn't include > server 3 > > > > (4-letter conf or zkCli config) as all three servers list only 1 and > 2. > > > > > > > > I think I saw some discussion about accepting unannounced new > members, > > > but > > > > couldn't find it now. Even then, the new member obviously should be > > part > > > of > > > > the official configuration. > > > > > > > > BR, > > > > Niko Vuokko > > > > > > > > > > > > > > > > 2014-06-25 10:10 GMT+03:00 Niko Vuokko <[email protected]>: > > > > > > > > > JIRA+patch available @ > > > > > https://issues.apache.org/jira/browse/ZOOKEEPER-1946 > > > > > > > > > > > > > > > 2014-06-19 17:52 GMT+03:00 Alexander Shraer <[email protected]>: > > > > > > > > > > Hi, > > > > >> > > > > >> 1) yes. The only consistent way to change a configuration is to > > agree > > > > on a > > > > >> new one and currently you need to use the reconfig api to do that. > > > > >> 2) yeah seems confusing indeed. It probably makes sense to update > > the > > > > >> printed info when the config is updated (in this case when it > learns > > > it > > > > >> during sync). Can you open a JIRA for this? If you can take a stab > > at > > > > >> fixing it I'll review it. > > > > >> > > > > >> Thanks > > > > >> Alex > > > > >> On Jun 19, 2014 4:52 PM, "Niko Vuokko" <[email protected]> > > wrote: > > > > >> > > > > >> > I basically try to simulate having a clean slate server coming > up > > > with > > > > >> > myid=3. So it would have no data, no dynamic configuration file > > > since > > > > >> the > > > > >> > ZK service has never run. I was hoping to test the > reconfiguration > > > API > > > > >> to > > > > >> > let the new server 3 in to the quorum, but then this happened. > > > > >> > > > > > >> > "Static configuration" = $ZOOCFG on service start > > > > >> > Reconfiguration API used: Not yet... > > > > >> > Manual change: Yes, I kill server 3 manually, delete all its > data, > > > > >> change > > > > >> > its configuration and start it up as "new, empty server" > > > > >> > > > > > >> > Yes, the remaining quorum obviously tells the "new" server 3 to > > > > >> configure > > > > >> > itself with the old ports, but that just sounds *really* weird > to > > > me. > > > > >> Two > > > > >> > questions: > > > > >> > > > > > >> > 1) Is it really so that it doesn't matter how I configure the > new > > > > >> member if > > > > >> > the quorum already knows about an old server with the same myid? > > The > > > > old > > > > >> > configuration will just be forced upon the new server even if > that > > > is > > > > >> > unwanted. > > > > >> > 2) All the log entries refer to the new port 2184 although that > is > > > not > > > > >> > actually used. For example, once the "new" server 3 has joined > as > > a > > > > >> > follower, I'm still getting rows like > > > > >> > > > > > >> > 2014-06-19 16:25:10,883 [myid:3] - INFO > > > > >> > [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2184:QuorumPeer@972] - > > > FOLLOWING > > > > >> > > > > > >> > And as I mentioned, there is nothing in port 2184, the "new" > > server > > > 3 > > > > >> > responds at 2183... The problematic piece seems to come from > > > > >> QuorumPeer@857 > > > > >> > where the thread name is set. This bug causes confusion > > (especially > > > if > > > > >> > hostnames change as well), probably nothing worse in most cases > > > > though. > > > > >> > > > > > >> > > > > > >> > (btw, 4-letter word conf gives the currently known member > > > > configuration) > > > > >> > > > > > >> > Thanks for the help, > > > > >> > niko > > > > >> > > > > > >> > > > > > >> > > > > > >> > 2014-06-19 16:22 GMT+03:00 Alexander Shraer <[email protected] > >: > > > > >> > > > > > >> > > when you say "change its static configuration", what exactly > do > > > you > > > > >> mean > > > > >> > ? > > > > >> > > this part of the configuration should be located in the > > membership > > > > >> file > > > > >> > > (dynamic part of the config). do you use the reconfiguration > API > > > to > > > > >> > change > > > > >> > > it (this is the right way) ? do you manually change it at > > all/some > > > > of > > > > >> the > > > > >> > > servers (this would have no effect because servers read these > > > files > > > > >> only > > > > >> > > when they boot) ? > > > > >> > > > > > > >> > > In the scenario you describe I expect server 3 to come up. Its > > > > >> possible > > > > >> > if > > > > >> > > you changed the config files manually that during sync with > the > > > > leader > > > > >> > the > > > > >> > > leader pushes the old config to server 3 that it has in memory > > > > (since > > > > >> the > > > > >> > > files you updated are not read) that's why it effectively has > > the > > > > old > > > > >> > > parameters. > > > > >> > > > > > > >> > > Please use the dynamic reconfig API when you make changes to > > > > existing > > > > >> > > servers or add/remove servers. > > > > >> > > > > > > >> > > the "config" command in CLI can show you what config is latest > > at > > > > the > > > > >> > > server you're connected too (if I remember correctly there's > > also > > > a > > > > 4 > > > > >> > > letter command). You can also check the config file(s) of > > server 3 > > > > >> after > > > > >> > it > > > > >> > > syncs with the leader. I suspect that it will contain the old > > > > config. > > > > >> > > > > > > >> > > > > > > >> > > Alex > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > On Thu, Jun 19, 2014 at 2:03 PM, Niko Vuokko < > > > [email protected] > > > > > > > > > >> > > wrote: > > > > >> > > > > > > >> > > > Starting from a stable 3-member quorum: > > > > >> > > > > > > > >> > > > server.1=localhost:2801:3801;2181 > > > > >> > > > server.2=localhost:2802:3802;2182 > > > > >> > > > server.3=localhost:2803:3803;2183 > > > > >> > > > > > > > >> > > > I then kill server 3, clear its data directory, keep its > > myid=3 > > > > and > > > > >> > > change > > > > >> > > > its static configuration to > > > > >> > > > > > > > >> > > > server.1=localhost:2801:3801;2181 > > > > >> > > > server.2=localhost:2802:3802;2182 > > > > >> > > > server.3=localhost:2804:3804;2184 > > > > >> > > > > > > > >> > > > Now what I would expect is that this "new" server 3 will not > > > join > > > > >> the > > > > >> > > > quorum since the ports don't match what the servers 1 and 2 > > > > expect. > > > > >> > > > However, it can join. The problem is that the "new" server 3 > > > does > > > > >> not > > > > >> > > > respect its configuration. Its logs will contain the new > port > > > > number > > > > >> > > 2184, > > > > >> > > > but it will actually pick up the dynamic configuration > offered > > > by > > > > >> the > > > > >> > > > quorum and open up the old ports 2183 etc. After joining > > again, > > > > the > > > > >> > > dynamic > > > > >> > > > configuration file for server 3 contains > > > > >> > > > > > > > >> > > > server.3=localhost:2803:3803:participant;0.0.0.0:2183 > > > > >> > > > > > > > >> > > > Also, echo conf | localhost 2184 never replies but echo > conf | > > > > >> > localhost > > > > >> > > > 2183 returns > > > > >> > > > > > > > >> > > > server.3=localhost:2803:3803:participant;0.0.0.0:2183 > > > > >> > > > > > > > >> > > > Is this actually intentional or a bug? > > > > >> > > > > > > > >> > > > > > > > >> > > > Best, > > > > >> > > > Niko Vuokko > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > > > >
