To be fair - the "addition case" would be there either way. If you are adding nodes, the clients would see it only after a while (after you update the client side hbase-site.xml.)
So - knock that off my mail. I think the replacement / re-IP issue is still valid. --Suraj On Sun, Dec 4, 2011 at 9:18 AM, Suraj Varma <[email protected]> wrote: > Thanks for summarizing this - ok, so I see the setup. > > I'm wondering what the implications are: So - let's say you decide to > add more zookeeper nodes or to "replace" a zookeeper node due to > failure or whatever. > > 1) Addition case: > a) You would add 6.zookeeper, 7.zookeeper to hbase-site.xml and bounce > cluster. 6 or 7 could become the leader, potentially > b) You would add 6 and 7 to roundrobindnsentry > c) But - between steps a & b ... would there be any issues if 6 and 7 > are not seen by the clients? > > 2) Replacement case: > a) You replace 5.zookeeper with a new hardware and replace the > 5.zookeeper dns entry with this new IP. > b) You update roundrobindnsentry replacing 5.zookeeper's IP address. > c) Between a & b ... would there be any issues if cluster writes to > new node, but client can't see it? > > I'm guessing that as long as the quorum is met, clients should be ok > with the above partial zk view. Would it still be ok with one of these > new nodes happen to take the leader role? > > We were also planning to setup dns entries for individual zk nodes (so > that we can replace /re-IP as needed), but my thought was that clients > need view of full zk to maintain quorum (even during that window) > > Thanks, > --Suraj > > > On Sun, Dec 4, 2011 at 7:09 AM, Rita <[email protected]> wrote: >> To add, >> >> The correct way to setup your client/application is like this >> >> >> <?xml version="1.0"?> >> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> >> <configuration> >> <property> >> <name>hbase.zookeeper.quorum</name> >> <value>roundrobindnsdnsentry</value> >> <description>The directory shared by region servers. >> </description> >> </property> >> </configuration> >> >> >> Configuration config = HBaseConfiguration.create(); >> config.set("hbase.zookeeper.quorum", "roundrobindnsentry"); // Here >> we are running zookeeper locally >> >> >> On Sun, Dec 4, 2011 at 10:03 AM, Rita <[email protected]> wrote: >> >>> Thanks for the nice responses and advice. >>> >>> To sum up this thread >>> >>> This will not work, >>> >>> $ host roundrobindnsentry >>> roundrobindnsentry has address 192.168.0.2 >>> roundrobindnsentry has address 192.168.0.5 >>> roundrobindnsentry has address 192.168.0.6 >>> >>> >>> $ host roundrobindnsentry >>> roundrobindnsentry has address 192.168.0.5 >>> roundrobindnsentry has address 192.168.0.6 >>> roundrobindnsentry has address 192.168.0.2 >>> >>> >>> Ín your hbase-site.xml >>> >>> <property> >>> <name>hbase.zookeeper.quorum</name> >>> <value>roundrobindnsentry</value> >>> </description> >>> </property> >>> >>> >>> This will not work because when you start up the hbase cluster it will >>> complain the zookeeper is already running. Basically it will get confused >>> because of the round robin DNS entries. >>> >>> However, this WILL work if in your application you set, roundrobingdns >>> entry. >>> >>> >>> >>> >>> >>> >>> >>> On Sat, Dec 3, 2011 at 8:09 PM, Shrijeet Paliwal >>> <[email protected]>wrote: >>> >>>> >>Now any HBase client that needs to connect to zookeeper, can talk to >>>> quorum via "zookeeper-quorum.example.com" >>>> >>>> Must add the reason above works is zookeeper uses >>>> InetAddress.getAllByName >>>> < >>>> http://docs.oracle.com/javase/1.3/docs/api/java/net/InetAddress.html#getAllByName(java.lang.String) >>>> >to >>>> build the server address list >>>> >>>> On Sat, Dec 3, 2011 at 5:06 PM, Shrijeet Paliwal <[email protected] >>>> >wrote: >>>> >>>> > Suraj, >>>> > Combine what J-D & David said. For example in our case >>>> > >>>> > 1. We let HBase manage zookeeper >>>> > 2. Set the config in hbase-site.xml the way David mentioned >>>> > : >>>> <value>0.zookeeper,1.zookeeper,2.zookeeper,3.zookeeper,4.zookeeper</value> >>>> > 3. Also have a DNS entry "zookeeper-quorum.example.com" with >>>> multiple >>>> > IPs (all mentioned above) resolving to this entry >>>> > 4. Now any HBase client that needs to connect to zookeeper, can talk >>>> > to quorum via "zookeeper-quorum.example.com" >>>> > >>>> > One might think why we cant have hbase-site use "" >>>> > zookeeper-quorum.example.com" as well. My understanding is when you >>>> start >>>> > a zookeeper daemon on a server, the hbase managed zookeeper service will >>>> > fetch the the default host name using hadoop's DNS.getDefaultHost >>>> method. >>>> > The hostname returned may or may not find a match in the server list you >>>> > mentioned in hbase-site.xml. If does not match you get the error >>>> mentioned >>>> > here http://wiki.apache.org/hadoop/Hbase/Troubleshooting#A9. >>>> > >>>> > For example, in our case zookeeper host has two hostnames associated >>>> with >>>> > default ethernet interface. First name is host.domain.com and second is >>>> > the special DNS entry I talked about in (4). The DNS.getDefaultHost >>>> returns >>>> > host.domain.com and zookeeper fails to start. >>>> > >>>> > I would be interested if some one has a different way of handling the >>>> > situation I described. >>>> > >>>> > On Sat, Dec 3, 2011 at 4:45 PM, Suraj Varma <[email protected]> >>>> wrote: >>>> > >>>> >> Yes - this makes sense. But, I thought what Rita suggested was a >>>> >> single appquorum dns entry ... which was surprising. >>>> >> >>>> >> Hence my question. >>>> >> --Suraj >>>> >> >>>> >> On Sat, Dec 3, 2011 at 10:40 AM, Dave Barr <[email protected]> >>>> wrote: >>>> >> > What we do is make N.zookeeper.$DC.$DOMAIN entries in DNS. We have >>>> >> > one ZK cluster per DC. Our configs then just point to >>>> >> > >>>> >> > >>>> >> >>>> <value>0.zookeeper,1.zookeeper,2.zookeeper,3.zookeeper,4.zookeeper</value> >>>> >> > >>>> >> > --Dave >>>> >> > >>>> >> > On Sat, Dec 3, 2011 at 6:15 AM, Suraj Varma <[email protected]> >>>> >> wrote: >>>> >> >> J-D: >>>> >> >> Did you mean that a _single_ dns entry returns all five ips >>>> belonging >>>> >> >> to individual zk nodes? >>>> >> >> >>>> >> >> Is this used only by "clients" ... or even within the cluster? >>>> >> >> And ... the zk nodes self-identify by IP ... and is this how region >>>> >> >> server nodes reach out specifically to the "leader" zk node? >>>> >> >> --Suraj >>>> >> >> >>>> >> >> >>>> >> >> On Wed, Nov 30, 2011 at 4:14 PM, Jean-Daniel Cryans < >>>> >> [email protected]> wrote: >>>> >> >>> It's pretty much what we do, works well. >>>> >> >>> >>>> >> >>> J-D >>>> >> >>> >>>> >> >>> On Wed, Nov 30, 2011 at 3:49 PM, Rita <[email protected]> >>>> wrote: >>>> >> >>>> Hello, >>>> >> >>>> >>>> >> >>>> >>>> >> >>>> Previously, I assigned 5 servers as part of the zookeeper quorum. >>>> >> >>>> Everything works fine but I was hard coding these 5 servers >>>> >> everywhere and >>>> >> >>>> I was thinking of creating a dns entry called appquorum which will >>>> >> always >>>> >> >>>> return these 5 servers IPs. >>>> >> >>>> >>>> >> >>>> Any thoughts about this? >>>> >> >>>> >>>> >> >>>> >>>> >> >>>> >>>> >> >>>> -- >>>> >> >>>> --- Get your facts first, then you can distort them as you >>>> please.-- >>>> >> >>>> > >>>> > >>>> >>> >>> >>> >>> -- >>> --- Get your facts first, then you can distort them as you please.-- >>> >> >> >> >> -- >> --- Get your facts first, then you can distort them as you please.--
