[GitHub] [hbase] busbey commented on a change in pull request #1167: Hbase 18095: Zookeeper-less client connection implementation
busbey commented on a change in pull request #1167: Hbase 18095: Zookeeper-less client connection implementation URL: https://github.com/apache/hbase/pull/1167#discussion_r378922404 ## File path: src/main/asciidoc/_chapters/configuration.adoc ## @@ -563,38 +563,63 @@ Changes here will require a cluster restart for HBase to notice the change thoug If you are running HBase in standalone mode, you don't need to configure anything for your client to work provided that they are all on the same machine. -Since the HBase Master may move around, clients bootstrap by looking to ZooKeeper for current critical locations. -ZooKeeper is where all these values are kept. -Thus clients require the location of the ZooKeeper ensemble before they can do anything else. -Usually this ensemble location is kept out in the _hbase-site.xml_ and is picked up by the client from the `CLASSPATH`. +Starting release 3.0.0, the default connection registry has been switched to a master based implementation. Refer to <> for more details about +what a connection registry is and implications of this change. Depending on your HBase version, following is the expected minimal client configuration. -If you are configuring an IDE to run an HBase client, you should include the _conf/_ directory on your classpath so _hbase-site.xml_ settings can be found (or add _src/test/resources_ to pick up the hbase-site.xml used by tests). + Up until 2.x.y releases +In 2.x.y releases, the default connection registry was based on ZooKeeper as the source of truth. This means that the clients always looked up ZooKeeper znodes to fetch +the required metadata. For example, if an active master crashed and the a new master is elected, clients looked up the master znode to fetch +the active master address (similarly for meta locations). This meant that the clients needed to have access to ZooKeeper and need to know +the ZooKeeper ensemble information before they can do anything. This can be configured in the client configuration xml as follows: -For Java applications using Maven, including the hbase-shaded-client module is the recommended dependency when connecting to a cluster: [source,xml] - - org.apache.hbase - hbase-shaded-client - 2.0.0 - + + + + +hbase.zookeeper.quorum +example1,example2,example3 + Zookeeper ensemble information + + -A basic example _hbase-site.xml_ for client only may look as follows: + Starting 3.0.0 release + +The default implementation was switched to a master based connection registry. With this implementation, clients always contact the active or +stand-by master RPC end points to fetch the the connection registry information. This means that the clients should have access to the list of active and master +end points before they can do anything. This can be configured in the client configuration xml as follows: Review comment: This specifically doesn't need to be _all_ of the Masters configured for the cluster, right? So if I wanted to hedge against clients ddosing masters to the point that I can't get an active master for the cluster I could e.g. only give them half of the Masters. Presuming that's the case, a brief entry in the troubleshooting section that basically says to do this in case clients are causing masters to die would help us get ahead of that concern. I say this as someone who has had to troubleshoot clusters where bad client behavior essentially meant every hbase operation hit ZK for meta and as a result said cluster would die. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] busbey commented on a change in pull request #1167: Hbase 18095: Zookeeper-less client connection implementation
busbey commented on a change in pull request #1167: Hbase 18095: Zookeeper-less client connection implementation URL: https://github.com/apache/hbase/pull/1167#discussion_r378918970 ## File path: src/main/asciidoc/_chapters/architecture.adoc ## @@ -577,11 +644,24 @@ If the active Master loses its lease in ZooKeeper (or the Master shuts down), th [[master.runtime]] === Runtime Impact -A common dist-list question involves what happens to an HBase cluster when the Master goes down. +A common dist-list question involves what happens to an HBase cluster when the Master goes down. This information has changed staring 3.0.0. + + Up until releases 2.x.y Because the HBase client talks directly to the RegionServers, the cluster can still function in a "steady state". Additionally, per <>, `hbase:meta` exists as an HBase table and is not resident in the Master. However, the Master controls critical functions such as RegionServer failover and completing region splits. So while the cluster can still run for a short time without the Master, the Master should be restarted as soon as possible. + Staring release 3.0.0 +As mentioned in section <>, the default connection registry for clients is now based on master rpc end points. Hence the requirements for +masters' uptime are even tighter starting this release. + +- At least one active or stand by master is needed for a connection set up, unlike before when all the clients needed was a ZooKeeper ensemble. +- Master is now in critical path for read/write operations. For example, if the meta region bounces off to a different region server, clients +need master to fetch the new locations. Earlier this was done by fetching this information directly from ZooKeeper. +- Masters will now have higher connection load than before. So, the server side configuration might need adjustment depending on the load. + +Overall, the master uptime requirements starting with this version are even higher for the client operations to go through. Review comment: Should phrase this change in uptime needs of the master ensemble in terms of when the feature is enabled rather than in terms of hbase versions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] busbey commented on a change in pull request #1167: Hbase 18095: Zookeeper-less client connection implementation
busbey commented on a change in pull request #1167: Hbase 18095: Zookeeper-less client connection implementation URL: https://github.com/apache/hbase/pull/1167#discussion_r378920122 ## File path: dev-support/design-docs/HBASE-18095-Zookeeper-less-client-connection-design.adoc ## @@ -0,0 +1,112 @@ + +/** + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + += HBASE-18095: Zookeeper-less client connection + + +== Context +Currently, Zookeeper (ZK) lies in the critical code path of connection init. To set up a connection to a given HBase cluster, client relies on the zookeeper quorum configured in the client hbase-site.xml and attempts to fetch the following information. + +* ClusterID +* Active HMaster server name +* Meta table region locations + +ZK is deemed the source of truth since other processes that maintain the cluster state persist the changes to this data into ZK. So it is an obvious place to look at for clients to fetch the latest cluster state. However this comes with it’s own set of problems, some of them are below. + +* Timeouts and retry logic for ZK clients are managed separately from HBase configuration. This is more administration overhead for end users (example: multiple timeouts are to be configured for different types of RPCs. client->master, client->ZK etc.). This prevents HBase from having a single holistic timeout configuration that applies to any RPCs. +* If there is any issue with ZK (like connection overload / timeouts), the entire HBase service appears frozen and there is little visibility into it. +* Exposing zookeeper to all the clients can be risky since it can potentially be abused to DDOS. +* Embedded ZK client is bundled with hbase client jar as a dependency (with it’s log spew :-]). Review comment: Also clients need an additional set of config files in secure deployments because zk uses JAAS This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
