http://git-wip-us.apache.org/repos/asf/hbase-site/blob/e0fb1fde/book.html ---------------------------------------------------------------------- diff --git a/book.html b/book.html index a35b472..2770027 100644 --- a/book.html +++ b/book.html @@ -138,7 +138,7 @@ <li><a href="#hbase_mob">75. Storing Medium-sized Objects (MOB)</a></li> </ul> </li> -<li><a href="#casestudies">Backup and Restore</a> +<li><a href="#backuprestore">Backup and Restore</a> <ul class="sectlevel1"> <li><a href="#br.overview">76. Overview</a></li> <li><a href="#br.terminology">77. Terminology</a></li> @@ -227,78 +227,77 @@ <li><a href="#trouble.zookeeper">132. ZooKeeper</a></li> <li><a href="#trouble.ec2">133. Amazon EC2</a></li> <li><a href="#trouble.versions">134. HBase and Hadoop version issues</a></li> -<li><a href="#_ipc_configuration_conflicts_with_hadoop">135. IPC Configuration Conflicts with Hadoop</a></li> -<li><a href="#_hbase_and_hdfs">136. HBase and HDFS</a></li> -<li><a href="#trouble.tests">137. Running unit or integration tests</a></li> -<li><a href="#trouble.casestudy">138. Case Studies</a></li> -<li><a href="#trouble.crypto">139. Cryptographic Features</a></li> -<li><a href="#_operating_system_specific_issues">140. Operating System Specific Issues</a></li> -<li><a href="#_jdk_issues">141. JDK Issues</a></li> +<li><a href="#_hbase_and_hdfs">135. HBase and HDFS</a></li> +<li><a href="#trouble.tests">136. Running unit or integration tests</a></li> +<li><a href="#trouble.casestudy">137. Case Studies</a></li> +<li><a href="#trouble.crypto">138. Cryptographic Features</a></li> +<li><a href="#_operating_system_specific_issues">139. Operating System Specific Issues</a></li> +<li><a href="#_jdk_issues">140. JDK Issues</a></li> </ul> </li> <li><a href="#casestudies">Apache HBase Case Studies</a> <ul class="sectlevel1"> -<li><a href="#casestudies.overview">142. Overview</a></li> -<li><a href="#casestudies.schema">143. Schema Design</a></li> -<li><a href="#casestudies.perftroub">144. Performance/Troubleshooting</a></li> +<li><a href="#casestudies.overview">141. Overview</a></li> +<li><a href="#casestudies.schema">142. Schema Design</a></li> +<li><a href="#casestudies.perftroub">143. Performance/Troubleshooting</a></li> </ul> </li> <li><a href="#ops_mgt">Apache HBase Operational Management</a> <ul class="sectlevel1"> -<li><a href="#tools">145. HBase Tools and Utilities</a></li> -<li><a href="#ops.regionmgt">146. Region Management</a></li> -<li><a href="#node.management">147. Node Management</a></li> -<li><a href="#hbase_metrics">148. HBase Metrics</a></li> -<li><a href="#ops.monitoring">149. HBase Monitoring</a></li> -<li><a href="#_cluster_replication">150. Cluster Replication</a></li> -<li><a href="#_running_multiple_workloads_on_a_single_cluster">151. Running Multiple Workloads On a Single Cluster</a></li> -<li><a href="#ops.backup">152. HBase Backup</a></li> -<li><a href="#ops.snapshots">153. HBase Snapshots</a></li> -<li><a href="#snapshots_azure">154. Storing Snapshots in Microsoft Azure Blob Storage</a></li> -<li><a href="#ops.capacity">155. Capacity Planning and Region Sizing</a></li> -<li><a href="#table.rename">156. Table Rename</a></li> -<li><a href="#rsgroup">157. RegionServer Grouping</a></li> -<li><a href="#normalizer">158. Region Normalizer</a></li> +<li><a href="#tools">144. HBase Tools and Utilities</a></li> +<li><a href="#ops.regionmgt">145. Region Management</a></li> +<li><a href="#node.management">146. Node Management</a></li> +<li><a href="#hbase_metrics">147. HBase Metrics</a></li> +<li><a href="#ops.monitoring">148. HBase Monitoring</a></li> +<li><a href="#_cluster_replication">149. Cluster Replication</a></li> +<li><a href="#_running_multiple_workloads_on_a_single_cluster">150. Running Multiple Workloads On a Single Cluster</a></li> +<li><a href="#ops.backup">151. HBase Backup</a></li> +<li><a href="#ops.snapshots">152. HBase Snapshots</a></li> +<li><a href="#snapshots_azure">153. Storing Snapshots in Microsoft Azure Blob Storage</a></li> +<li><a href="#ops.capacity">154. Capacity Planning and Region Sizing</a></li> +<li><a href="#table.rename">155. Table Rename</a></li> +<li><a href="#rsgroup">156. RegionServer Grouping</a></li> +<li><a href="#normalizer">157. Region Normalizer</a></li> </ul> </li> <li><a href="#developer">Building and Developing Apache HBase</a> <ul class="sectlevel1"> -<li><a href="#getting.involved">159. Getting Involved</a></li> -<li><a href="#repos">160. Apache HBase Repositories</a></li> -<li><a href="#_ides">161. IDEs</a></li> -<li><a href="#build">162. Building Apache HBase</a></li> -<li><a href="#releasing">163. Releasing Apache HBase</a></li> -<li><a href="#hbase.rc.voting">164. Voting on Release Candidates</a></li> -<li><a href="#documentation">165. Generating the HBase Reference Guide</a></li> -<li><a href="#hbase.org">166. Updating <a href="https://hbase.apache.org";>hbase.apache.org</a></a></li> -<li><a href="#hbase.tests">167. Tests</a></li> -<li><a href="#developing">168. Developer Guidelines</a></li> +<li><a href="#getting.involved">158. Getting Involved</a></li> +<li><a href="#repos">159. Apache HBase Repositories</a></li> +<li><a href="#_ides">160. IDEs</a></li> +<li><a href="#build">161. Building Apache HBase</a></li> +<li><a href="#releasing">162. Releasing Apache HBase</a></li> +<li><a href="#hbase.rc.voting">163. Voting on Release Candidates</a></li> +<li><a href="#documentation">164. Generating the HBase Reference Guide</a></li> +<li><a href="#hbase.org">165. Updating <a href="https://hbase.apache.org";>hbase.apache.org</a></a></li> +<li><a href="#hbase.tests">166. Tests</a></li> +<li><a href="#developing">167. Developer Guidelines</a></li> </ul> </li> <li><a href="#unit.tests">Unit Testing HBase Applications</a> <ul class="sectlevel1"> -<li><a href="#_junit">169. JUnit</a></li> -<li><a href="#mockito">170. Mockito</a></li> -<li><a href="#_mrunit">171. MRUnit</a></li> -<li><a href="#_integration_testing_with_an_hbase_mini_cluster">172. Integration Testing with an HBase Mini-Cluster</a></li> +<li><a href="#_junit">168. JUnit</a></li> +<li><a href="#mockito">169. Mockito</a></li> +<li><a href="#_mrunit">170. MRUnit</a></li> +<li><a href="#_integration_testing_with_an_hbase_mini_cluster">171. Integration Testing with an HBase Mini-Cluster</a></li> </ul> </li> <li><a href="#protobuf">Protobuf in HBase</a> <ul class="sectlevel1"> -<li><a href="#_protobuf">173. Protobuf</a></li> +<li><a href="#_protobuf">172. Protobuf</a></li> </ul> </li> <li><a href="#zookeeper">ZooKeeper</a> <ul class="sectlevel1"> -<li><a href="#_using_existing_zookeeper_ensemble">174. Using existing ZooKeeper ensemble</a></li> -<li><a href="#zk.sasl.auth">175. SASL Authentication with ZooKeeper</a></li> +<li><a href="#_using_existing_zookeeper_ensemble">173. Using existing ZooKeeper ensemble</a></li> +<li><a href="#zk.sasl.auth">174. SASL Authentication with ZooKeeper</a></li> </ul> </li> <li><a href="#community">Community</a> <ul class="sectlevel1"> -<li><a href="#_decisions">176. Decisions</a></li> -<li><a href="#community.roles">177. Community Roles</a></li> -<li><a href="#hbase.commit.msg.format">178. Commit Message format</a></li> +<li><a href="#_decisions">175. Decisions</a></li> +<li><a href="#community.roles">176. Community Roles</a></li> +<li><a href="#hbase.commit.msg.format">177. Commit Message format</a></li> </ul> </li> <li><a href="#_appendix">Appendix</a> @@ -308,7 +307,6 @@ <li><a href="#hbck.in.depth">Appendix C: hbck In Depth</a></li> <li><a href="#appendix_acl_matrix">Appendix D: Access Control Matrix</a></li> <li><a href="#compression">Appendix E: Compression and Data Block Encoding In HBase</a></li> -<li><a href="#data.block.encoding.enable">179. Enable Data Block Encoding</a></li> <li><a href="#sql">Appendix F: SQL over HBase</a></li> <li><a href="#ycsb">Appendix G: YCSB</a></li> <li><a href="#_hfile_format_2">Appendix H: HFile format</a></li> @@ -317,8 +315,8 @@ <li><a href="#asf">Appendix K: HBase and the Apache Software Foundation</a></li> <li><a href="#orca">Appendix L: Apache HBase Orca</a></li> <li><a href="#tracing">Appendix M: Enabling Dapper-like Tracing in HBase</a></li> -<li><a href="#tracing.client.modifications">180. Client Modifications</a></li> -<li><a href="#tracing.client.shell">181. Tracing from HBase Shell</a></li> +<li><a href="#tracing.client.modifications">178. Client Modifications</a></li> +<li><a href="#tracing.client.shell">179. Tracing from HBase Shell</a></li> <li><a href="#hbase.rpc">Appendix N: 0.95 RPC Specification</a></li> </ul> </li> @@ -468,34 +466,6 @@ table, enable or disable the table, and start and stop HBase.</p> <div class="paragraph"> <p>Apart from downloading HBase, this procedure should take less than 10 minutes.</p> </div> -<div id="loopback.ip" class="admonitionblock note"> -<table> -<tr> -<td class="icon"> -<i class="fa icon-note" title="Note"></i> -</td> -<td class="content"> -<div class="paragraph"> -<p>Prior to HBase 0.94.x, HBase expected the loopback IP address to be 127.0.0.1. -Ubuntu and some other distributions default to 127.0.1.1 and this will cause -problems for you. See <a href="https://web-beta.archive.org/web/20140104070155/http://blog.devving.com/why-does-hbase-care-about-etchosts";>Why does HBase care about /etc/hosts?</a> for detail</p> -</div> -<div class="paragraph"> -<p>The following <em>/etc/hosts</em> file works correctly for HBase 0.94.x and earlier, on Ubuntu. Use this as a template if you run into trouble.</p> -</div> -<div class="listingblock"> -<div class="content"> -<pre>127.0.0.1 localhost -127.0.0.1 ubuntu.ubuntu-domain ubuntu</pre> -</div> -</div> -<div class="paragraph"> -<p>This issue has been fixed in hbase-0.96.0 and beyond.</p> -</div> -</td> -</tr> -</table> -</div> <div class="sect2"> <h3 id="_jdk_version_requirements"><a class="anchor" href="#_jdk_version_requirements"></a>2.1. JDK Version Requirements</h3> <div class="paragraph"> @@ -949,10 +919,10 @@ Running multiple HRegionServers on the same system can be useful for testing in The <code>local-regionservers.sh</code> command allows you to run multiple RegionServers. It works in a similar way to the <code>local-master-backup.sh</code> command, in that each parameter you provide represents the port offset for an instance. Each RegionServer requires two ports, and the default ports are 16020 and 16030. -However, the base ports for additional RegionServers are not the default ports since the default ports are used by the HMaster, which is also a RegionServer since HBase version 1.0.0. -The base ports are 16200 and 16300 instead. -You can run 99 additional RegionServers that are not a HMaster or backup HMaster, on a server. -The following command starts four additional RegionServers, running on sequential ports starting at 16202/16302 (base ports 16200/16300 plus 2).</p> +Since HBase version 1.1.0, HMaster doesn’t use region server ports, this leaves 10 ports (16020 to 16029 and 16030 to 16039) to be used for RegionServers. +For supporting additional RegionServers, base ports can be changed in script 'local-regionservers.sh' to appropriate value. +e.g. With values 16200 and 16300 for base ports, 99 additional RegionServers can be supported, on a server. +The following command starts four additional RegionServers, running on sequential ports starting at 16022/16032 (base ports 16020/16030 plus 2).</p> </div> <div class="listingblock"> <div class="content"> @@ -1486,7 +1456,7 @@ For most configurations, a restart is needed for servers to pick up changes. Dyn <i class="fa icon-note" title="Note"></i> </td> <td class="content"> -HBase will neither build nor compile with Java 6. +HBase will neither build nor run with Java 6. </td> </tr> </table> @@ -1512,12 +1482,7 @@ You must set <code>JAVA_HOME</code> on each node of your cluster. <em>hbase-env. </dd> <dt class="hdlist1">DNS</dt> <dd> -<p>HBase uses the local hostname to self-report its IP address. Both forward and reverse DNS resolving must work in versions of HBase previous to 0.92.0. The <a href="https://github.com/sujee/hadoop-dns-checker";>hadoop-dns-checker</a> tool can be used to verify DNS is working correctly on the cluster. The project <code>README</code> file provides detailed instructions on usage.</p> -</dd> -<dt class="hdlist1">Loopback IP</dt> -<dd> -<p>Prior to hbase-0.96.0, HBase only used the IP address <code>127.0.0.1</code> to refer to <code>localhost</code>, and this was not configurable. -See <a href="#loopback.ip">Loopback IP</a> for more details.</p> +<p>HBase uses the local hostname to self-report its IP address.</p> </dd> <dt class="hdlist1">NTP</dt> <dd> @@ -1583,16 +1548,17 @@ hadoop - nproc 32000</pre> </dd> <dt class="hdlist1">Windows</dt> <dd> -<p>Prior to HBase 0.96, running HBase on Microsoft Windows was limited only for testing purposes. -Running production systems on Windows machines is not recommended.</p> +<p>Running production systems on Windows machines is not recommended.</p> </dd> </dl> </div> <div class="sect2"> <h3 id="hadoop"><a class="anchor" href="#hadoop"></a>4.1. <a href="https://hadoop.apache.org";>Hadoop</a></h3> <div class="paragraph"> -<p>The following table summarizes the versions of Hadoop supported with each version of HBase. -Based on the version of HBase, you should select the most appropriate version of Hadoop. +<p>The following table summarizes the versions of Hadoop supported with each version of HBase. Older versions not appearing in this table are considered unsupported and likely missing necessary features, while newer versions are untested but may be suitable.</p> +</div> +<div class="paragraph"> +<p>Based on the version of HBase, you should select the most appropriate version of Hadoop. You can use Apache Hadoop, or a vendor’s distribution of Hadoop. No distinction is made here. See <a href="https://wiki.apache.org/hadoop/Distributions%20and%20Commercial%20Support";>the Hadoop wiki</a> for information about vendors of Hadoop.</p> @@ -1651,30 +1617,6 @@ earlier versions of Hadoop. See the table below for requirements specific to dif </thead> <tbody> <tr> -<td class="tableblock halign-left valign-top"><p class="tableblock">Hadoop-2.0.x-alpha</p></td> -<td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td> -<td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td> -<td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td> -</tr> -<tr> -<td class="tableblock halign-left valign-top"><p class="tableblock">Hadoop-2.1.0-beta</p></td> -<td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td> -<td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td> -<td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td> -</tr> -<tr> -<td class="tableblock halign-left valign-top"><p class="tableblock">Hadoop-2.2.0</p></td> -<td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td> -<td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td> -<td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td> -</tr> -<tr> -<td class="tableblock halign-left valign-top"><p class="tableblock">Hadoop-2.3.x</p></td> -<td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td> -<td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td> -<td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td> -</tr> -<tr> <td class="tableblock halign-left valign-top"><p class="tableblock">Hadoop-2.4.x</p></td> <td class="tableblock halign-left valign-top"><p class="tableblock">S</p></td> <td class="tableblock halign-left valign-top"><p class="tableblock">S</p></td> @@ -1844,10 +1786,7 @@ For example:</p> <div class="sect2"> <h3 id="zookeeper.requirements"><a class="anchor" href="#zookeeper.requirements"></a>4.2. ZooKeeper Requirements</h3> <div class="paragraph"> -<p>ZooKeeper 3.4.x is required. -HBase makes use of the <code>multi</code> functionality that is only available since Zookeeper 3.4.0. The <code>hbase.zookeeper.useMulti</code> configuration property defaults to <code>true</code>. -Refer to <a href="https://issues.apache.org/jira/browse/HBASE-12241";>HBASE-12241 (The crash of regionServer when taking deadserver’s replication queue breaks replication)</a> and <a href="https://issues.apache.org/jira/browse/HBASE-6775";>HBASE-6775 (Use ZK.multi when available for HBASE-6710 0.92/0.94 compatibility fix)</a> for background. -The property is deprecated and useMulti is always enabled in HBase 2.0.</p> +<p>ZooKeeper 3.4.x is required.</p> </div> </div> </div> @@ -5175,14 +5114,14 @@ Usually this ensemble location is kept out in the <em>hbase-site.xml</em> and is <p>If you are configuring an IDE to run an HBase client, you should include the <em>conf/</em> directory on your classpath so <em>hbase-site.xml</em> settings can be found (or add <em>src/test/resources</em> to pick up the hbase-site.xml used by tests).</p> </div> <div class="paragraph"> -<p>Minimally, an HBase client needs hbase-client module in its dependencies when connecting to a cluster:</p> +<p>For Java applications using Maven, including the hbase-shaded-client module is the recommended dependency when connecting to a cluster:</p> </div> <div class="listingblock"> <div class="content"> <pre class="CodeRay highlight"><code data-lang="xml"><span class="tag"><dependency></span> <span class="tag"><groupId></span>org.apache.hbase<span class="tag"></groupId></span> - <span class="tag"><artifactId></span>hbase-client<span class="tag"></artifactId></span> - <span class="tag"><version></span>1.2.4<span class="tag"></version></span> + <span class="tag"><artifactId></span>hbase-shaded-client<span class="tag"></artifactId></span> + <span class="tag"><version></span>2.0.0<span class="tag"></version></span> <span class="tag"></dependency></span></code></pre> </div> </div> @@ -5453,7 +5392,6 @@ See <a href="#manual_region_splitting_decisions">manual region splitting decisio </div> <div class="paragraph"> <p>Instead of allowing HBase to split your regions automatically, you can choose to manage the splitting yourself. -This feature was added in HBase 0.90.0. Manually managing splits works if you know your keyspace well, otherwise let HBase figure where to split for you. Manual splitting can mitigate region creation and movement under load. It also makes it so region boundaries are known and invariant (if you disable region splitting). If you use manual splits, it is easier doing staggered, time-based major compactions to spread out your network IO load.</p> @@ -5491,14 +5429,13 @@ It is important to understand that the data growth causes compaction storms and </div> <div class="paragraph"> <p>If the regions are split into too many large regions, you can increase the major compaction interval by configuring <code>HConstants.MAJOR_COMPACTION_PERIOD</code>. -HBase 0.90 introduced <code>org.apache.hadoop.hbase.util.RegionSplitter</code>, which provides a network-IO-safe rolling split of all regions.</p> +The <code>org.apache.hadoop.hbase.util.RegionSplitter</code> utility also provides a network-IO-safe rolling split of all regions.</p> </div> </div> <div class="sect3"> <h4 id="managed.compactions"><a class="anchor" href="#managed.compactions"></a>9.2.7. Managed Compactions</h4> <div class="paragraph"> -<p>By default, major compactions are scheduled to run once in a 7-day period. -Prior to HBase 0.96.x, major compactions were scheduled to happen once per day by default.</p> +<p>By default, major compactions are scheduled to run once in a 7-day period.</p> </div> <div class="paragraph"> <p>If you need to control exactly when and how often major compaction runs, you can disable managed major compactions. @@ -5639,8 +5576,8 @@ See the <a href="http://docs.oracle.com/javase/8/docs/technotes/guides/managemen Historically, besides above port mentioned, JMX opens two additional random TCP listening ports, which could lead to port conflict problem. (See <a href="https://issues.apache.org/jira/browse/HBASE-10289";>HBASE-10289</a> for details)</p> </div> <div class="paragraph"> -<p>As an alternative, You can use the coprocessor-based JMX implementation provided by HBase. -To enable it in 0.99 or above, add below property in <em>hbase-site.xml</em>:</p> +<p>As an alternative, you can use the coprocessor-based JMX implementation provided by HBase. +To enable it, add below property in <em>hbase-site.xml</em>:</p> </div> <div class="listingblock"> <div class="content"> @@ -5769,8 +5706,8 @@ To enable the HBase JMX implementation on Master, you also need to add below pro <h2 id="dyn_config"><a class="anchor" href="#dyn_config"></a>10. Dynamic Configuration</h2> <div class="sectionbody"> <div class="paragraph"> -<p>Since HBase 1.0.0, it is possible to change a subset of the configuration without requiring a server restart. -In the HBase shell, there are new operators, <code>update_config</code> and <code>update_all_config</code> that will prompt a server or all servers to reload configuration.</p> +<p>It is possible to change a subset of the configuration without requiring a server restart. +In the HBase shell, the operations <code>update_config</code> and <code>update_all_config</code> will prompt a server or all servers to reload configuration.</p> </div> <div class="paragraph"> <p>Only a subset of all configurations can currently be changed in the running server. @@ -5993,19 +5930,7 @@ Here are those configurations:</p> <div class="openblock partintro"> <div class="content"> <div class="paragraph"> -<p>You cannot skip major versions when upgrading. If you are upgrading from version 0.90.x to 0.94.x, you must first go from 0.90.x to 0.92.x and then go from 0.92.x to 0.94.x.</p> -</div> -<div class="admonitionblock note"> -<table> -<tr> -<td class="icon"> -<i class="fa icon-note" title="Note"></i> -</td> -<td class="content"> -It may be possible to skip across versions — for example go from 0.92.2 straight to 0.98.0 just following the 0.96.x upgrade instructions — but these scenarios are untested. -</td> -</tr> -</table> +<p>You cannot skip major versions when upgrading. If you are upgrading from version 0.98.x to 2.x, you must first go from 0.98.x to 1.2.x and then go from 1.2.x to 2.x.</p> </div> <div class="paragraph"> <p>Review <a href="#configuration">Apache HBase Configuration</a>, in particular <a href="#hadoop"><a href="https://hadoop.apache.org";>Hadoop</a></a>. Familiarize yourself with <a href="#hbase_supported_tested_definitions">Support and Testing Expectations</a>.</p> @@ -6015,11 +5940,8 @@ It may be possible to skip across versions — for example go fr <div class="sect1"> <h2 id="hbase.versioning"><a class="anchor" href="#hbase.versioning"></a>11. HBase version number and compatibility</h2> <div class="sectionbody"> -<div class="paragraph"> -<p>HBase has two versioning schemes, pre-1.0 and post-1.0. Both are detailed below.</p> -</div> <div class="sect2"> -<h3 id="hbase.versioning.post10"><a class="anchor" href="#hbase.versioning.post10"></a>11.1. Post 1.0 versions</h3> +<h3 id="hbase.versioning.post10"><a class="anchor" href="#hbase.versioning.post10"></a>11.1. Aspirational Semantic Versioning</h3> <div class="paragraph"> <p>Starting with the 1.0.0 release, HBase is working towards <a href="http://semver.org/";>Semantic Versioning</a> for its release versioning. In summary:</p> </div> @@ -6351,40 +6273,14 @@ Classes which are defined as <code>IA.Private</code> may be used as parameters o </dd> </dl> </div> -</div> -</div> -<div class="sect2"> -<h3 id="hbase.versioning.pre10"><a class="anchor" href="#hbase.versioning.pre10"></a>11.2. Pre 1.0 versions</h3> -<div class="admonitionblock note"> -<table> -<tr> -<td class="icon"> -<i class="fa icon-note" title="Note"></i> -</td> -<td class="content"> -<div class="title">HBase Pre-1.0 versions are all EOM</div> -For new installations, do not deploy 0.94.y, 0.96.y, or 0.98.y. Deploy our stable version. See <a href="https://issues.apache.org/jira/browse/HBASE-11642";>EOL 0.96</a>, <a href="https://issues.apache.org/jira/browse/HBASE-16215";>clean up of EOM releases</a>, and <a href="https://www.apache.org/dist/hbase/";>the header of our downloads</a>. -</td> -</tr> -</table> -</div> -<div class="paragraph"> -<p>Before the semantic versioning scheme pre-1.0, HBase tracked either Hadoop’s versions (0.2x) or 0.9x versions. If you are into the arcane, checkout our old wiki page on <a href="https://web.archive.org/web/20150905071342/https://wiki.apache.org/hadoop/Hbase/HBaseVersions";>HBase Versioning</a> which tries to connect the HBase version dots. Below sections cover ONLY the releases before 1.0.</p> -</div> -<div id="hbase.development.series" class="paragraph"> -<div class="title">Odd/Even Versioning or "Development" Series Releases</div> -<p>Ahead of big releases, we have been putting up preview versions to start the feedback cycle turning-over earlier. These "Development" Series releases, always odd-numbered, come with no guarantees, not even regards being able to upgrade between two sequential releases (we reserve the right to break compatibility across "Development" Series releases). Needless to say, these releases are not for production deploys. They are a preview of what is coming in the hope that interested parties will take the release for a test drive and flag us early if we there are issues we’ve missed ahead of our rolling a production-worthy release.</p> -</div> -<div class="paragraph"> -<p>Our first "Development" Series was the 0.89 set that came out ahead of HBase 0.90.0. HBase 0.95 is another "Development" Series that portends HBase 0.96.0. 0.99.x is the last series in "developer preview" mode before 1.0. Afterwards, we will be using semantic versioning naming scheme (see above).</p> -</div> <div id="hbase.binary.compatibility" class="paragraph"> <div class="title">Binary Compatibility</div> -<p>When we say two HBase versions are compatible, we mean that the versions are wire and binary compatible. Compatible HBase versions means that clients can talk to compatible but differently versioned servers. It means too that you can just swap out the jars of one version and replace them with the jars of another, compatible version and all will just work. Unless otherwise specified, HBase point versions are (mostly) binary compatible. You can safely do rolling upgrades between binary compatible versions; i.e. across point versions: e.g. from 0.94.5 to 0.94.6. See link:[Does compatibility between versions also mean binary compatibility?] discussion on the HBase dev mailing list.</p> +<p>When we say two HBase versions are compatible, we mean that the versions are wire and binary compatible. Compatible HBase versions means that clients can talk to compatible but differently versioned servers. It means too that you can just swap out the jars of one version and replace them with the jars of another, compatible version and all will just work. Unless otherwise specified, HBase point versions are (mostly) binary compatible. You can safely do rolling upgrades between binary compatible versions; i.e. across maintenance releases: e.g. from 1.2.4 to 1.2.6. See link:[Does compatibility between versions also mean binary compatibility?] discussion on the HBase dev mailing list.</p> +</div> </div> </div> <div class="sect2"> -<h3 id="hbase.rolling.upgrade"><a class="anchor" href="#hbase.rolling.upgrade"></a>11.3. Rolling Upgrades</h3> +<h3 id="hbase.rolling.upgrade"><a class="anchor" href="#hbase.rolling.upgrade"></a>11.2. Rolling Upgrades</h3> <div class="paragraph"> <p>A rolling upgrade is the process by which you update the servers in your cluster a server at a time. You can rolling upgrade across HBase versions if they are binary or wire compatible. See <a href="#hbase.rolling.restart">Rolling Upgrade Between Versions that are Binary/Wire Compatible</a> for more on what this means. Coarsely, a rolling upgrade is a graceful stop each server, update the software, and then restart. You do this for each server in the cluster. Usually you upgrade the Master first and then the RegionServers. See <a href="#rolling">Rolling Restart</a> for tools that can help use the rolling upgrade process.</p> </div> @@ -6401,10 +6297,10 @@ For new installations, do not deploy 0.94.y, 0.96.y, or 0.98.y. Deploy our stab </div> <div id="hbase.rolling.restart" class="paragraph"> <div class="title">Rolling Upgrade Between Versions that are Binary/Wire Compatible</div> -<p>Unless otherwise specified, HBase point versions are binary compatible. You can do a <a href="#hbase.rolling.upgrade">Rolling Upgrades</a> between HBase point versions. For example, you can go to 0.94.6 from 0.94.5 by doing a rolling upgrade across the cluster replacing the 0.94.5 binary with a 0.94.6 binary.</p> +<p>Unless otherwise specified, HBase minor versions are binary compatible. You can do a <a href="#hbase.rolling.upgrade">Rolling Upgrades</a> between HBase point versions. For example, you can go to 1.2.4 from 1.2.6 by doing a rolling upgrade across the cluster replacing the 1.2.4 binary with a 1.2.6 binary.</p> </div> <div class="paragraph"> -<p>In the minor version-particular sections below, we call out where the versions are wire/protocol compatible and in this case, it is also possible to do a <a href="#hbase.rolling.upgrade">Rolling Upgrades</a>. For example, in <a href="#upgrade1.0.rolling.upgrade">Rolling upgrade from 0.98.x to HBase 1.0.0</a>, we state that it is possible to do a rolling upgrade between hbase-0.98.x and hbase-1.0.0.</p> +<p>In the minor version-particular sections below, we call out where the versions are wire/protocol compatible and in this case, it is also possible to do a <a href="#hbase.rolling.upgrade">Rolling Upgrades</a>.</p> </div> </div> </div> @@ -6683,6 +6579,15 @@ Quitting...</code></pre> <li> <p>hbase.balancer.tablesOnMaster wasn’t removed, strictly speaking, but its meaning has fundamentally changed and users should not set it. See the section <a id="upgrade2.0.regions.on.master"></a> for details.</p> </li> +<li> +<p>hbase.master.distributed.log.replay See the section <a id="upgrade2.0.distributed.log.replay"></a> for details</p> +</li> +<li> +<p>hbase.regionserver.disallow.writes.when.recovering See the section <a id="upgrade2.0.distributed.log.replay"></a> for details</p> +</li> +<li> +<p>hbase.regionserver.wal.logreplay.batch.size See the section <a id="upgrade2.0.distributed.log.replay"></a> for details</p> +</li> </ul> </div> <div id="upgrade2.0.changed.defaults" class="paragraph"> @@ -6743,6 +6648,10 @@ Quitting...</code></pre> </li> </ul> </div> +<div id="upgrade2.0.distributed.log.replay" class="paragraph"> +<div class="title">"Distributed Log Replay" feature broken and removed</div> +<p>The Distributed Log Replay feature was broken and has been removed from HBase 2.y+. As a consequence all related configs, metrics, RPC fields, and logging have also been removed. Note that this feature was found to be unreliable in the run up to HBase 1.0, defaulted to being unused, and was effectively removed in HBase 1.2.0 when we started ignoring the config that turns it on (<a href="https://issues.apache.org/jira/browse/HBASE-14465";>HBASE-14465</a>). If you are currently using the feature, be sure to perform a clean shutdown, ensure all DLR work is complete, and disable the feature prior to upgrading.</p> +</div> <div id="upgrade2.0.metrics" class="paragraph"> <div class="title">Changed metrics</div> <p>The following metrics have changed names:</p> @@ -6764,6 +6673,16 @@ Quitting...</code></pre> </li> </ul> </div> +<div class="paragraph"> +<p>The following metrics have been removed:</p> +</div> +<div class="ulist"> +<ul> +<li> +<p>Metrics related to the Distributed Log Replay feature are no longer present. They were previsouly found in the region server context under the name 'replay'. See the section <a id="upgrade2.0.distributed.log.replay"></a> for details.</p> +</li> +</ul> +</div> <div id="upgrade2.0.zkconfig" class="paragraph"> <div class="title">ZooKeeper configs no longer read from zoo.cfg</div> <p>HBase no longer optionally reads the 'zoo.cfg' file for ZooKeeper related configuration settings. If you previously relied on the 'hbase.config.read.zookeeper.config' config for this functionality, you should migrate any needed settings to the hbase-site.xml file while adding the prefix 'hbase.zookeeper.property.' to each property name.</p> @@ -6957,433 +6876,15 @@ Quitting...</code></pre> </div> </div> <div class="sect2"> -<h3 id="upgrade1.0"><a class="anchor" href="#upgrade1.0"></a>13.3. Upgrading from 0.98.x to 1.x</h3> -<div class="paragraph"> -<p>In this section we first note the significant changes that come in with 1.0.0+ HBase and then we go over the upgrade process. Be sure to read the significant changes section with care so you avoid surprises.</p> -</div> -<div class="sect3"> -<h4 id="_changes_of_note_2"><a class="anchor" href="#_changes_of_note_2"></a>13.3.1. Changes of Note!</h4> -<div class="paragraph"> -<p>In here we list important changes that are in 1.0.0+ since 0.98.x., changes you should be aware that will go into effect once you upgrade.</p> -</div> -<div id="zookeeper.3.4" class="paragraph"> -<div class="title">ZooKeeper 3.4 is required in HBase 1.0.0+</div> -<p>See <a href="#zookeeper.requirements">ZooKeeper Requirements</a>.</p> -</div> -<div id="default.ports.changed" class="paragraph"> -<div class="title">HBase Default Ports Changed</div> -<p>The ports used by HBase changed. They used to be in the 600XX range. In HBase 1.0.0 they have been moved up out of the ephemeral port range and are 160XX instead (Master web UI was 60010 and is now 16010; the RegionServer web UI was 60030 and is now 16030, etc.). If you want to keep the old port locations, copy the port setting configs from <em>hbase-default.xml</em> into <em>hbase-site.xml</em>, change them back to the old values from the HBase 0.98.x era, and ensure you’ve distributed your configurations before you restart.</p> -</div> -<div class="paragraph"> -<div class="title">HBase Master Port Binding Change</div> -<p>In HBase 1.0.x, the HBase Master binds the RegionServer ports as well as the Master -ports. This behavior is changed from HBase versions prior to 1.0. In HBase 1.1 and 2.0 branches, -this behavior is reverted to the pre-1.0 behavior of the HBase master not binding the RegionServer -ports.</p> -</div> -<div id="upgrade1.0.hbase.bucketcache.percentage.in.combinedcache" class="paragraph"> -<div class="title">hbase.bucketcache.percentage.in.combinedcache configuration has been REMOVED</div> -<p>You may have made use of this configuration if you are using BucketCache. If NOT using BucketCache, this change does not affect you. Its removal means that your L1 LruBlockCache is now sized using <code>hfile.block.cache.size</code> — i.e. the way you would size the on-heap L1 LruBlockCache if you were NOT doing BucketCache — and the BucketCache size is not whatever the setting for <code>hbase.bucketcache.size</code> is. You may need to adjust configs to get the LruBlockCache and BucketCache sizes set to what they were in 0.98.x and previous. If you did not set this config., its default value was 0.9. If you do nothing, your BucketCache will increase in size by 10%. Your L1 LruBlockCache will become <code>hfile.block.cache.size</code> times your java heap size (<code>hfile.block.cache.size</code> is a float between 0.0 and 1.0). To read more, see <a href="https://issues.apache.org/jira/browse/HBASE-11520";>HBASE-11520 Simplify offheap cache conf ig by removing the confusing "hbase.bucketcache.percentage.in.combinedcache"</a>.</p> -</div> -<div id="hbase-12068" class="paragraph"> -<div class="title">If you have your own customer filters.</div> -<p>See the release notes on the issue <a href="https://issues.apache.org/jira/browse/HBASE-12068";>HBASE-12068 [Branch-1] Avoid need to always do KeyValueUtil#ensureKeyValue for Filter transformCell</a>; be sure to follow the recommendations therein.</p> -</div> -<div class="paragraph"> -<div class="title">Mismatch Of <code>hbase.client.scanner.max.result.size</code> Between Client and Server</div> -<p>If either the client or server version is lower than 0.98.11/1.0.0 and the server -has a smaller value for <code>hbase.client.scanner.max.result.size</code> than the client, scan -requests that reach the server’s <code>hbase.client.scanner.max.result.size</code> are likely -to miss data. In particular, 0.98.11 defaults <code>hbase.client.scanner.max.result.size</code> -to 2 MB but other versions default to larger values. For this reason, be very careful -using 0.98.11 servers with any other client version.</p> -</div> -<div class="paragraph"> -<div class="title">Availability of Date Tiered Compaction.</div> -<p>The Date Tiered Compaction feature available as of 0.98.19 is available in the 1.y release line starting in release 1.3.0. If you have enabled this feature for any tables you must upgrade to version 1.3.0 or later. If you attempt to use an earlier 1.y release, any tables configured to use date tiered compaction will fail to have their regions open.</p> -</div> -</div> -<div class="sect3"> -<h4 id="upgrade1.0.rolling.upgrade"><a class="anchor" href="#upgrade1.0.rolling.upgrade"></a>13.3.2. Rolling upgrade from 0.98.x to HBase 1.0.0</h4> -<div class="admonitionblock note"> -<table> -<tr> -<td class="icon"> -<i class="fa icon-note" title="Note"></i> -</td> -<td class="content"> -<div class="title">From 0.96.x to 1.0.0</div> -You cannot do a <a href="#hbase.rolling.upgrade">rolling upgrade</a> from 0.96.x to 1.0.0 without first doing a rolling upgrade to 0.98.x. See comment in <a href="https://issues.apache.org/jira/browse/HBASE-11164?focusedCommentId=14182330&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14182330";>HBASE-11164 Document and test rolling updates from 0.98 → 1.0</a> for the why. Also because HBase 1.0.0 enables HFile v3 by default, <a href="https://issues.apache.org/jira/browse/HBASE-9801";>HBASE-9801 Change the default HFile version to V3</a>, and support for HFile v3 only arrives in 0.98, this is another reason you cannot rolling upgrade from HBase 0.96.x; if the rolling upgrade stalls, the 0.96.x servers cannot open files written by the servers running the newer HBase 1.0.0 with HFile’s of version 3. -</td> -</tr> -</table> -</div> -<div class="paragraph"> -<p>There are no known issues running a <a href="#hbase.rolling.upgrade">rolling upgrade</a> from HBase 0.98.x to HBase 1.0.0.</p> -</div> -</div> -<div class="sect3"> -<h4 id="upgrade1.0.scanner.caching"><a class="anchor" href="#upgrade1.0.scanner.caching"></a>13.3.3. Scanner Caching has Changed</h4> -<div class="paragraph"> -<div class="title">From 0.98.x to 1.x</div> -<p>In hbase-1.x, the default Scan caching 'number of rows' changed. -Where in 0.98.x, it defaulted to 100, in later HBase versions, the -default became Integer.MAX_VALUE. Not setting a cache size can make -for Scans that run for a long time server-side, especially if -they are running with stringent filtering. See -<a href="https://issues.apache.org/jira/browse/HBASE-16973";>Revisiting default value for hbase.client.scanner.caching</a>; -for further discussion.</p> -</div> -</div> -<div class="sect3"> -<h4 id="upgrade1.0.from.0.94"><a class="anchor" href="#upgrade1.0.from.0.94"></a>13.3.4. Upgrading to 1.0 from 0.94</h4> -<div class="paragraph"> -<p>You cannot rolling upgrade from 0.94.x to 1.x.x. You must stop your cluster, install the 1.x.x software, run the migration described at <a href="#executing.the.0.96.upgrade">Executing the 0.96 Upgrade</a> (substituting 1.x.x. wherever we make mention of 0.96.x in the section below), and then restart. Be sure to upgrade your ZooKeeper if it is a version less than the required 3.4.x.</p> -</div> -</div> -</div> -<div class="sect2"> -<h3 id="upgrade0.98"><a class="anchor" href="#upgrade0.98"></a>13.4. Upgrading from 0.96.x to 0.98.x</h3> -<div class="paragraph"> -<p>A rolling upgrade from 0.96.x to 0.98.x works. The two versions are not binary compatible.</p> -</div> -<div class="paragraph"> -<p>Additional steps are required to take advantage of some of the new features of 0.98.x, including cell visibility labels, cell ACLs, and transparent server side encryption. See <a href="#security">Securing Apache HBase</a> for more information. Significant performance improvements include a change to the write ahead log threading model that provides higher transaction throughput under high load, reverse scanners, MapReduce over snapshot files, and striped compaction.</p> -</div> -<div class="paragraph"> -<p>Clients and servers can run with 0.98.x and 0.96.x versions. However, applications may need to be recompiled due to changes in the Java API.</p> -</div> -</div> -<div class="sect2"> -<h3 id="_upgrading_from_0_94_x_to_0_98_x"><a class="anchor" href="#_upgrading_from_0_94_x_to_0_98_x"></a>13.5. Upgrading from 0.94.x to 0.98.x</h3> -<div class="paragraph"> -<p>A rolling upgrade from 0.94.x directly to 0.98.x does not work. The upgrade path follows the same procedures as <a href="#upgrade0.96">Upgrading from 0.94.x to 0.96.x</a>. Additional steps are required to use some of the new features of 0.98.x. See <a href="#upgrade0.98">Upgrading from 0.96.x to 0.98.x</a> for an abbreviated list of these features.</p> -</div> -</div> -<div class="sect2"> -<h3 id="upgrade0.96"><a class="anchor" href="#upgrade0.96"></a>13.6. Upgrading from 0.94.x to 0.96.x</h3> -<div class="sect3"> -<h4 id="_the_singularity"><a class="anchor" href="#_the_singularity"></a>13.6.1. The "Singularity"</h4> -<div class="paragraph"> -<p>You will have to stop your old 0.94.x cluster completely to upgrade. If you are replicating between clusters, both clusters will have to go down to upgrade. Make sure it is a clean shutdown. The less WAL files around, the faster the upgrade will run (the upgrade will split any log files it finds in the filesystem as part of the upgrade process). All clients must be upgraded to 0.96 too.</p> -</div> -<div class="paragraph"> -<p>The API has changed. You will need to recompile your code against 0.96 and you may need to adjust applications to go against new APIs (TODO: List of changes).</p> -</div> -</div> -<div class="sect3"> -<h4 id="executing.the.0.96.upgrade"><a class="anchor" href="#executing.the.0.96.upgrade"></a>13.6.2. Executing the 0.96 Upgrade</h4> -<div class="admonitionblock note"> -<table> -<tr> -<td class="icon"> -<i class="fa icon-note" title="Note"></i> -</td> -<td class="content"> -<div class="title">HDFS and ZooKeeper must be up!</div> -HDFS and ZooKeeper should be up and running during the upgrade process. -</td> -</tr> -</table> -</div> -<div class="paragraph"> -<p>HBase 0.96.0 comes with an upgrade script. Run</p> -</div> -<div class="listingblock"> -<div class="content"> -<pre class="CodeRay highlight"><code data-lang="bash">$ bin/hbase upgrade</code></pre> -</div> -</div> -<div class="paragraph"> -<p>to see its usage. The script has two main modes: <code>-check</code>, and <code>-execute</code>.</p> -</div> -<div class="paragraph"> -<div class="title">check</div> -<p>The check step is run against a running 0.94 cluster. Run it from a downloaded 0.96.x binary. The check step is looking for the presence of HFile v1 files. These are unsupported in HBase 0.96.0. To have them rewritten as HFile v2 you must run a compaction.</p> -</div> -<div class="paragraph"> -<p>The check step prints stats at the end of its run (grep for <code>âResult:â</code> in the log) printing absolute path of the tables it scanned, any HFile v1 files found, the regions containing said files (these regions will need a major compaction), and any corrupted files if found. A corrupt file is unreadable, and so is undefined (neither HFile v1 nor HFile v2).</p> -</div> -<div class="paragraph"> -<p>To run the check step, run</p> -</div> -<div class="listingblock"> -<div class="content"> -<pre class="CodeRay highlight"><code data-lang="bash">$ bin/hbase upgrade -check</code></pre> -</div> -</div> -<div class="paragraph"> -<p>Here is sample output:</p> -</div> -<div class="listingblock"> -<div class="content"> -<pre>Tables Processed: -hdfs://localhost:41020/myHBase/.META. -hdfs://localhost:41020/myHBase/usertable -hdfs://localhost:41020/myHBase/TestTable -hdfs://localhost:41020/myHBase/t - -Count of HFileV1: 2 -HFileV1: -hdfs://localhost:41020/myHBase/usertable /fa02dac1f38d03577bd0f7e666f12812/family/249450144068442524 -hdfs://localhost:41020/myHBase/usertable /ecdd3eaee2d2fcf8184ac025555bb2af/family/249450144068442512 - -Count of corrupted files: 1 -Corrupted Files: -hdfs://localhost:41020/myHBase/usertable/fa02dac1f38d03577bd0f7e666f12812/family/1 -Count of Regions with HFileV1: 2 -Regions to Major Compact: -hdfs://localhost:41020/myHBase/usertable/fa02dac1f38d03577bd0f7e666f12812 -hdfs://localhost:41020/myHBase/usertable/ecdd3eaee2d2fcf8184ac025555bb2af - -There are some HFileV1, or corrupt files (files with incorrect major version)</pre> -</div> -</div> -<div class="paragraph"> -<p>In the above sample output, there are two HFile v1 files in two regions, and one corrupt file. Corrupt files should probably be removed. The regions that have HFile v1s need to be major compacted. To major compact, start up the hbase shell and review how to compact an individual region. After the major compaction is done, rerun the check step and the HFile v1 files should be gone, replaced by HFile v2 instances.</p> -</div> -<div class="paragraph"> -<p>By default, the check step scans the HBase root directory (defined as <code>hbase.rootdir</code> in the configuration). To scan a specific directory only, pass the <code>-dir</code> option.</p> -</div> -<div class="listingblock"> -<div class="content"> -<pre class="CodeRay highlight"><code data-lang="bash">$ bin/hbase upgrade -check -dir /myHBase/testTable</code></pre> -</div> -</div> -<div class="paragraph"> -<p>The above command would detect HFile v1 files in the <em>/myHBase/testTable</em> directory.</p> -</div> -<div class="paragraph"> -<p>Once the check step reports all the HFile v1 files have been rewritten, it is safe to proceed with the upgrade.</p> -</div> -<div class="paragraph"> -<div class="title">execute</div> -<p>After the <em>check</em> step shows the cluster is free of HFile v1, it is safe to proceed with the upgrade. Next is the <em>execute</em> step. You must <strong>SHUTDOWN YOUR 0.94.x CLUSTER</strong> before you can run the execute step. The execute step will not run if it detects running HBase masters or RegionServers.</p> -</div> -<div class="admonitionblock note"> -<table> -<tr> -<td class="icon"> -<i class="fa icon-note" title="Note"></i> -</td> -<td class="content"> -<div class="paragraph"> -<p>HDFS and ZooKeeper should be up and running during the upgrade process. If zookeeper is managed by HBase, then you can start zookeeper so it is available to the upgrade by running</p> -</div> -<div class="listingblock"> -<div class="content"> -<pre class="CodeRay highlight"><code data-lang="bash">$ ./hbase/bin/hbase-daemon.sh start zookeeper</code></pre> -</div> -</div> -</td> -</tr> -</table> -</div> -<div class="paragraph"> -<p>The execute upgrade step is made of three substeps.</p> -</div> -<div class="ulist"> -<ul> -<li> -<p>Namespaces: HBase 0.96.0 has support for namespaces. The upgrade needs to reorder directories in the filesystem for namespaces to work.</p> -</li> -<li> -<p>ZNodes: All znodes are purged so that new ones can be written in their place using a new protobuf’ed format and a few are migrated in place: e.g. replication and table state znodes</p> -</li> -<li> -<p>WAL Log Splitting: If the 0.94.x cluster shutdown was not clean, we’ll split WAL logs as part of migration before we startup on 0.96.0. This WAL splitting runs slower than the native distributed WAL splitting because it is all inside the single upgrade process (so try and get a clean shutdown of the 0.94.0 cluster if you can).</p> -</li> -</ul> -</div> -<div class="paragraph"> -<p>To run the <em>execute</em> step, make sure that first you have copied HBase 0.96.0 binaries everywhere under servers and under clients. Make sure the 0.94.0 cluster is down. Then do as follows:</p> -</div> -<div class="listingblock"> -<div class="content"> -<pre class="CodeRay highlight"><code data-lang="bash">$ bin/hbase upgrade -execute</code></pre> -</div> -</div> -<div class="paragraph"> -<p>Here is some sample output.</p> -</div> -<div class="listingblock"> -<div class="content"> -<pre>Starting Namespace upgrade -Created version file at hdfs://localhost:41020/myHBase with version=7 -Migrating table testTable to hdfs://localhost:41020/myHBase/.data/default/testTable -..... -Created version file at hdfs://localhost:41020/myHBase with version=8 -Successfully completed NameSpace upgrade. -Starting Znode upgrade -..... -Successfully completed Znode upgrade - -Starting Log splitting -... -Successfully completed Log splitting</pre> -</div> -</div> -<div class="paragraph"> -<p>If the output from the execute step looks good, stop the zookeeper instance you started to do the upgrade:</p> -</div> -<div class="listingblock"> -<div class="content"> -<pre class="CodeRay highlight"><code data-lang="bash">$ ./hbase/bin/hbase-daemon.sh stop zookeeper</code></pre> -</div> -</div> -<div class="paragraph"> -<p>Now start up hbase-0.96.0.</p> -</div> -</div> -</div> -<div class="sect2"> -<h3 id="s096.migration.troubleshooting"><a class="anchor" href="#s096.migration.troubleshooting"></a>13.7. Troubleshooting</h3> -<div id="s096.migration.troubleshooting.old.client" class="paragraph"> -<div class="title">Old Client connecting to 0.96 cluster</div> -<p>It will fail with an exception like the below. Upgrade.</p> -</div> -<div class="listingblock"> -<div class="content"> -<pre>17:22:15 Exception in thread "main" java.lang.IllegalArgumentException: Not a host:port pair: PBUF -17:22:15 * -17:22:15 api-compat-8.ent.cloudera.com �� ���( -17:22:15 at org.apache.hadoop.hbase.util.Addressing.parseHostname(Addressing.java:60) -17:22:15 at org.apache.hadoop.hbase.ServerName.&init>(ServerName.java:101) -17:22:15 at org.apache.hadoop.hbase.ServerName.parseVersionedServerName(ServerName.java:283) -17:22:15 at org.apache.hadoop.hbase.MasterAddressTracker.bytesToServerName(MasterAddressTracker.java:77) -17:22:15 at org.apache.hadoop.hbase.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:61) -17:22:15 at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:703) -17:22:15 at org.apache.hadoop.hbase.client.HBaseAdmin.&init>(HBaseAdmin.java:126) -17:22:15 at Client_4_3_0.setup(Client_4_3_0.java:716) -17:22:15 at Client_4_3_0.main(Client_4_3_0.java:63)</pre> -</div> -</div> -<div class="sect3"> -<h4 id="_upgrading_code_meta_code_to_use_protocol_buffers_protobuf"><a class="anchor" href="#_upgrading_code_meta_code_to_use_protocol_buffers_protobuf"></a>13.7.1. Upgrading <code>META</code> to use Protocol Buffers (Protobuf)</h4> -<div class="paragraph"> -<p>When you upgrade from versions prior to 0.96, <code>META</code> needs to be converted to use protocol buffers. This is controlled by the configuration option <code>hbase.MetaMigrationConvertingToPB</code>, which is set to <code>true</code> by default. Therefore, by default, no action is required on your part.</p> -</div> +<h3 id="upgrade1.0"><a class="anchor" href="#upgrade1.0"></a>13.3. Upgrading to 1.x</h3> <div class="paragraph"> -<p>The migration is a one-time event. However, every time your cluster starts, <code>META</code> is scanned to ensure that it does not need to be converted. If you have a very large number of regions, this scan can take a long time. Starting in 0.98.5, you can set <code>hbase.MetaMigrationConvertingToPB</code> to <code>false</code> in <em>hbase-site.xml</em>, to disable this start-up scan. This should be considered an expert-level setting.</p> -</div> +<p>Please consult the documentation published specifically for the version of HBase that you are upgrading to for details on the upgrade process.</p> </div> </div> <div class="sect2"> -<h3 id="upgrade0.94"><a class="anchor" href="#upgrade0.94"></a>13.8. Upgrading from 0.92.x to 0.94.x</h3> +<h3 id="upgrade2.0"><a class="anchor" href="#upgrade2.0"></a>13.4. Upgrading to 2.x</h3> <div class="paragraph"> -<p>We used to think that 0.92 and 0.94 were interface compatible and that you can do a rolling upgrade between these versions but then we figured that <a href="https://issues.apache.org/jira/browse/HBASE-5357";>HBASE-5357 Use builder pattern in HColumnDescriptor</a> changed method signatures so rather than return <code>void</code> they instead return <code>HColumnDescriptor</code>. This will throw <code>java.lang.NoSuchMethodError: org.apache.hadoop.hbase.HColumnDescriptor.setMaxVersions(I)V</code> so 0.92 and 0.94 are NOT compatible. You cannot do a rolling upgrade between them.</p> -</div> -</div> -<div class="sect2"> -<h3 id="upgrade0.92"><a class="anchor" href="#upgrade0.92"></a>13.9. Upgrading from 0.90.x to 0.92.x</h3> -<div class="sect3"> -<h4 id="_upgrade_guide"><a class="anchor" href="#_upgrade_guide"></a>13.9.1. Upgrade Guide</h4> -<div class="paragraph"> -<p>You will find that 0.92.0 runs a little differently to 0.90.x releases. Here are a few things to watch out for upgrading from 0.90.x to 0.92.0.</p> -</div> -<div class="admonitionblock note"> -<table> -<tr> -<td class="icon"> -<i class="fa icon-note" title="Note"></i> -</td> -<td class="content"> -<div class="title">tl:dr</div> -<div class="paragraph"> -<p>These are the important things to know before upgrading. -. Once you upgrade, you canât go back.</p> -</div> -<div class="olist arabic"> -<ol class="arabic"> -<li> -<p>MSLAB is on by default. Watch that heap usage if you have a lot of regions.</p> -</li> -<li> -<p>Distributed Log Splitting is on by default. It should make RegionServer failover faster.</p> -</li> -<li> -<p>Thereâs a separate tarball for security.</p> -</li> -<li> -<p>If <code>-XX:MaxDirectMemorySize</code> is set in your <em>hbase-env.sh</em>, itâs going to enable the experimental off-heap cache (You may not want this).</p> -</li> -</ol> -</div> -</td> -</tr> -</table> -</div> -<div class="paragraph"> -<div class="title">You canât go back!</div> -<p>To move to 0.92.0, all you need to do is shutdown your cluster, replace your HBase 0.90.x with HBase 0.92.0 binaries (be sure you clear out all 0.90.x instances) and restart (You cannot do a rolling restart from 0.90.x to 0.92.x — you must restart). On startup, the <code>.META.</code> table content is rewritten removing the table schema from the <code>info:regioninfo</code> column. Also, any flushes done post first startup will write out data in the new 0.92.0 file format, <a href="#hfilev2">HBase file format with inline blocks (version 2)</a>. This means you cannot go back to 0.90.x once youâve started HBase 0.92.0 over your HBase data directory.</p> -</div> -<div class="paragraph"> -<div class="title">MSLAB is ON by default</div> -<p>In 0.92.0, the <code><a href="#hbase.hregion.memstore.mslab.enabled">hbase.hregion.memstore.mslab.enabled</a></code> flag is set to <code>true</code> (See <a href="#gcpause">Long GC pauses</a>). In 0.90.x it was false. When it is enabled, memstores will step allocate memory in MSLAB 2MB chunks even if the memstore has zero or just a few small elements. This is fine usually but if you had lots of regions per RegionServer in a 0.90.x cluster (and MSLAB was off), you may find yourself OOME’ing on upgrade because the <code>thousands of regions * number of column families * 2MB MSLAB</code> (at a minimum) puts your heap over the top. Set <code>hbase.hregion.memstore.mslab.enabled</code> to <code>false</code> or set the MSLAB size down from 2MB by setting <code>hbase.hregion.memstore.mslab.chunksize</code> to something less.</p> -</div> -<div id="dls" class="paragraph"> -<div class="title">Distributed Log Splitting is on by default</div> -<p>Previous, WAL logs on crash were split by the Master alone. In 0.92.0, log splitting is done by the cluster (See <a href="https://issues.apache.org/jira/browse/hbase-1364";>HBASE-1364 [performance] Distributed splitting of regionserver commit logs</a> or see the blog post <a href="http://blog.cloudera.com/blog/2012/07/hbase-log-splitting/";>Apache HBase Log Splitting</a>). This should cut down significantly on the amount of time it takes splitting logs and getting regions back online again.</p> -</div> -<div class="paragraph"> -<div class="title">Memory accounting is different now</div> -<p>In 0.92.0, <a href="#hfilev2">HBase file format with inline blocks (version 2)</a> indices and bloom filters take up residence in the same LRU used caching blocks that come from the filesystem. In 0.90.x, the HFile v1 indices lived outside of the LRU so they took up space even if the index was on a âcoldâ file, one that wasnât being actively used. With the indices now in the LRU, you may find you have less space for block caching. Adjust your block cache accordingly. See the <a href="#block.cache">Block Cache</a> for more detail. The block size default size has been changed in 0.92.0 from 0.2 (20 percent of heap) to 0.25.</p> -</div> -<div class="paragraph"> -<div class="title">On the Hadoop version to use</div> -<p>Run 0.92.0 on Hadoop 1.0.x (or CDH3u3). The performance benefits are worth making the move. Otherwise, our Hadoop prescription is as it has been; you need an Hadoop that supports a working sync. See <a href="#hadoop"><a href="https://hadoop.apache.org";>Hadoop</a></a>.</p> -</div> -<div class="paragraph"> -<p>If running on Hadoop 1.0.x (or CDH3u3), enable local read. See <a href="http://files.meetup.com/1350427/hug_ebay_jdcryans.pdf";>Practical Caching</a> presentation for ruminations on the performance benefits âgoing localâ (and for how to enable local reads).</p> -</div> -<div class="paragraph"> -<div class="title">HBase 0.92.0 ships with ZooKeeper 3.4.2</div> -<p>If you can, upgrade your ZooKeeper. If you canât, 3.4.2 clients should work against 3.3.X ensembles (HBase makes use of 3.4.2 API).</p> -</div> -<div class="paragraph"> -<div class="title">Online alter is off by default</div> -<p>In 0.92.0, weâve added an experimental online schema alter facility (See <a href="#hbase.online.schema.update.enable">hbase.online.schema.update.enable</a>). It’s off by default. Enable it at your own risk. Online alter and splitting tables do not play well together so be sure your cluster quiescent using this feature (for now).</p> -</div> -<div class="paragraph"> -<div class="title">WebUI</div> -<p>The web UI has had a few additions made in 0.92.0. It now shows a list of the regions currently transitioning, recent compactions/flushes, and a process list of running processes (usually empty if all is well and requests are being handled promptly). Other additions including requests by region, a debugging servlet dump, etc.</p> -</div> -<div class="paragraph"> -<div class="title">Security tarball</div> -<p>We now ship with two tarballs; secure and insecure HBase. Documentation on how to setup a secure HBase is on the way.</p> -</div> -<div class="paragraph"> -<div class="title">Changes in HBase replication</div> -<p>0.92.0 adds two new features: multi-slave and multi-master replication. The way to enable this is the same as adding a new peer, so in order to have multi-master you would just run add_peer for each cluster that acts as a master to the other slave clusters. Collisions are handled at the timestamp level which may or may not be what you want, this needs to be evaluated on a per use case basis. Replication is still experimental in 0.92 and is disabled by default, run it at your own risk.</p> -</div> -<div class="paragraph"> -<div class="title">RegionServer now aborts if OOME</div> -<p>If an OOME, we now have the JVM kill -9 the RegionServer process so it goes down fast. Previous, a RegionServer might stick around after incurring an OOME limping along in some wounded state. To disable this facility, and recommend you leave it in place, youâd need to edit the bin/hbase file. Look for the addition of the -XX:OnOutOfMemoryError="kill -9 %p" arguments (See <a href="https://issues.apache.org/jira/browse/HBASE-4769";>HBASE-4769 - âAbort RegionServer Immediately on OOMEâ</a>).</p> -</div> -<div class="paragraph"> -<div class="title">HFile v2 and the âBigger, Fewerâ Tendency</div> -<p>0.92.0 stores data in a new format, <a href="#hfilev2">HBase file format with inline blocks (version 2)</a>. As HBase runs, it will move all your data from HFile v1 to HFile v2 format. This auto-migration will run in the background as flushes and compactions run. HFile v2 allows HBase run with larger regions/files. In fact, we encourage that all HBasers going forward tend toward Facebook axiom #1, run with larger, fewer regions. If you have lots of regions now — more than 100s per host — you should look into setting your region size up after you move to 0.92.0 (In 0.92.0, default size is now 1G, up from 256M), and then running online merge tool (See <a href="https://issues.apache.org/jira/browse/HBASE-1621";>HBASE-1621 merge tool should work on online cluster, but disabled table</a>).</p> -</div> -</div> -</div> -<div class="sect2"> -<h3 id="upgrade0.90"><a class="anchor" href="#upgrade0.90"></a>13.10. Upgrading to HBase 0.90.x from 0.20.x or 0.89.x</h3> -<div class="paragraph"> -<p>This version of 0.90.x HBase can be started on data written by HBase 0.20.x or HBase 0.89.x. There is no need of a migration step. HBase 0.89.x and 0.90.x does write out the name of region directories differently — it names them with a md5 hash of the region name rather than a jenkins hash — so this means that once started, there is no going back to HBase 0.20.x.</p> -</div> -<div class="paragraph"> -<p>Be sure to remove the <em>hbase-default.xml</em> from your <em>conf</em> directory on upgrade. A 0.20.x version of this file will have sub-optimal configurations for 0.90.x HBase. The <em>hbase-default.xml</em> file is now bundled into the HBase jar and read from there. If you would like to review the content of this file, see it in the src tree at <em>src/main/resources/hbase-default.xml</em> or see <a href="#hbase_default_configurations">HBase Default Configuration</a>.</p> -</div> -<div class="paragraph"> -<p>Finally, if upgrading from 0.20.x, check your .META. schema in the shell. In the past we would recommend that users run with a 16kb MEMSTORE_FLUSHSIZE. Run</p> -</div> -<div class="listingblock"> -<div class="content"> -<pre>hbase> scan '-ROOT-'</pre> -</div> -</div> -<div class="paragraph"> -<p>in the shell. This will output the current <code>.META.</code> schema. Check <code>MEMSTORE_FLUSHSIZE</code> size. Is it 16kb (16384)? If so, you will need to change this (The 'normal'/default value is 64MB (67108864)). Run the script <code>bin/set_meta_memstore_size.rb</code>. This will make the necessary edit to your <code>.META.</code> schema. Failure to run this change will make for a slow cluster. See <a href="https://issues.apache.org/jira/browse/HBASE-3499";>HBASE-3499 Users upgrading to 0.90.0 need to have their .META. table updated with the right MEMSTORE_SIZE</a>.</p> +<p>Coming soon…​</p> </div> </div> </div> @@ -16024,11 +15525,14 @@ For instance, the following command creates a table with regions that split at e <dd> <p>The RegionSplitter tool is provided with HBase, and uses a <em>SplitAlgorithm</em> to determine split points for you. As parameters, you give it the algorithm, desired number of regions, and column families. -It includes two split algorithms. +It includes three split algorithms. The first is the <code><a href="https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/util/RegionSplitter.HexStringSplit.html";>HexStringSplit</a></code> algorithm, which assumes the row keys are hexadecimal strings. -The second, +The second is the +<code><a href="https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/util/RegionSplitter.DecimalStringSplit.html";>DecimalStringSplit</a></code> +algorithm, which assumes the row keys are decimal strings in the range 00000000 to 99999999. +The third, <code><a href="https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/util/RegionSplitter.UniformSplit.html";>UniformSplit</a></code>, assumes the row keys are random byte arrays. You will probably need to develop your own @@ -18139,7 +17643,7 @@ hbase> major_compact 't1', 'c1â, âMOBâ</pre> </div> </div> </div> -<h1 id="casestudies" class="sect0"><a class="anchor" href="#casestudies"></a>Backup and Restore</h1> +<h1 id="backuprestore" class="sect0"><a class="anchor" href="#backuprestore"></a>Backup and Restore</h1> <div class="sect1"> <h2 id="br.overview"><a class="anchor" href="#br.overview"></a>76. Overview</h2> <div class="sectionbody"> @@ -24116,7 +23620,7 @@ In the worst case, if you really need to collocate both, set MR to use less Map <h2 id="perf.casestudy"><a class="anchor" href="#perf.casestudy"></a>121. Case Studies</h2> <div class="sectionbody"> <div class="paragraph"> -<p>For Performance and Troubleshooting Case Studies, see <a href="#casestudies">Backup and Restore</a>.</p> +<p>For Performance and Troubleshooting Case Studies, see <a href="#casestudies">Apache HBase Case Studies</a>.</p> </div> </div> </div> @@ -24214,6 +23718,11 @@ Analyze.</p> </div> <div class="sect2"> <h3 id="trouble.log.gc"><a class="anchor" href="#trouble.log.gc"></a>123.3. JVM Garbage Collection Logs</h3> +<div class="listingblock"> +<div class="content"> +<pre>All example Garbage Collection logs in this section are based on Java 8 output. The introduction of Unified Logging in Java 9 and newer will result in very different looking logs.</pre> +</div> +</div> <div class="paragraph"> <p>HBase is memory intensive, and using the default GC you can see long pauses in all threads including the <em>Juliet Pause</em> aka "GC of Death". To help debug this or confirm this is happening GC logging can be turned on in the Java virtual machine.</p> </div> @@ -24392,8 +23901,7 @@ A quality question that includes all context and exhibits evidence the author ha <div class="sect3"> <h4 id="trouble.tools.builtin.webmaster"><a class="anchor" href="#trouble.tools.builtin.webmaster"></a>125.1.1. Master Web Interface</h4> <div class="paragraph"> -<p>The Master starts a web-interface on port 16010 by default. -(Up to and including 0.98 this was port 60010)</p> +<p>The Master starts a web-interface on port 16010 by default.</p> </div> <div class="paragraph"> <p>The Master web UI lists created tables and their definition (e.g., ColumnFamilies, blocksize, etc.). Additionally, the available RegionServers in the cluster are listed along with selected high-level metrics (requests, number of regions, usedHeap, maxHeap). The Master web UI allows navigation to each RegionServer’s web UI.</p> @@ -24402,8 +23910,7 @@ A quality question that includes all context and exhibits evidence the author ha <div class="sect3"> <h4 id="trouble.tools.builtin.webregion"><a class="anchor" href="#trouble.tools.builtin.webregion"></a>125.1.2. RegionServer Web Interface</h4> <div class="paragraph"> -<p>RegionServers starts a web-interface on port 16030 by default. -(Up to an including 0.98 this was port 60030)</p> +<p>RegionServers starts a web-interface on port 16030 by default.</p> </div> <div class="paragraph"> <p>The RegionServer web UI lists online regions and their start/end keys, as well as point-in-time RegionServer metrics (requests, regions, storeFileIndexSize, compactionQueueSize, etc.).</p> @@ -24752,18 +24259,7 @@ You can also tail all the logs at the same time, edit files, etc.</p> <p>For more information on the HBase client, see <a href="#architecture.client">client</a>.</p> </div> <div class="sect2"> -<h3 id="_missed_scan_results_due_to_mismatch_of_code_hbase_client_scanner_max_result_size_code_between_client_and_server"><a class="anchor" href="#_missed_scan_results_due_to_mismatch_of_code_hbase_client_scanner_max_result_size_code_between_client_and_server"></a>126.1. Missed Scan Results Due To Mismatch Of <code>hbase.client.scanner.max.result.size</code> Between Client and Server</h3> -<div class="paragraph"> -<p>If either the client or server version is lower than 0.98.11/1.0.0 and the server -has a smaller value for <code>hbase.client.scanner.max.result.size</code> than the client, scan -requests that reach the server’s <code>hbase.client.scanner.max.result.size</code> are likely -to miss data. In particular, 0.98.11 defaults <code>hbase.client.scanner.max.result.size</code> -to 2 MB but other versions default to larger values. For this reason, be very careful -using 0.98.11 servers with any other client version.</p> -</div> -</div> -<div class="sect2"> -<h3 id="trouble.client.scantimeout"><a class="anchor" href="#trouble.client.scantimeout"></a>126.2. ScannerTimeoutException or UnknownScannerException</h3> +<h3 id="trouble.client.scantimeout"><a class="anchor" href="#trouble.client.scantimeout"></a>126.1. ScannerTimeoutException or UnknownScannerException</h3> <div class="paragraph"> <p>This is thrown if the time between RPC calls from the client to RegionServer exceeds the scan timeout. For example, if <code>Scan.setCaching</code> is set to 500, then there will be an RPC call to fetch the next batch of rows every 500 <code>.next()</code> calls on the ResultScanner because data is being transferred in blocks of 500 rows to the client. @@ -24774,7 +24270,7 @@ Reducing the setCaching value may be an option, but setting this value too low m </div> </div> <div class="sect2"> -<h3 id="_performance_differences_in_thrift_and_java_apis"><a class="anchor" href="#_performance_differences_in_thrift_and_java_apis"></a>126.3. Performance Differences in Thrift and Java APIs</h3> +<h3 id="_performance_differences_in_thrift_and_java_apis"><a class="anchor" href="#_performance_differences_in_thrift_and_java_apis"></a>126.2. Performance Differences in Thrift and Java APIs</h3> <div class="paragraph"> <p>Poor performance, or even <code>ScannerTimeoutExceptions</code>, can occur if <code>Scan.setCaching</code> is too high, as discussed in <a href="#trouble.client.scantimeout">ScannerTimeoutException or UnknownScannerException</a>. If the Thrift client uses the wrong caching settings for a given workload, performance can suffer compared to the Java API. @@ -24786,7 +24282,7 @@ In one case, it was found that reducing the cache for Thrift scans from 1000 to </div> </div> <div class="sect2"> -<h3 id="trouble.client.lease.exception"><a class="anchor" href="#trouble.client.lease.exception"></a>126.4. <code>LeaseException</code> when calling <code>Scanner.next</code></h3> +<h3 id="trouble.client.lease.exception"><a class="anchor" href="#trouble.client.lease.exception"></a>126.3. <code>LeaseException</code> when calling <code>Scanner.next</code></h3> <div class="paragraph"> <p>In some situations clients that fetch data from a RegionServer get a LeaseException instead of the usual <a href="#trouble.client.scantimeout">ScannerTimeoutException or UnknownScannerException</a>. Usually the source of the exception is <code>org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:230)</code> (line number may vary). It tends to happen in the context of a slow/freezing <code>RegionServer#next</code> call. @@ -24795,7 +24291,7 @@ Harsh J investigated the issue as part of the mailing list thread <a href="https </div> </div> <div class="sect2"> -<h3 id="trouble.client.scarylogs"><a class="anchor" href="#trouble.client.scarylogs"></a>126.5. Shell or client application throws lots of scary exceptions during normal operation</h3> +<h3 id="trouble.client.scarylogs"><a class="anchor" href="#trouble.client.scarylogs"></a>126.4. Shell or client application throws lots of scary exceptions during normal operation</h3> <div class="paragraph"> <p>Since 0.20.0 the default log level for `org.apache.hadoop.hbase.*`is DEBUG.</p> </div> @@ -24804,7 +24300,7 @@ Harsh J investigated the issue as part of the mailing list thread <a href="https </div> </div> <div class="sect2"> -<h3 id="trouble.client.longpauseswithcompression"><a class="anchor" href="#trouble.client.longpauseswithcompression"></a>126.6. Long Client Pauses With Compression</h3> +<h3 id="trouble.client.longpauseswithcompression"><a class="anchor" href="#trouble.client.longpauseswithcompression"></a>126.5. Long Client Pauses With Compression</h3> <div class="paragraph"> <p>This is a fairly frequent question on the Apache HBase dist-list. The scenario is that a client is typically inserting a lot of data into a relatively un-optimized HBase cluster. @@ -24830,7 +24326,7 @@ Without compression the files are much bigger and don’t need as much compa </div> </div> <div class="sect2"> -<h3 id="trouble.client.security.rpc.krb"><a class="anchor" href="#trouble.client.security.rpc.krb"></a>126.7. Secure Client Connect ([Caused by GSSException: No valid credentials provided…​])</h3> +<h3 id="trouble.client.security.rpc.krb"><a class="anchor" href="#trouble.client.security.rpc.krb"></a>126.6. Secure Client Connect ([Caused by GSSException: No valid credentials provided…​])</h3> <div class="paragraph"> <p>You may encounter the following error:</p> </div> @@ -24853,7 +24349,7 @@ See JIRA <a href="https://issues.apache.org/jira/browse/HBASE-10379";>HBASE-10379 </div> </div> <div class="sect2"> -<h3 id="trouble.client.zookeeper"><a class="anchor" href="#trouble.client.zookeeper"></a>126.8. ZooKeeper Client Connection Errors</h3> +<h3 id="trouble.client.zookeeper"><a class="anchor" href="#trouble.client.zookeeper"></a>126.7. ZooKeeper Client Connection Errors</h3> <div class="paragraph"> <p>Errors like this…​</p> </div> @@ -24885,7 +24381,7 @@ See JIRA <a href="https://issues.apache.org/jira/browse/HBASE-10379";>HBASE-10379 </div> </div> <div class="sect2"> -<h3 id="trouble.client.oome.directmemory.leak"><a class="anchor" href="#trouble.client.oome.directmemory.leak"></a>126.9. Client running out of memory though heap size seems to be stable (but the off-heap/direct heap keeps growing)</h3> +<h3 id="trouble.client.oome.directmemory.leak"><a class="anchor" href="#trouble.client.oome.directmemory.leak"></a>126.8. Client running out of memory though heap size seems to be stable (but the off-heap/direct heap keeps growing)</h3> <div class="paragraph"> <p>You are likely running into the issue that is described and worked through in the mail thread <a href="http://search-hadoop.com/m/ubhrX8KvcH/Suspected+memory+leak&subj=Re+Suspected+memory+leak";>HBase, mail # user - Suspected memory leak</a> and continued over in <a href="http://search-hadoop.com/m/p2Agc1Zy7Va/MaxDirectMemorySize+Was%253A+Suspected+memory+leak&subj=Re+FeedbackRe+Suspected+memory+leak";>HBase, mail # dev - FeedbackRe: Suspected memory leak</a>. A workaround is passing your client-side JVM a reasonable value for <code>-XX:MaxDirectMemorySize</code>. @@ -24894,14 +24390,7 @@ You want to make this setting client-side only especially if you are running the </div> </div> <div class="sect2"> -<h3 id="trouble.client.slowdown.admin"><a class="anchor" href="#trouble.client.slowdown.admin"></a>126.10. Client Slowdown When Calling Admin Methods (flush, compact, etc.)</h3> -<div class="paragraph"> -<p>This is a client issue fixed by <a href="https://issues.apache.org/jira/browse/HBASE-5073";>HBASE-5073</a> in 0.90.6. -There was a ZooKeeper leak in the client and the client was getting pummeled by ZooKeeper events with each additional invocation of the admin API.</p> -</div> -</div> -<div class="sect2"> -<h3 id="trouble.client.security.rpc"><a class="anchor" href="#trouble.client.security.rpc"></a>126.11. Secure Client Cannot Connect ([Caused by GSSException: No valid credentials provided(Mechanism level: Failed to find any Kerberos tgt)])</h3> +<h3 id="trouble.client.security.rpc"><a class="anchor" href="#trouble.client.security.rpc"></a>126.9. Secure Client Cannot Connect ([Caused by GSSException: No valid credentials provided(Mechanism level: Failed to find any Kerberos tgt)])</h3> <div class="paragraph"> <p>There can be several causes that produce this symptom.</p> </div> @@ -25149,8 +24638,7 @@ remember that WALs are saved when replication is disabled, as long as there are <div class="sect2"> <h3 id="trouble.network.loopback"><a class="anchor" href="#trouble.network.loopback"></a>129.2. Loopback IP</h3> <div class="paragraph"> -<p>HBase expects the loopback IP Address to be 127.0.0.1. -See the Getting Started section on <a href="#loopback.ip">[loopback.ip]</a>.</p> +<p>HBase expects the loopback IP Address to be 127.0.0.1.</p> </div> </div> <div class="sect2"> @@ -25370,15 +24858,7 @@ This exception is returned back to the client and then the client goes back to < </div> </div> <div class="sect3"> -<h4 id="trouble.rs.runtime.double_listed_regions"><a class="anchor" href="#trouble.rs.runtime.double_listed_regions"></a>130.2.9. Regions listed by domain name, then IP</h4> -<div class="paragraph"> -<p>Fix your DNS. -In versions of Apache HBase before 0.92.x, reverse DNS needs to give same answer as forward lookup. -See <a href="https://issues.apache.org/jira/browse/HBASE-3431";>HBASE 3431 RegionServer is not using the name given it by the master; double entry in master listing of servers</a> for gory details.</p> -</div> -</div> -<div class="sect3"> -<h4 id="brand.new.compressor"><a class="anchor" href="#brand.new.compressor"></a>130.2.10. Logs flooded with '2011-01-10 12:40:48,407 INFO org.apache.hadoop.io.compress.CodecPool: Gotbrand-new compressor' messages</h4> +<h4 id="brand.new.compressor"><a class="anchor" href="#brand.new.compressor"></a>130.2.9. Logs flooded with '2011-01-10 12:40:48,407 INFO org.apache.hadoop.io.compress.CodecPool: Gotbrand-new compressor' messages</h4> <div class="paragraph"> <p>We are not using the native versions of compression libraries. See <a href="https://issues.apache.org/jira/browse/HBASE-1900";>HBASE-1900 Put back native support when hadoop 0.21 is released</a>. @@ -25386,7 +24866,7 @@ Copy the native libs from hadoop under HBase lib dir or symlink them into place </div> </div> <div class="sect3"> -<h4 id="trouble.rs.runtime.client_went_away"><a class="anchor" href="#trouble.rs.runtime.client_went_away"></a>130.2.11. Server handler X on 60020 caught: java.nio.channels.ClosedChannelException</h4> +<h4 id="trouble.rs.runtime.client_went_away"><a class="anchor" href="#trouble.rs.runtime.client_went_away"></a>130.2.10. Server handler X on 60020 caught: java.nio.channels.ClosedChannelException</h4> <div class="paragraph"> <p>If you see this type of message it means that the region server was trying to read/send data from/to a client but it already went away. Typical causes for this are if the client was killed (you see a storm of messages like this when a MapReduce job is killed or fails) or if the client receives a SocketTimeoutException. @@ -25536,33 +25016,7 @@ Search for old threads using <a href="http://search-hadoop.com/";>Search Hadoop</ <h2 id="trouble.versions"><a class="anchor" href="#trouble.versions"></a>134. HBase and Hadoop version issues</h2> <div class="sectionbody"> <div class="sect2"> -<h3 id="trouble.versions.205"><a class="anchor" href="#trouble.versions.205"></a>134.1. <code>NoClassDefFoundError</code> when trying to run 0.90.x on hadoop-0.20.205.x (or hadoop-1.0.x)</h3> -<div class="paragraph"> -<p>Apache HBase 0.90.x does not ship with hadoop-0.20.205.x, etc. -To make it run, you need to replace the hadoop jars that Apache HBase shipped with in its <em>lib</em> directory with those of the Hadoop you want to run HBase on. -If even after replacing Hadoop jars you get the below exception:</p> -</div> -<div class="listingblock"> -<div class="content"> -<pre class="CodeRay highlight"><code data-lang="java">sv4r6s38: <span class="exception">Exception</span> in thread <span class="string"><span class="delimiter">"</span><span class="content">main</span><span class="delimiter">"</span></span> java.lang.NoClassDefFoundError: org/apache/commons/configuration/<span class="predefined-type">Configuration</span> -sv4r6s38: at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:<span class="integer">37</span>) -sv4r6s38: at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:<span class="integer">34</span>) -sv4r6s38: at org.apache.hadoop.security.UgiInstrumentation.create(UgiInstrumentation.java:<span class="integer">51</span>) -sv4r6s38: at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:<span class="integer">209</span>) -sv4r6s38: at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:<span class="integer">177</span>) -sv4r6s38: at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:<span class="integer">229</span>) -sv4r6s38: at org.apache.hadoop.security.KerberosName.<clinit>(KerberosName.java:<span class="integer">83</span>) -sv4r6s38: at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:<span class="integer">202</span>) -sv4r6s38: at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:<span class="integer">177</span>)</code></pre> -</div> -</div> -<div class="paragraph"> -<p>you need to copy under <em>hbase/lib</em>, the <em>commons-configuration-X.jar</em> you find in your Hadoop’s <em>lib</em> directory. -That should fix the above complaint.</p> -</div> -</div> -<div class="sect2"> -<h3 id="trouble.wrong.version"><a class="anchor" href="#trouble.wrong.version"></a>134.2. …​cannot communicate with client version…​</h3> +<h3 id="trouble.wrong.version"><a class="anchor" href="#trouble.wrong.version"></a>134.1. …​cannot communicate with client version…​</h3> <div class="paragraph"> <p>If you see something like the following in your logs <span class="computeroutput">... 2012-09-24 10:20:52,168 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting @@ -25573,96 +25027,7 @@ That should fix the above complaint.</p> </div> </div> <div class="sect1"> -<h2 id="_ipc_configuration_conflicts_with_hadoop"><a class="anchor" href="#_ipc_configuration_conflicts_with_hadoop"></a>135. IPC Configuration Conflicts with Hadoop</h2> -<div class="sectionbody"> -<div class="paragraph"> -<p>If the Hadoop configuration is loaded after the HBase configuration, and you have configured custom IPC settings in both HBase and Hadoop, the Hadoop values may overwrite the HBase values. -There is normally no need to change these settings for HBase, so this problem is an edge case. -However, <a href="https://issues.apache.org/jira/browse/HBASE-11492";>HBASE-11492</a> renames these settings for HBase to remove the chance of a conflict. -Each of the setting names have been prefixed with <code>hbase.</code>, as shown in the following table. -No action is required related to these changes unless you are already experiencing a conflict.</p> -</div> -<div class="paragraph"> -<p>These changes were backported to HBase 0.98.x and apply to all newer versions.</p> -</div> -<table class="tableblock frame-all grid-all spread"> -<colgroup> -<col style="width: 50%;"> -<col style="width: 50%;"> -</colgroup> -<thead> -<tr> -<th class="tableblock halign-left valign-top">Pre-0.98.x</th> -<th class="tableblock halign-left valign-top">0.98-x And Newer</th> -</tr> -</thead> -<tbody> -<tr> -<td class="tableblock halign-left valign-top"><p class="tableblock">ipc.server.listen.queue.size</p></td> -<td class="tableblock halign-left valign-top"><p class="tableblock">hbase.ipc.server.listen.queue.size</p></td> -</tr> -<tr> -<td class="tableblock halign-left valign-top"><p class="tableblock">ipc.server.max.callqueue.size</p></td> -<td class="tableblock halign-left valign-top"><p class="tableblock">hbase.ipc.server.max.callqueue.size</p></td> -</tr> -<tr> -<td class="tableblock halign-left valign-top"><p class="tableblock">ipc.server.callqueue.handler.factor</p></td> -<td class="tableblock halign-left valign-top"><p class="tableblock">hbase.ipc.server.callqueue.handler.factor</p></td> -</tr> -<tr> -<td class="tableblock halign-left valign-top"><p class="tableblock">ipc.server.callqueue.read.share</p></td> -<td class="tableblock halign-left valign-top"><p class="tableblock">hbase.ipc.server.callqueue.read.share</p></td> -</tr> -<tr> -<td class="tableblock halign-left valign-top"><p class="tableblock">ipc.server.callqueue.type</p></td> -<td class="tableblock halign-left valign-top"><p class="tableblock">hbase.ipc.server.callqueue.type</p></td> -</tr> -<tr> -<td class="tableblock halign-left valign-top"><p class="tableblock">ipc.server.queue.max.call.delay</p></td> -<td class="tableblock halign-left valign-top"><p class="tableblock">hbase.ipc.server.queue.max.call.delay</p></td> -</tr> -<tr> -<td class="tableblock halign-left valign-top"><p class="tableblock">ipc.server.max.callqueue.length</p></td> -<td class="tableblock halign-left valign-top"><p class="tableblock">hbase.ipc.server.max.callqueue.length</p></td> -</tr> -<tr> -<td class="tableblock halign-left valign-top"><p class="tableblock">ipc.server.read.threadpool.size</p></td> -<td class="tableblock halign-left valign-top"><p class="tableblock">hbase.ipc.server.read.threadpool.size</p></td> -</tr> -<tr> -<td class="tableblock halign-left valign-top"><p class="tableblock">ipc.server.tcpkeepalive</p></td> -<td class="tableblock halign-left valign-top"><p class="tableblock">hbase.ipc.server.tcpkeepalive</p></td> -</tr> -<tr> -<td class="tableblock halign-left valign-top"><p class="tableblock">ipc.server.tcpnodelay</p></td> -<td class="tableblock halign-left valign-top"><p class="tableblock">hbase.ipc.server.tcpnodelay</p></td> -</tr> -<tr> -<td class="tableblock halign-left valign-top"><p class="tableblock">ipc.client.call.purge.timeout</p></td> -<td class="tableblock halign-left valign-top"><p class="tableblock">hbase.ipc.client.call.purge.timeout</p></td> -</tr> -<tr> -<td class="tableblock halign-left valign-top"><p class="tableblock">ipc.client.connection.maxidletime</p></td> -<td class="tableblock halign-left valign-top"><p class="tableblock">hbase.ipc.client.connection.maxidletime</p></td> -</tr> -<tr> -<td class="tableblock halign-left valign-top"><p class="tableblock">ipc.client.idlethreshold</p></td> -<td class="tableblock halign-left valign-top"><p class="tableblock">hbase.ipc.client.idlethreshold</p></td> -</tr> -<tr> -<td class="tableblock halign-left valign-top"><p class="tableblock">ipc.client.kill.max</p></td> -<td class="tableblock halign-left valign-top"><p class="tableblock">hbase.ipc.client.kill.max</p></td> -</tr> -<tr> -<td class="tableblock halign-left valign-top"><p class="tableblock">ipc.server.scan.vtime.weight</p></td> -<td class="tableblock halign-left valign-top"><p class="tableblock">hbase.ipc.server.scan.vtime.weight</p></td> -</tr> -</tbody> -</table> -</div> -</div> -<div class="sect1"> -<h2 id="_hbase_and_hdfs"><a class="anchor" href="#_hbase_and_hdfs"></a>136. HBase and HDFS</h2> +<h2 id="_hbase_and_hdfs"><a class="anchor" href="#_hbase_and_hdfs"></a>135. HBase and HDFS</h2> <div class="sectionbody"> <div class="paragraph"> <p>General configuration guidance for Apache HDFS is out of the scope of this guide. @@ -25802,10 +25167,10 @@ The default is 8 minutes, expressed as milliseconds.</p> </div> </div> <div class="sect1"> -<h2 id="trouble.tests"><a class="anchor" href="#trouble.tests"></a>137. Running unit or integration tests</h2> +<h2 id="trouble.tests"><a class="anchor" href="#trouble.tests"></a>136. Running unit or integration tests</h2> <div class="sectionbody"> <div class="sect2"> -<h3 id="trouble.hdfs_2556"><a class="anchor" href="#trouble.hdfs_2556"></a>137.1. Runtime exceptions from MiniDFSCluster when running tests</h3> +<h3 id="trouble.hdfs_2556"><a class="anchor" href="#trouble.hdfs_2556"></a>136.1. Runtime exceptions from MiniDFSCluster when running tests</h3> <div class="paragraph"> <p>If you see something like the following</p> </div> @@ -25844,18 +25209,18 @@ This is a workaround for <a href="https://issues.apache.org/jira/browse/HDFS-255 </div> </div> <div class="sect1"> -<h2 id="trouble.casestudy"><a class="anchor" href="#trouble.casestudy"></a>138.
<TRUNCATED>
