http://git-wip-us.apache.org/repos/asf/hbase-site/blob/c4b2cc17/book.html ---------------------------------------------------------------------- diff --git a/book.html b/book.html index e96c1ac..5df6ac7 100644 --- a/book.html +++ b/book.html @@ -258,46 +258,47 @@ <li><a href="#ops.capacity">155. Capacity Planning and Region Sizing</a></li> <li><a href="#table.rename">156. Table Rename</a></li> <li><a href="#rsgroup">157. RegionServer Grouping</a></li> +<li><a href="#normalizer">158. Region Normalizer</a></li> </ul> </li> <li><a href="#developer">Building and Developing Apache HBase</a> <ul class="sectlevel1"> -<li><a href="#getting.involved">158. Getting Involved</a></li> -<li><a href="#repos">159. Apache HBase Repositories</a></li> -<li><a href="#_ides">160. IDEs</a></li> -<li><a href="#build">161. Building Apache HBase</a></li> -<li><a href="#releasing">162. Releasing Apache HBase</a></li> -<li><a href="#hbase.rc.voting">163. Voting on Release Candidates</a></li> -<li><a href="#documentation">164. Generating the HBase Reference Guide</a></li> -<li><a href="#hbase.org">165. Updating <a href="https://hbase.apache.org";>hbase.apache.org</a></a></li> -<li><a href="#hbase.tests">166. Tests</a></li> -<li><a href="#developing">167. Developer Guidelines</a></li> +<li><a href="#getting.involved">159. Getting Involved</a></li> +<li><a href="#repos">160. Apache HBase Repositories</a></li> +<li><a href="#_ides">161. IDEs</a></li> +<li><a href="#build">162. Building Apache HBase</a></li> +<li><a href="#releasing">163. Releasing Apache HBase</a></li> +<li><a href="#hbase.rc.voting">164. Voting on Release Candidates</a></li> +<li><a href="#documentation">165. Generating the HBase Reference Guide</a></li> +<li><a href="#hbase.org">166. Updating <a href="https://hbase.apache.org";>hbase.apache.org</a></a></li> +<li><a href="#hbase.tests">167. Tests</a></li> +<li><a href="#developing">168. Developer Guidelines</a></li> </ul> </li> <li><a href="#unit.tests">Unit Testing HBase Applications</a> <ul class="sectlevel1"> -<li><a href="#_junit">168. JUnit</a></li> -<li><a href="#mockito">169. Mockito</a></li> -<li><a href="#_mrunit">170. MRUnit</a></li> -<li><a href="#_integration_testing_with_an_hbase_mini_cluster">171. Integration Testing with an HBase Mini-Cluster</a></li> +<li><a href="#_junit">169. JUnit</a></li> +<li><a href="#mockito">170. Mockito</a></li> +<li><a href="#_mrunit">171. MRUnit</a></li> +<li><a href="#_integration_testing_with_an_hbase_mini_cluster">172. Integration Testing with an HBase Mini-Cluster</a></li> </ul> </li> <li><a href="#protobuf">Protobuf in HBase</a> <ul class="sectlevel1"> -<li><a href="#_protobuf">172. Protobuf</a></li> +<li><a href="#_protobuf">173. Protobuf</a></li> </ul> </li> <li><a href="#zookeeper">ZooKeeper</a> <ul class="sectlevel1"> -<li><a href="#_using_existing_zookeeper_ensemble">173. Using existing ZooKeeper ensemble</a></li> -<li><a href="#zk.sasl.auth">174. SASL Authentication with ZooKeeper</a></li> +<li><a href="#_using_existing_zookeeper_ensemble">174. Using existing ZooKeeper ensemble</a></li> +<li><a href="#zk.sasl.auth">175. SASL Authentication with ZooKeeper</a></li> </ul> </li> <li><a href="#community">Community</a> <ul class="sectlevel1"> -<li><a href="#_decisions">175. Decisions</a></li> -<li><a href="#community.roles">176. Community Roles</a></li> -<li><a href="#hbase.commit.msg.format">177. Commit Message format</a></li> +<li><a href="#_decisions">176. Decisions</a></li> +<li><a href="#community.roles">177. Community Roles</a></li> +<li><a href="#hbase.commit.msg.format">178. Commit Message format</a></li> </ul> </li> <li><a href="#_appendix">Appendix</a> @@ -307,7 +308,7 @@ <li><a href="#hbck.in.depth">Appendix C: hbck In Depth</a></li> <li><a href="#appendix_acl_matrix">Appendix D: Access Control Matrix</a></li> <li><a href="#compression">Appendix E: Compression and Data Block Encoding In HBase</a></li> -<li><a href="#data.block.encoding.enable">178. Enable Data Block Encoding</a></li> +<li><a href="#data.block.encoding.enable">179. Enable Data Block Encoding</a></li> <li><a href="#sql">Appendix F: SQL over HBase</a></li> <li><a href="#ycsb">Appendix G: YCSB</a></li> <li><a href="#_hfile_format_2">Appendix H: HFile format</a></li> @@ -316,8 +317,8 @@ <li><a href="#asf">Appendix K: HBase and the Apache Software Foundation</a></li> <li><a href="#orca">Appendix L: Apache HBase Orca</a></li> <li><a href="#tracing">Appendix M: Enabling Dapper-like Tracing in HBase</a></li> -<li><a href="#tracing.client.modifications">179. Client Modifications</a></li> -<li><a href="#tracing.client.shell">180. Tracing from HBase Shell</a></li> +<li><a href="#tracing.client.modifications">180. Client Modifications</a></li> +<li><a href="#tracing.client.shell">181. Tracing from HBase Shell</a></li> <li><a href="#hbase.rpc">Appendix N: 0.95 RPC Specification</a></li> </ul> </li> @@ -2154,21 +2155,6 @@ Some configurations would only appear in source code; the only way to identify t </dd> </dl> </div> -<div id="hbase.fs.tmp.dir" class="dlist"> -<dl> -<dt class="hdlist1"><code>hbase.fs.tmp.dir</code></dt> -<dd> -<div class="paragraph"> -<div class="title">Description</div> -<p>A staging directory in default file system (HDFS) for keeping temporary data.</p> -</div> -<div class="paragraph"> -<div class="title">Default</div> -<p><code>/user/${user.name}/hbase-staging</code></p> -</div> -</dd> -</dl> -</div> <div id="hbase.cluster.distributed" class="dlist"> <dl> <dt class="hdlist1"><code>hbase.cluster.distributed</code></dt> @@ -2310,7 +2296,7 @@ Some configurations would only appear in source code; the only way to identify t <dd> <div class="paragraph"> <div class="title">Description</div> -<p>How long a Procedure WAL stays will remain in the {hbase.rootdir}/oldWALs/masterProcedureWALs directory, after which it will be cleaned by a Master thread. The value is in milliseconds.</p> +<p>How long a Procedure WAL will remain in the {hbase.rootdir}/MasterProcedureWALs directory, after which it will be cleaned by a Master thread. The value is in milliseconds.</p> </div> <div class="paragraph"> <div class="title">Default</div> @@ -2340,7 +2326,7 @@ Some configurations would only appear in source code; the only way to identify t <dd> <div class="paragraph"> <div class="title">Description</div> -<p>Whether or not the Master listens to the Master web UI port (hbase.master.info.port) and redirects requests to the web UI server shared by the Master and RegionServer.</p> +<p>Whether or not the Master listens to the Master web UI port (hbase.master.info.port) and redirects requests to the web UI server shared by the Master and RegionServer. Config. makes sense when Master is serving Regions (not the default).</p> </div> <div class="paragraph"> <div class="title">Default</div> @@ -2349,6 +2335,21 @@ Some configurations would only appear in source code; the only way to identify t </dd> </dl> </div> +<div id="hbase.master.fileSplitTimeout" class="dlist"> +<dl> +<dt class="hdlist1"><code>hbase.master.fileSplitTimeout</code></dt> +<dd> +<div class="paragraph"> +<div class="title">Description</div> +<p>Splitting a region, how long to wait on the file-splitting step before aborting the attempt. Default: 600000. This setting used to be known as hbase.regionserver.fileSplitTimeout in hbase-1.x. Split is now run master-side hence the rename (If a 'hbase.master.fileSplitTimeout' setting found, will use it to prime the current 'hbase.master.fileSplitTimeout' Configuration.</p> +</div> +<div class="paragraph"> +<div class="title">Default</div> +<p><code>600000</code></p> +</div> +</dd> +</dl> +</div> <div id="hbase.regionserver.port" class="dlist"> <dl> <dt class="hdlist1"><code>hbase.regionserver.port</code></dt> @@ -2415,7 +2416,7 @@ Some configurations would only appear in source code; the only way to identify t <dd> <div class="paragraph"> <div class="title">Description</div> -<p>Count of RPC Listener instances spun up on RegionServers. Same property is used by the Master for count of master handlers.</p> +<p>Count of RPC Listener instances spun up on RegionServers. Same property is used by the Master for count of master handlers. Too many handlers can be counter-productive. Make it a multiple of CPU count. If mostly read-only, handlers count close to cpu count does well. Start with twice the CPU count and tune from there.</p> </div> <div class="paragraph"> <div class="title">Default</div> @@ -4010,6 +4011,36 @@ Some configurations would only appear in source code; the only way to identify t </dd> </dl> </div> +<div id="hbase.coprocessor.master.classes" class="dlist"> +<dl> +<dt class="hdlist1"><code>hbase.coprocessor.master.classes</code></dt> +<dd> +<div class="paragraph"> +<div class="title">Description</div> +<p>A comma-separated list of org.apache.hadoop.hbase.coprocessor.MasterObserver coprocessors that are loaded by default on the active HMaster process. For any implemented coprocessor methods, the listed classes will be called in order. After implementing your own MasterObserver, just put it in HBase’s classpath and add the fully qualified class name here.</p> +</div> +<div class="paragraph"> +<div class="title">Default</div> +<p>none</p> +</div> +</dd> +</dl> +</div> +<div id="hbase.coprocessor.abortonerror" class="dlist"> +<dl> +<dt class="hdlist1"><code>hbase.coprocessor.abortonerror</code></dt> +<dd> +<div class="paragraph"> +<div class="title">Description</div> +<p>Set to true to cause the hosting server (master or regionserver) to abort if a coprocessor fails to load, fails to initialize, or throws an unexpected Throwable object. Setting this to false will allow the server to continue execution but the system wide state of the coprocessor in question will become inconsistent as it will be properly executing in only a subset of servers, so this is most useful for debugging only.</p> +</div> +<div class="paragraph"> +<div class="title">Default</div> +<p><code>true</code></p> +</div> +</dd> +</dl> +</div> <div id="hbase.rest.port" class="dlist"> <dl> <dt class="hdlist1"><code>hbase.rest.port</code></dt> @@ -4100,36 +4131,6 @@ Some configurations would only appear in source code; the only way to identify t </dd> </dl> </div> -<div id="hbase.coprocessor.master.classes" class="dlist"> -<dl> -<dt class="hdlist1"><code>hbase.coprocessor.master.classes</code></dt> -<dd> -<div class="paragraph"> -<div class="title">Description</div> -<p>A comma-separated list of org.apache.hadoop.hbase.coprocessor.MasterObserver coprocessors that are loaded by default on the active HMaster process. For any implemented coprocessor methods, the listed classes will be called in order. After implementing your own MasterObserver, just put it in HBase’s classpath and add the fully qualified class name here.</p> -</div> -<div class="paragraph"> -<div class="title">Default</div> -<p>none</p> -</div> -</dd> -</dl> -</div> -<div id="hbase.coprocessor.abortonerror" class="dlist"> -<dl> -<dt class="hdlist1"><code>hbase.coprocessor.abortonerror</code></dt> -<dd> -<div class="paragraph"> -<div class="title">Description</div> -<p>Set to true to cause the hosting server (master or regionserver) to abort if a coprocessor fails to load, fails to initialize, or throws an unexpected Throwable object. Setting this to false will allow the server to continue execution but the system wide state of the coprocessor in question will become inconsistent as it will be properly executing in only a subset of servers, so this is most useful for debugging only.</p> -</div> -<div class="paragraph"> -<div class="title">Default</div> -<p><code>true</code></p> -</div> -</dd> -</dl> -</div> <div id="hbase.table.lock.enable" class="dlist"> <dl> <dt class="hdlist1"><code>hbase.table.lock.enable</code></dt> @@ -4670,6 +4671,21 @@ Some configurations would only appear in source code; the only way to identify t </dd> </dl> </div> +<div id="hbase.master.loadbalance.bytable" class="dlist"> +<dl> +<dt class="hdlist1"><code>hbase.master.loadbalance.bytable</code></dt> +<dd> +<div class="paragraph"> +<div class="title">Description</div> +<p>Factor Table name when the balancer runs. Default: false.</p> +</div> +<div class="paragraph"> +<div class="title">Default</div> +<p><code>false</code></p> +</div> +</dd> +</dl> +</div> <div id="hbase.master.normalizer.class" class="dlist"> <dl> <dt class="hdlist1"><code>hbase.master.normalizer.class</code></dt> @@ -4886,7 +4902,7 @@ Some configurations would only appear in source code; the only way to identify t <dd> <div class="paragraph"> <div class="title">Description</div> -<p>By default, in replication we can not make sure the order of operations in slave cluster is same as the order in master. If set REPLICATION_SCOPE to 2, we will push edits by the order of written. This configure is to set how long (in ms) we will wait before next checking if a log can not push right now because there are some logs written before it have not been pushed. A larger waiting will decrease the number of queries on hbase:meta but will enlarge the delay of replication. This feature relies on zk-less assignment, so users must set hbase.assignment.usezk to false to support it.</p> +<p>By default, in replication we can not make sure the order of operations in slave cluster is same as the order in master. If set REPLICATION_SCOPE to 2, we will push edits by the order of written. This configuration is to set how long (in ms) we will wait before next checking if a log can NOT be pushed because there are some logs written before it that have yet to be pushed. A larger waiting will decrease the number of queries on hbase:meta but will enlarge the delay of replication. This feature relies on zk-less assignment, so users must set hbase.assignment.usezk to false to support it.</p> </div> <div class="paragraph"> <div class="title">Default</div> @@ -5081,7 +5097,7 @@ Some configurations would only appear in source code; the only way to identify t <dd> <div class="paragraph"> <div class="title">Description</div> -<p>Timeout for master for the snapshot procedure execution</p> +<p>Timeout for master for the snapshot procedure execution.</p> </div> <div class="paragraph"> <div class="title">Default</div> @@ -5096,7 +5112,7 @@ Some configurations would only appear in source code; the only way to identify t <dd> <div class="paragraph"> <div class="title">Description</div> -<p>Timeout for regionservers to keep threads in snapshot request pool waiting</p> +<p>Timeout for regionservers to keep threads in snapshot request pool waiting.</p> </div> <div class="paragraph"> <div class="title">Default</div> @@ -29447,6 +29463,165 @@ before you start moving out the dead. Move in good live nodes first if you have </div> </div> </div> +<div class="sect1"> +<h2 id="normalizer"><a class="anchor" href="#normalizer"></a>158. Region Normalizer</h2> +<div class="sectionbody"> +<div class="paragraph"> +<p>The Region Normalizer tries to make Regions all in a table about the same in size. +It does this by finding a rough average. Any region that is larger than twice this +size is split. Any region that is much smaller is merged into an adjacent region. +It is good to run the Normalizer on occasion on a down time after the cluster has +been running a while or say after a burst of activity such as a large delete.</p> +</div> +<div class="paragraph"> +<p>(The bulk of the below detail was copied wholesale from the blog by Romil Choksi at +<a href="https://community.hortonworks.com/articles/54987/hbase-region-normalizer.html";>HBase Region Normalizer</a>)</p> +</div> +<div class="paragraph"> +<p>The Region Normalizer is feature available since HBase-1.2. It runs a set of +pre-calculated merge/split actions to resize regions that are either too +large or too small compared to the average region size for a given table. Region +Normalizer when invoked computes a normalization 'plan' for all of the tables in +HBase. System tables (such as hbase:meta, hbase:namespace, Phoenix system tables +etc) and user tables with normalization disabled are ignored while computing the +plan. For normalization enabled tables, normalization plan is carried out in +parallel across multiple tables.</p> +</div> +<div class="paragraph"> +<p>Normalizer can be enabled or disabled globally for the entire cluster using the +ânormalizer_switchâ command in the HBase shell. Normalization can also be +controlled on a per table basis, which is disabled by default when a table is +created. Normalization for a table can be enabled or disabled by setting the +NORMALIZATION_ENABLED table attribute to true or false.</p> +</div> +<div class="paragraph"> +<p>To check normalizer status and enable/disable normalizer</p> +</div> +<div class="listingblock"> +<div class="content"> +<pre class="CodeRay highlight"><code data-lang="bash">hbase(main):001:0> normalizer_enabled +true +0 row(s) in 0.4870 seconds + +hbase(main):002:0> normalizer_switch false +true +0 row(s) in 0.0640 seconds + +hbase(main):003:0> normalizer_enabled +false +0 row(s) in 0.0120 seconds + +hbase(main):004:0> normalizer_switch true +false +0 row(s) in 0.0200 seconds + +hbase(main):005:0> normalizer_enabled +true +0 row(s) in 0.0090 seconds</code></pre> +</div> +</div> +<div class="paragraph"> +<p>When enabled, Normalizer is invoked in the background every 5 mins (by default), +which can be configured using <code>hbase.normalization.period</code> in <code>hbase-site.xml</code>. +Normalizer can also be invoked manually/programmatically at will using HBase shellâs +<code>normalize</code> command. HBase by default uses <code>SimpleRegionNormalizer</code>, but users can +design their own normalizer as long as they implement the RegionNormalizer Interface. +Details about the logic used by <code>SimpleRegionNormalizer</code> to compute its normalization +plan can be found <a href="https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.html";>here</a>.</p> +</div> +<div class="paragraph"> +<p>The below example shows a normalization plan being computed for an user table, and +merge action being taken as a result of the normalization plan computed by SimpleRegionNormalizer.</p> +</div> +<div class="paragraph"> +<p>Consider an user table with some pre-split regions having 3 equally large regions +(about 100K rows) and 1 relatively small region (about 25K rows). Following is the +snippet from an hbase meta table scan showing each of the pre-split regions for +the user table.</p> +</div> +<div class="listingblock"> +<div class="content"> +<pre>table_p8ddpd6q5z,,1469494305548.68b9892220865cb6048 column=info:regioninfo, timestamp=1469494306375, value={ENCODED => 68b9892220865cb604809c950d1adf48, NAME => 'table_p8ddpd6q5z,,1469494305548.68b989222 09c950d1adf48. 0865cb604809c950d1adf48.', STARTKEY => '', ENDKEY => '1'} +.... +table_p8ddpd6q5z,1,1469494317178.867b77333bdc75a028 column=info:regioninfo, timestamp=1469494317848, value={ENCODED => 867b77333bdc75a028bb4c5e4b235f48, NAME => 'table_p8ddpd6q5z,1,1469494317178.867b7733 bb4c5e4b235f48. 3bdc75a028bb4c5e4b235f48.', STARTKEY => '1', ENDKEY => '3'} +.... +table_p8ddpd6q5z,3,1469494328323.98f019a753425e7977 column=info:regioninfo, timestamp=1469494328486, value={ENCODED => 98f019a753425e7977ab8636e32deeeb, NAME => 'table_p8ddpd6q5z,3,1469494328323.98f019a7 ab8636e32deeeb. 53425e7977ab8636e32deeeb.', STARTKEY => '3', ENDKEY => '7'} +.... +table_p8ddpd6q5z,7,1469494339662.94c64e748979ecbb16 column=info:regioninfo, timestamp=1469494339859, value={ENCODED => 94c64e748979ecbb166f6cc6550e25c6, NAME => 'table_p8ddpd6q5z,7,1469494339662.94c64e74 6f6cc6550e25c6. 8979ecbb166f6cc6550e25c6.', STARTKEY => '7', ENDKEY => '8'} +.... +table_p8ddpd6q5z,8,1469494339662.6d2b3f5fd1595ab8e7 column=info:regioninfo, timestamp=1469494339859, value={ENCODED => 6d2b3f5fd1595ab8e7c031876057b1ee, NAME => 'table_p8ddpd6q5z,8,1469494339662.6d2b3f5f c031876057b1ee. d1595ab8e7c031876057b1ee.', STARTKEY => '8', ENDKEY => ''}</pre> +</div> +</div> +<div class="paragraph"> +<p>Invoking the normalizer using ânormalizeâ int the HBase shell, the below log snippet +from HMaster log shows the normalization plan computed as per the logic defined for +SimpleRegionNormalizer. Since the total region size (in MB) for the adjacent smallest +regions in the table is less than the average region size, the normalizer computes a +plan to merge these two regions.</p> +</div> +<div class="listingblock"> +<div class="content"> +<pre>2016-07-26 07:08:26,928 DEBUG [B.fifo.QRpcServer.handler=20,queue=2,port=20000] master.HMaster: Skipping normalization for table: hbase:namespace, as it's either system table or doesn't have auto +normalization turned on +2016-07-26 07:08:26,928 DEBUG [B.fifo.QRpcServer.handler=20,queue=2,port=20000] master.HMaster: Skipping normalization for table: hbase:backup, as it's either system table or doesn't have auto normalization turned on +2016-07-26 07:08:26,928 DEBUG [B.fifo.QRpcServer.handler=20,queue=2,port=20000] master.HMaster: Skipping normalization for table: hbase:meta, as it's either system table or doesn't have auto normalization turned on +2016-07-26 07:08:26,928 DEBUG [B.fifo.QRpcServer.handler=20,queue=2,port=20000] master.HMaster: Skipping normalization for table: table_h2osxu3wat, as it's either system table or doesn't have autonormalization turned on +2016-07-26 07:08:26,928 DEBUG [B.fifo.QRpcServer.handler=20,queue=2,port=20000] normalizer.SimpleRegionNormalizer: Computing normalization plan for table: table_p8ddpd6q5z, number of regions: 5 +2016-07-26 07:08:26,929 DEBUG [B.fifo.QRpcServer.handler=20,queue=2,port=20000] normalizer.SimpleRegionNormalizer: Table table_p8ddpd6q5z, total aggregated regions size: 12 +2016-07-26 07:08:26,929 DEBUG [B.fifo.QRpcServer.handler=20,queue=2,port=20000] normalizer.SimpleRegionNormalizer: Table table_p8ddpd6q5z, average region size: 2.4 +2016-07-26 07:08:26,929 INFO [B.fifo.QRpcServer.handler=20,queue=2,port=20000] normalizer.SimpleRegionNormalizer: Table table_p8ddpd6q5z, small region size: 0 plus its neighbor size: 0, less thanthe avg size 2.4, merging them +2016-07-26 07:08:26,971 INFO [B.fifo.QRpcServer.handler=20,queue=2,port=20000] normalizer.MergeNormalizationPlan: Executing merging normalization plan: MergeNormalizationPlan{firstRegion={ENCODED=> d51df2c58e9b525206b1325fd925a971, NAME => 'table_p8ddpd6q5z,,1469514755237.d51df2c58e9b525206b1325fd925a971.', STARTKEY => '', ENDKEY => '1'}, secondRegion={ENCODED => e69c6b25c7b9562d078d9ad3994f5330, NAME => 'table_p8ddpd6q5z,1,1469514767669.e69c6b25c7b9562d078d9ad3994f5330.', +STARTKEY => '1', ENDKEY => '3'}}</pre> +</div> +</div> +<div class="paragraph"> +<p>Region normalizer as per itâs computed plan, merged the region with start key as ââ +and end key as â1â, with another region having start key as â1â and end key as â3â. +Now, that these regions have been merged we see a single new region with start key +as ââ and end key as â3â</p> +</div> +<div class="listingblock"> +<div class="content"> +<pre>table_p8ddpd6q5z,,1469516907210.e06c9b83c4a252b130e column=info:mergeA, timestamp=1469516907431, +value=PBUF\x08\xA5\xD9\x9E\xAF\xE2*\x12\x1B\x0A\x07default\x12\x10table_p8ddpd6q5z\x1A\x00"\x011(\x000\x00 ea74d246741ba. 8\x00 +table_p8ddpd6q5z,,1469516907210.e06c9b83c4a252b130e column=info:mergeB, timestamp=1469516907431, +value=PBUF\x08\xB5\xBA\x9F\xAF\xE2*\x12\x1B\x0A\x07default\x12\x10table_p8ddpd6q5z\x1A\x011"\x013(\x000\x0 ea74d246741ba. 08\x00 +table_p8ddpd6q5z,,1469516907210.e06c9b83c4a252b130e column=info:regioninfo, timestamp=1469516907431, value={ENCODED => e06c9b83c4a252b130eea74d246741ba, NAME => 'table_p8ddpd6q5z,,1469516907210.e06c9b83c ea74d246741ba. 4a252b130eea74d246741ba.', STARTKEY => '', ENDKEY => '3'} +.... +table_p8ddpd6q5z,3,1469514778736.bf024670a847c0adff column=info:regioninfo, timestamp=1469514779417, value={ENCODED => bf024670a847c0adffb74b2e13408b32, NAME => 'table_p8ddpd6q5z,3,1469514778736.bf024670 b74b2e13408b32. a847c0adffb74b2e13408b32.' STARTKEY => '3', ENDKEY => '7'} +.... +table_p8ddpd6q5z,7,1469514790152.7c5a67bc755e649db2 column=info:regioninfo, timestamp=1469514790312, value={ENCODED => 7c5a67bc755e649db22f49af6270f1e1, NAME => 'table_p8ddpd6q5z,7,1469514790152.7c5a67bc 2f49af6270f1e1. 755e649db22f49af6270f1e1.', STARTKEY => '7', ENDKEY => '8'} +.... +table_p8ddpd6q5z,8,1469514790152.58e7503cda69f98f47 column=info:regioninfo, timestamp=1469514790312, value={ENCODED => 58e7503cda69f98f4755178e74288c3a, NAME => 'table_p8ddpd6q5z,8,1469514790152.58e7503c 55178e74288c3a. da69f98f4755178e74288c3a.', STARTKEY => '8', ENDKEY => ''}</pre> +</div> +</div> +<div class="paragraph"> +<p>A similar example can be seen for an user table with 3 smaller regions and 1 +relatively large region. For this example, we have an user table with 1 large region containing 100K rows, and 3 relatively smaller regions with about 33K rows each. As seen from the normalization plan, since the larger region is more than twice the average region size it ends being split into two regions â one with start key as â1â and end key as â154717â and the other region with start key as '154717' and end key as â3â</p> +</div> +<div class="listingblock"> +<div class="content"> +<pre>2016-07-26 07:39:45,636 DEBUG [B.fifo.QRpcServer.handler=7,queue=1,port=20000] master.HMaster: Skipping normalization for table: hbase:backup, as it's either system table or doesn't have auto normalization turned on +2016-07-26 07:39:45,636 DEBUG [B.fifo.QRpcServer.handler=7,queue=1,port=20000] normalizer.SimpleRegionNormalizer: Computing normalization plan for table: table_p8ddpd6q5z, number of regions: 4 +2016-07-26 07:39:45,636 DEBUG [B.fifo.QRpcServer.handler=7,queue=1,port=20000] normalizer.SimpleRegionNormalizer: Table table_p8ddpd6q5z, total aggregated regions size: 12 +2016-07-26 07:39:45,636 DEBUG [B.fifo.QRpcServer.handler=7,queue=1,port=20000] normalizer.SimpleRegionNormalizer: Table table_p8ddpd6q5z, average region size: 3.0 +2016-07-26 07:39:45,636 DEBUG [B.fifo.QRpcServer.handler=7,queue=1,port=20000] normalizer.SimpleRegionNormalizer: No normalization needed, regions look good for table: table_p8ddpd6q5z +2016-07-26 07:39:45,636 DEBUG [B.fifo.QRpcServer.handler=7,queue=1,port=20000] normalizer.SimpleRegionNormalizer: Computing normalization plan for table: table_h2osxu3wat, number of regions: 5 +2016-07-26 07:39:45,636 DEBUG [B.fifo.QRpcServer.handler=7,queue=1,port=20000] normalizer.SimpleRegionNormalizer: Table table_h2osxu3wat, total aggregated regions size: 7 +2016-07-26 07:39:45,636 DEBUG [B.fifo.QRpcServer.handler=7,queue=1,port=20000] normalizer.SimpleRegionNormalizer: Table table_h2osxu3wat, average region size: 1.4 +2016-07-26 07:39:45,636 INFO [B.fifo.QRpcServer.handler=7,queue=1,port=20000] normalizer.SimpleRegionNormalizer: Table table_h2osxu3wat, large region table_h2osxu3wat,1,1469515926544.27f2fdbb2b6612ea163eb6b40753c3db. has size 4, more than twice avg size, splitting +2016-07-26 07:39:45,640 INFO [B.fifo.QRpcServer.handler=7,queue=1,port=20000] normalizer.SplitNormalizationPlan: Executing splitting normalization plan: SplitNormalizationPlan{regionInfo={ENCODED => 27f2fdbb2b6612ea163eb6b40753c3db, NAME => 'table_h2osxu3wat,1,1469515926544.27f2fdbb2b6612ea163eb6b40753c3db.', STARTKEY => '1', ENDKEY => '3'}, splitPoint=null} +2016-07-26 07:39:45,656 DEBUG [B.fifo.QRpcServer.handler=7,queue=1,port=20000] master.HMaster: Skipping normalization for table: hbase:namespace, as it's either system table or doesn't have auto normalization turned on +2016-07-26 07:39:45,656 DEBUG [B.fifo.QRpcServer.handler=7,queue=1,port=20000] master.HMaster: Skipping normalization for table: hbase:meta, as it's either system table or doesn't +have auto normalization turned on â¦..â¦..â¦. +2016-07-26 07:39:46,246 DEBUG [AM.ZK.Worker-pool2-t278] master.RegionStates: Onlined 54de97dae764b864504704c1c8d3674a on hbase-test-rc-5.openstacklocal,16020,1469419333913 {ENCODED => 54de97dae764b864504704c1c8d3674a, NAME => 'table_h2osxu3wat,1,1469518785661.54de97dae764b864504704c1c8d3674a.', STARTKEY => '1', ENDKEY => '154717'} +2016-07-26 07:39:46,246 INFO [AM.ZK.Worker-pool2-t278] master.RegionStates: Transition {d6b5625df331cfec84dce4f1122c567f state=SPLITTING_NEW, ts=1469518786246, server=hbase-test-rc-5.openstacklocal,16020,1469419333913} to {d6b5625df331cfec84dce4f1122c567f state=OPEN, ts=1469518786246, +server=hbase-test-rc-5.openstacklocal,16020,1469419333913} +2016-07-26 07:39:46,246 DEBUG [AM.ZK.Worker-pool2-t278] master.RegionStates: Onlined d6b5625df331cfec84dce4f1122c567f on hbase-test-rc-5.openstacklocal,16020,1469419333913 {ENCODED => d6b5625df331cfec84dce4f1122c567f, NAME => 'table_h2osxu3wat,154717,1469518785661.d6b5625df331cfec84dce4f1122c567f.', STARTKEY => '154717', ENDKEY => '3'}</pre> +</div> +</div> +</div> +</div> <h1 id="developer" class="sect0"><a class="anchor" href="#developer"></a>Building and Developing Apache HBase</h1> <div class="openblock partintro"> <div class="content"> @@ -29455,7 +29630,7 @@ Being familiar with these guidelines will help the HBase committers to use your </div> </div> <div class="sect1"> -<h2 id="getting.involved"><a class="anchor" href="#getting.involved"></a>158. Getting Involved</h2> +<h2 id="getting.involved"><a class="anchor" href="#getting.involved"></a>159. Getting Involved</h2> <div class="sectionbody"> <div class="paragraph"> <p>Apache HBase gets better only when people contribute! If you are looking to contribute to Apache HBase, look for <a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20HBASE%20AND%20labels%20in%20(beginner)%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)">issues in JIRA tagged with the label 'beginner'</a>. @@ -29470,7 +29645,7 @@ See <a href="http://search-hadoop.com/m/DHED43re96";>What label <p>As Apache HBase is an Apache Software Foundation project, see <a href="#asf">asf</a> for more information about how the ASF functions.</p> </div> <div class="sect2"> -<h3 id="mailing.list"><a class="anchor" href="#mailing.list"></a>158.1. Mailing Lists</h3> +<h3 id="mailing.list"><a class="anchor" href="#mailing.list"></a>159.1. Mailing Lists</h3> <div class="paragraph"> <p>Sign up for the dev-list and the user-list. See the <a href="https://hbase.apache.org/mail-lists.html";>mailing lists</a> page. @@ -29478,14 +29653,14 @@ Posing questions - and helping to answer other people’s questions - is enc </div> </div> <div class="sect2"> -<h3 id="slack"><a class="anchor" href="#slack"></a>158.2. Slack</h3> +<h3 id="slack"><a class="anchor" href="#slack"></a>159.2. Slack</h3> <div class="paragraph"> <p>The Apache HBase project has its own link: <a href="http://apache-hbase.slack.com";>Slack Channel</a> for real-time questions and discussion. Mail <a href="mailto:[email protected]";>[email protected]</a> to request an invite.</p> </div> </div> <div class="sect2"> -<h3 id="irc"><a class="anchor" href="#irc"></a>158.3. Internet Relay Chat (IRC)</h3> +<h3 id="irc"><a class="anchor" href="#irc"></a>159.3. Internet Relay Chat (IRC)</h3> <div class="paragraph"> <p>(NOTE: Our IRC channel seems to have been deprecated in favor of the above Slack channel)</p> </div> @@ -29495,7 +29670,7 @@ FreeNode offers a web-based client, but most people prefer a native client, and </div> </div> <div class="sect2"> -<h3 id="_jira"><a class="anchor" href="#_jira"></a>158.4. Jira</h3> +<h3 id="_jira"><a class="anchor" href="#_jira"></a>159.4. Jira</h3> <div class="paragraph"> <p>Check for existing issues in <a href="https://issues.apache.org/jira/projects/HBASE/issues";>Jira</a>. If it’s either a new feature request, enhancement, or a bug, file a ticket.</p> @@ -29532,7 +29707,7 @@ background to flesh out yourself.</p> <p>Bugs and tests have the highest priority and should be actionable.</p> </div> <div class="sect3"> -<h4 id="_guidelines_for_reporting_effective_issues"><a class="anchor" href="#_guidelines_for_reporting_effective_issues"></a>158.4.1. Guidelines for reporting effective issues</h4> +<h4 id="_guidelines_for_reporting_effective_issues"><a class="anchor" href="#_guidelines_for_reporting_effective_issues"></a>159.4.1. Guidelines for reporting effective issues</h4> <div class="ulist"> <ul> <li> @@ -29631,7 +29806,7 @@ a similar issue in the future.</p> </div> </div> <div class="sect3"> -<h4 id="_working_on_an_issue"><a class="anchor" href="#_working_on_an_issue"></a>158.4.2. Working on an issue</h4> +<h4 id="_working_on_an_issue"><a class="anchor" href="#_working_on_an_issue"></a>159.4.2. Working on an issue</h4> <div class="paragraph"> <p>To check for existing issues which you can tackle as a beginner, search for <a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20HBASE%20AND%20labels%20in%20(beginner)%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)">issues in JIRA tagged with the label 'beginner'</a>.</p> </div> @@ -29675,7 +29850,7 @@ code snippet </div> </div> <div class="sect1"> -<h2 id="repos"><a class="anchor" href="#repos"></a>159. Apache HBase Repositories</h2> +<h2 id="repos"><a class="anchor" href="#repos"></a>160. Apache HBase Repositories</h2> <div class="sectionbody"> <div class="paragraph"> <p>There are two different repositories for Apache HBase: Subversion (SVN) and Git. @@ -29689,12 +29864,12 @@ See <a href="https://hbase.apache.org/source-repository.html";>Source Code </div> </div> <div class="sect1"> -<h2 id="_ides"><a class="anchor" href="#_ides"></a>160. IDEs</h2> +<h2 id="_ides"><a class="anchor" href="#_ides"></a>161. IDEs</h2> <div class="sectionbody"> <div class="sect2"> -<h3 id="eclipse"><a class="anchor" href="#eclipse"></a>160.1. Eclipse</h3> +<h3 id="eclipse"><a class="anchor" href="#eclipse"></a>161.1. Eclipse</h3> <div class="sect3"> -<h4 id="eclipse.code.formatting"><a class="anchor" href="#eclipse.code.formatting"></a>160.1.1. Code Formatting</h4> +<h4 id="eclipse.code.formatting"><a class="anchor" href="#eclipse.code.formatting"></a>161.1.1. Code Formatting</h4> <div class="paragraph"> <p>Under the <em>dev-support/</em> folder, you will find <em>hbase_eclipse_formatter.xml</em>. We encourage you to have this formatter in place in eclipse when editing HBase code.</p> @@ -29710,13 +29885,13 @@ edited lines' is selected.</p> </div> </div> <div class="sect3"> -<h4 id="eclipse.git.plugin"><a class="anchor" href="#eclipse.git.plugin"></a>160.1.2. Eclipse Git Plugin</h4> +<h4 id="eclipse.git.plugin"><a class="anchor" href="#eclipse.git.plugin"></a>161.1.2. Eclipse Git Plugin</h4> <div class="paragraph"> <p>If you cloned the project via git, download and install the Git plugin (EGit). Attach to your local git repo (via the <span class="label">Git Repositories</span> window) and you’ll be able to see file revision history, generate patches, etc.</p> </div> </div> <div class="sect3"> -<h4 id="eclipse.maven.setup"><a class="anchor" href="#eclipse.maven.setup"></a>160.1.3. HBase Project Setup in Eclipse using <code>m2eclipse</code></h4> +<h4 id="eclipse.maven.setup"><a class="anchor" href="#eclipse.maven.setup"></a>161.1.3. HBase Project Setup in Eclipse using <code>m2eclipse</code></h4> <div class="paragraph"> <p>The easiest way is to use the m2eclipse plugin for Eclipse. Eclipse Indigo or newer includes m2eclipse, or you can download it from <a href="http://www.eclipse.org/m2e/"; class="bare">http://www.eclipse.org/m2e/</a>. It provides Maven integration for Eclipse, and even lets you use the direct Maven commands from within Eclipse to compile and test your project.</p> @@ -29753,7 +29928,7 @@ doesn't exist</pre> </div> </div> <div class="sect3"> -<h4 id="eclipse.commandline"><a class="anchor" href="#eclipse.commandline"></a>160.1.4. HBase Project Setup in Eclipse Using the Command Line</h4> +<h4 id="eclipse.commandline"><a class="anchor" href="#eclipse.commandline"></a>161.1.4. HBase Project Setup in Eclipse Using the Command Line</h4> <div class="paragraph"> <p>Instead of using <code>m2eclipse</code>, you can generate the Eclipse files from the command line.</p> </div> @@ -29783,7 +29958,7 @@ You only need to do this once.</p> </div> </div> <div class="sect3"> -<h4 id="eclipse.maven.class"><a class="anchor" href="#eclipse.maven.class"></a>160.1.5. Maven Classpath Variable</h4> +<h4 id="eclipse.maven.class"><a class="anchor" href="#eclipse.maven.class"></a>161.1.5. Maven Classpath Variable</h4> <div class="paragraph"> <p>The <code>$M2_REPO</code> classpath variable needs to be set up for the project. This needs to be set to your local Maven repository, which is usually <em>~/.m2/repository</em></p> @@ -29802,7 +29977,7 @@ Unbound classpath variable: 'M2_REPO/com/google/protobuf/protobuf-java/2.3.0/pro </div> </div> <div class="sect3"> -<h4 id="eclipse.issues"><a class="anchor" href="#eclipse.issues"></a>160.1.6. Eclipse Known Issues</h4> +<h4 id="eclipse.issues"><a class="anchor" href="#eclipse.issues"></a>161.1.6. Eclipse Known Issues</h4> <div class="paragraph"> <p>Eclipse will currently complain about <em>Bytes.java</em>. It is not possible to turn these errors off.</p> @@ -29817,14 +29992,14 @@ Access restriction: The method getLong(Object, long) from the type Unsafe is not </div> </div> <div class="sect3"> -<h4 id="eclipse.more"><a class="anchor" href="#eclipse.more"></a>160.1.7. Eclipse - More Information</h4> +<h4 id="eclipse.more"><a class="anchor" href="#eclipse.more"></a>161.1.7. Eclipse - More Information</h4> <div class="paragraph"> <p>For additional information on setting up Eclipse for HBase development on Windows, see <a href="http://michaelmorello.blogspot.com/2011/09/hbase-subversion-eclipse-windows.html";>Michael Morello’s blog</a> on the topic.</p> </div> </div> </div> <div class="sect2"> -<h3 id="_intellij_idea"><a class="anchor" href="#_intellij_idea"></a>160.2. IntelliJ IDEA</h3> +<h3 id="_intellij_idea"><a class="anchor" href="#_intellij_idea"></a>161.2. IntelliJ IDEA</h3> <div class="paragraph"> <p>You can set up IntelliJ IDEA for similar functionality as Eclipse. Follow these steps.</p> @@ -29850,7 +30025,7 @@ required</span> is selected, and click <b class="button">Next</b>.</p> </div> </div> <div class="sect2"> -<h3 id="_other_ides"><a class="anchor" href="#_other_ides"></a>160.3. Other IDEs</h3> +<h3 id="_other_ides"><a class="anchor" href="#_other_ides"></a>161.3. Other IDEs</h3> <div class="paragraph"> <p>It would be useful to mirror the <a href="#eclipse">eclipse</a> set-up instructions for other IDEs. If you would like to assist, please have a look at <a href="https://issues.apache.org/jira/browse/HBASE-11704";>HBASE-11704</a>.</p> @@ -29859,10 +30034,10 @@ If you would like to assist, please have a look at <a href="https://issues.apach </div> </div> <div class="sect1"> -<h2 id="build"><a class="anchor" href="#build"></a>161. Building Apache HBase</h2> +<h2 id="build"><a class="anchor" href="#build"></a>162. Building Apache HBase</h2> <div class="sectionbody"> <div class="sect2"> -<h3 id="build.basic"><a class="anchor" href="#build.basic"></a>161.1. Basic Compile</h3> +<h3 id="build.basic"><a class="anchor" href="#build.basic"></a>162.1. Basic Compile</h3> <div class="paragraph"> <p>HBase is compiled using Maven. You must use at least Maven 3.0.4. @@ -29885,7 +30060,7 @@ See <a href="#java">java</a> for more complete information about supported JDK v </table> </div> <div class="sect3"> -<h4 id="maven.build.commands"><a class="anchor" href="#maven.build.commands"></a>161.1.1. Maven Build Commands</h4> +<h4 id="maven.build.commands"><a class="anchor" href="#maven.build.commands"></a>162.1.1. Maven Build Commands</h4> <div class="paragraph"> <p>All commands are executed from the local HBase project directory.</p> </div> @@ -29951,13 +30126,13 @@ To create the full installable HBase package takes a little bit more work, so re </div> </div> <div class="sect3"> -<h4 id="maven.build.commands.unitall"><a class="anchor" href="#maven.build.commands.unitall"></a>161.1.2. Running all or individual Unit Tests</h4> +<h4 id="maven.build.commands.unitall"><a class="anchor" href="#maven.build.commands.unitall"></a>162.1.2. Running all or individual Unit Tests</h4> <div class="paragraph"> <p>See the <a href="#hbase.unittests.cmds">hbase.unittests.cmds</a> section in <a href="#hbase.unittests">hbase.unittests</a></p> </div> </div> <div class="sect3"> -<h4 id="maven.build.hadoop"><a class="anchor" href="#maven.build.hadoop"></a>161.1.3. Building against various hadoop versions.</h4> +<h4 id="maven.build.hadoop"><a class="anchor" href="#maven.build.hadoop"></a>162.1.3. Building against various hadoop versions.</h4> <div class="paragraph"> <p>HBase supports building against Apache Hadoop versions: 2.y and 3.y (early release artifacts). By default we build against Hadoop 2.x.</p> </div> @@ -29986,7 +30161,7 @@ Tests may not all pass so you may need to pass <code>-DskipTests</code> unless y </div> </div> <div class="sect3"> -<h4 id="build.protobuf"><a class="anchor" href="#build.protobuf"></a>161.1.4. Build Protobuf</h4> +<h4 id="build.protobuf"><a class="anchor" href="#build.protobuf"></a>162.1.4. Build Protobuf</h4> <div class="paragraph"> <p>You may need to change the protobuf definitions that reside in the <em>hbase-protocol</em> module or other modules.</p> </div> @@ -30018,7 +30193,7 @@ You can use maven profile <code>compile-protobuf</code> to do this.</p> </div> </div> <div class="sect3"> -<h4 id="build.thrift"><a class="anchor" href="#build.thrift"></a>161.1.5. Build Thrift</h4> +<h4 id="build.thrift"><a class="anchor" href="#build.thrift"></a>162.1.5. Build Thrift</h4> <div class="paragraph"> <p>You may need to change the thrift definitions that reside in the <em>hbase-thrift</em> module or other modules.</p> </div> @@ -30042,7 +30217,7 @@ You can use maven profile <code>compile-thrift</code> to do this.</p> </div> </div> <div class="sect3"> -<h4 id="_build_a_tarball"><a class="anchor" href="#_build_a_tarball"></a>161.1.6. Build a Tarball</h4> +<h4 id="_build_a_tarball"><a class="anchor" href="#_build_a_tarball"></a>162.1.6. Build a Tarball</h4> <div class="paragraph"> <p>You can build a tarball without going through the release process described in <a href="#releasing">releasing</a>, by running the following command:</p> </div> @@ -30069,7 +30244,7 @@ You can use maven profile <code>compile-thrift</code> to do this.</p> </div> </div> <div class="sect3"> -<h4 id="build.gotchas"><a class="anchor" href="#build.gotchas"></a>161.1.7. Build Gotchas</h4> +<h4 id="build.gotchas"><a class="anchor" href="#build.gotchas"></a>162.1.7. Build Gotchas</h4> <div class="paragraph"> <p>If you see <code>Unable to find resource 'VM_global_library.vm'</code>, ignore it. It’s not an error. @@ -30080,7 +30255,7 @@ It is <a href="https://issues.apache.org/jira/browse/MSITE-286";>officially ugly< </div> </div> <div class="sect1"> -<h2 id="releasing"><a class="anchor" href="#releasing"></a>162. Releasing Apache HBase</h2> +<h2 id="releasing"><a class="anchor" href="#releasing"></a>163. Releasing Apache HBase</h2> <div class="sectionbody"> <div class="admonitionblock note"> <table> @@ -30147,7 +30322,7 @@ For the build to sign them for you, you a properly configured <em>settings.xml</ </div> </div> <div class="sect2"> -<h3 id="maven.release"><a class="anchor" href="#maven.release"></a>162.1. Making a Release Candidate</h3> +<h3 id="maven.release"><a class="anchor" href="#maven.release"></a>163.1. Making a Release Candidate</h3> <div class="paragraph"> <p>Only committers may make releases of hbase artifacts.</p> </div> @@ -30520,7 +30695,7 @@ Ensure it actually gets published by checking <a href="https://dist.apache.org/r </div> </div> <div class="sect2"> -<h3 id="maven.snapshot"><a class="anchor" href="#maven.snapshot"></a>162.2. Publishing a SNAPSHOT to maven</h3> +<h3 id="maven.snapshot"><a class="anchor" href="#maven.snapshot"></a>163.2. Publishing a SNAPSHOT to maven</h3> <div class="paragraph"> <p>Make sure your <em>settings.xml</em> is set up properly (see <a href="#maven.settings.xml">Example <em>~/.m2/settings.xml</em> File</a>). Make sure the hbase version includes <code>-SNAPSHOT</code> as a suffix. @@ -30541,7 +30716,7 @@ It will put a snapshot up into the apache snapshot repository for you.</p> </div> </div> <div class="sect1"> -<h2 id="hbase.rc.voting"><a class="anchor" href="#hbase.rc.voting"></a>163. Voting on Release Candidates</h2> +<h2 id="hbase.rc.voting"><a class="anchor" href="#hbase.rc.voting"></a>164. Voting on Release Candidates</h2> <div class="sectionbody"> <div class="paragraph"> <p>Everyone is encouraged to try and vote on HBase release candidates. @@ -30559,7 +30734,7 @@ for how we arrived at this process.</p> </div> </div> <div class="sect1"> -<h2 id="documentation"><a class="anchor" href="#documentation"></a>164. Generating the HBase Reference Guide</h2> +<h2 id="documentation"><a class="anchor" href="#documentation"></a>165. Generating the HBase Reference Guide</h2> <div class="sectionbody"> <div class="paragraph"> <p>The manual is marked up using Asciidoc. @@ -30570,16 +30745,16 @@ See <a href="#appendix_contributing_to_documentation">appendix contributing to d </div> </div> <div class="sect1"> -<h2 id="hbase.org"><a class="anchor" href="#hbase.org"></a>165. Updating <a href="https://hbase.apache.org";>hbase.apache.org</a></h2> +<h2 id="hbase.org"><a class="anchor" href="#hbase.org"></a>166. Updating <a href="https://hbase.apache.org";>hbase.apache.org</a></h2> <div class="sectionbody"> <div class="sect2"> -<h3 id="hbase.org.site.contributing"><a class="anchor" href="#hbase.org.site.contributing"></a>165.1. Contributing to hbase.apache.org</h3> +<h3 id="hbase.org.site.contributing"><a class="anchor" href="#hbase.org.site.contributing"></a>166.1. Contributing to hbase.apache.org</h3> <div class="paragraph"> <p>See <a href="#appendix_contributing_to_documentation">appendix contributing to documentation</a> for more information on contributing to the documentation or website.</p> </div> </div> <div class="sect2"> -<h3 id="hbase.org.site.publishing"><a class="anchor" href="#hbase.org.site.publishing"></a>165.2. Publishing <a href="https://hbase.apache.org";>hbase.apache.org</a></h3> +<h3 id="hbase.org.site.publishing"><a class="anchor" href="#hbase.org.site.publishing"></a>166.2. Publishing <a href="https://hbase.apache.org";>hbase.apache.org</a></h3> <div class="paragraph"> <p>See <a href="#website_publish">Publishing the HBase Website and Documentation</a> for instructions on publishing the website and documentation.</p> </div> @@ -30587,7 +30762,7 @@ See <a href="#appendix_contributing_to_documentation">appendix contributing to d </div> </div> <div class="sect1"> -<h2 id="hbase.tests"><a class="anchor" href="#hbase.tests"></a>166. Tests</h2> +<h2 id="hbase.tests"><a class="anchor" href="#hbase.tests"></a>167. Tests</h2> <div class="sectionbody"> <div class="paragraph"> <p>Developers, at a minimum, should familiarize themselves with the unit test detail; unit tests in HBase have a character not usually seen in other projects.</p> @@ -30597,7 +30772,7 @@ See <a href="#appendix_contributing_to_documentation">appendix contributing to d For developing unit tests for your HBase applications, see <a href="#unit.tests">unit.tests</a>.</p> </div> <div class="sect2"> -<h3 id="hbase.moduletests"><a class="anchor" href="#hbase.moduletests"></a>166.1. Apache HBase Modules</h3> +<h3 id="hbase.moduletests"><a class="anchor" href="#hbase.moduletests"></a>167.1. Apache HBase Modules</h3> <div class="paragraph"> <p>As of 0.96, Apache HBase is split into multiple modules. This creates "interesting" rules for how and where tests are written. @@ -30606,7 +30781,7 @@ These tests can spin up a minicluster and will need to be categorized. For any other module, for example <code>hbase-common</code>, the tests must be strict unit tests and just test the class under test - no use of the HBaseTestingUtility or minicluster is allowed (or even possible given the dependency tree).</p> </div> <div class="sect3"> -<h4 id="hbase.moduletest.shell"><a class="anchor" href="#hbase.moduletest.shell"></a>166.1.1. Testing the HBase Shell</h4> +<h4 id="hbase.moduletest.shell"><a class="anchor" href="#hbase.moduletest.shell"></a>167.1.1. Testing the HBase Shell</h4> <div class="paragraph"> <p>The HBase shell and its tests are predominantly written in jruby. In order to make these tests run as a part of the standard build, there is a single JUnit test, <code>TestShell</code>, that takes care of loading the jruby implemented tests and running them. @@ -30647,7 +30822,7 @@ You can run all of the HBase admin related tests, including both the normal admi </div> </div> <div class="sect3"> -<h4 id="hbase.moduletest.run"><a class="anchor" href="#hbase.moduletest.run"></a>166.1.2. Running Tests in other Modules</h4> +<h4 id="hbase.moduletest.run"><a class="anchor" href="#hbase.moduletest.run"></a>167.1.2. Running Tests in other Modules</h4> <div class="paragraph"> <p>If the module you are developing in has no other dependencies on other HBase modules, then you can cd into that module and just run:</p> </div> @@ -30683,7 +30858,7 @@ For example, to skip the tests in <code>hbase-server</code> and <code>hbase-comm </div> </div> <div class="sect2"> -<h3 id="hbase.unittests"><a class="anchor" href="#hbase.unittests"></a>166.2. Unit Tests</h3> +<h3 id="hbase.unittests"><a class="anchor" href="#hbase.unittests"></a>167.2. Unit Tests</h3> <div class="paragraph"> <p>Apache HBase test cases are subdivided into four categories: small, medium, large, and integration with corresponding JUnit <a href="https://github.com/junit-team/junit4/wiki/Categories";>categories</a>: <code>SmallTests</code>, <code>MediumTests</code>, <code>LargeTests</code>, <code>IntegrationTests</code>. @@ -30752,9 +30927,9 @@ See <a href="#integration.tests">integration.tests</a> for more info.</p> </div> </div> <div class="sect2"> -<h3 id="hbase.unittests.cmds"><a class="anchor" href="#hbase.unittests.cmds"></a>166.3. Running tests</h3> +<h3 id="hbase.unittests.cmds"><a class="anchor" href="#hbase.unittests.cmds"></a>167.3. Running tests</h3> <div class="sect3"> -<h4 id="hbase.unittests.cmds.test"><a class="anchor" href="#hbase.unittests.cmds.test"></a>166.3.1. Default: small and medium category tests</h4> +<h4 id="hbase.unittests.cmds.test"><a class="anchor" href="#hbase.unittests.cmds.test"></a>167.3.1. Default: small and medium category tests</h4> <div class="paragraph"> <p>Running <code>mvn test</code> will execute all small tests in a single JVM (no fork) and then medium tests in a separate JVM for each test instance. Medium tests are NOT executed if there is an error in a small test. @@ -30763,7 +30938,7 @@ There is one report for small tests, and one report for medium tests if they are </div> </div> <div class="sect3"> -<h4 id="hbase.unittests.cmds.test.runalltests"><a class="anchor" href="#hbase.unittests.cmds.test.runalltests"></a>166.3.2. Running all tests</h4> +<h4 id="hbase.unittests.cmds.test.runalltests"><a class="anchor" href="#hbase.unittests.cmds.test.runalltests"></a>167.3.2. Running all tests</h4> <div class="paragraph"> <p>Running <code>mvn test -P runAllTests</code> will execute small tests in a single JVM then medium and large tests in a separate JVM for each test. Medium and large tests are NOT executed if there is an error in a small test. @@ -30772,7 +30947,7 @@ There is one report for small tests, and one report for medium and large tests i </div> </div> <div class="sect3"> -<h4 id="hbase.unittests.cmds.test.localtests.mytest"><a class="anchor" href="#hbase.unittests.cmds.test.localtests.mytest"></a>166.3.3. Running a single test or all tests in a package</h4> +<h4 id="hbase.unittests.cmds.test.localtests.mytest"><a class="anchor" href="#hbase.unittests.cmds.test.localtests.mytest"></a>167.3.3. Running a single test or all tests in a package</h4> <div class="paragraph"> <p>To run an individual test, e.g. <code>MyTest</code>, rum <code>mvn test -Dtest=MyTest</code> You can also pass multiple, individual tests as a comma-delimited list:</p> </div> @@ -30798,7 +30973,7 @@ However, you need to make sure the sum of <code>Tests run:</code> in the <code>R </div> </div> <div class="sect3"> -<h4 id="hbase.unittests.cmds.test.profiles"><a class="anchor" href="#hbase.unittests.cmds.test.profiles"></a>166.3.4. Other test invocation permutations</h4> +<h4 id="hbase.unittests.cmds.test.profiles"><a class="anchor" href="#hbase.unittests.cmds.test.profiles"></a>167.3.4. Other test invocation permutations</h4> <div class="paragraph"> <p>Running <code>mvn test -P runSmallTests</code> will execute "small" tests only, using a single JVM.</p> </div> @@ -30813,7 +30988,7 @@ However, you need to make sure the sum of <code>Tests run:</code> in the <code>R </div> </div> <div class="sect3"> -<h4 id="hbase.unittests.test.faster"><a class="anchor" href="#hbase.unittests.test.faster"></a>166.3.5. Running tests faster</h4> +<h4 id="hbase.unittests.test.faster"><a class="anchor" href="#hbase.unittests.test.faster"></a>167.3.5. Running tests faster</h4> <div class="paragraph"> <p>By default, <code>$ mvn test -P runAllTests</code> runs 5 tests in parallel. It can be increased on a developer’s machine. @@ -30855,7 +31030,7 @@ sudo mount -t tmpfs -o size=2048M tmpfs /ram2G</pre> </div> </div> <div class="sect3"> -<h4 id="hbase.unittests.cmds.test.hbasetests"><a class="anchor" href="#hbase.unittests.cmds.test.hbasetests"></a>166.3.6. hbasetests.sh</h4> +<h4 id="hbase.unittests.cmds.test.hbasetests"><a class="anchor" href="#hbase.unittests.cmds.test.hbasetests"></a>167.3.6. hbasetests.sh</h4> <div class="paragraph"> <p>It’s also possible to use the script hbasetests.sh. This script runs the medium and large tests in parallel with two maven instances, and provides a single report. @@ -30870,7 +31045,7 @@ Running ./dev-support/hbasetests.sh replayFailed will rerun the failed tests a s </div> </div> <div class="sect3"> -<h4 id="hbase.unittests.resource.checker"><a class="anchor" href="#hbase.unittests.resource.checker"></a>166.3.7. Test Resource Checker</h4> +<h4 id="hbase.unittests.resource.checker"><a class="anchor" href="#hbase.unittests.resource.checker"></a>167.3.7. Test Resource Checker</h4> <div class="paragraph"> <p>A custom Maven SureFire plugin listener checks a number of resources before and after each HBase unit test runs and logs its findings at the end of the test output files which can be found in <em>target/surefire-reports</em> per Maven module (Tests write test reports named for the test class into this directory. Check the <em>*-out.txt</em> files). The resources counted are the number of threads, the number of file descriptors, etc. @@ -30891,9 +31066,9 @@ ConnectionCount=1 (was 1)</pre> </div> </div> <div class="sect2"> -<h3 id="hbase.tests.writing"><a class="anchor" href="#hbase.tests.writing"></a>166.4. Writing Tests</h3> +<h3 id="hbase.tests.writing"><a class="anchor" href="#hbase.tests.writing"></a>167.4. Writing Tests</h3> <div class="sect3"> -<h4 id="hbase.tests.rules"><a class="anchor" href="#hbase.tests.rules"></a>166.4.1. General rules</h4> +<h4 id="hbase.tests.rules"><a class="anchor" href="#hbase.tests.rules"></a>167.4.1. General rules</h4> <div class="ulist"> <ul> <li> @@ -30914,7 +31089,7 @@ This class offers helper functions to create a temp directory and do the cleanup </div> </div> <div class="sect3"> -<h4 id="hbase.tests.categories"><a class="anchor" href="#hbase.tests.categories"></a>166.4.2. Categories and execution time</h4> +<h4 id="hbase.tests.categories"><a class="anchor" href="#hbase.tests.categories"></a>167.4.2. Categories and execution time</h4> <div class="ulist"> <ul> <li> @@ -30931,7 +31106,7 @@ This should ensure a good parallelization for people using it, and ease the anal </div> </div> <div class="sect3"> -<h4 id="hbase.tests.sleeps"><a class="anchor" href="#hbase.tests.sleeps"></a>166.4.3. Sleeps in tests</h4> +<h4 id="hbase.tests.sleeps"><a class="anchor" href="#hbase.tests.sleeps"></a>167.4.3. Sleeps in tests</h4> <div class="paragraph"> <p>Whenever possible, tests should not use <span class="method">Thread.sleep</span>, but rather waiting for the real event they need. This is faster and clearer for the reader. @@ -30944,7 +31119,7 @@ Waiting for a socket operation should be done in a 200 ms sleep loop.</p> </div> </div> <div class="sect3"> -<h4 id="hbase.tests.cluster"><a class="anchor" href="#hbase.tests.cluster"></a>166.4.4. Tests using a cluster</h4> +<h4 id="hbase.tests.cluster"><a class="anchor" href="#hbase.tests.cluster"></a>167.4.4. Tests using a cluster</h4> <div class="paragraph"> <p>Tests using a HRegion do not have to start a cluster: A region can use the local file system. Start/stopping a cluster cost around 10 seconds. @@ -30956,7 +31131,7 @@ This will allow to share the cluster later.</p> </div> </div> <div class="sect3"> -<h4 id="hbase.tests.example.code"><a class="anchor" href="#hbase.tests.example.code"></a>166.4.5. Tests Skeleton Code</h4> +<h4 id="hbase.tests.example.code"><a class="anchor" href="#hbase.tests.example.code"></a>167.4.5. Tests Skeleton Code</h4> <div class="paragraph"> <p>Here is a test skeleton code with Categorization and a Category-based timeout rule to copy and paste and use as basis for test contribution.</p> </div> @@ -31002,7 +31177,7 @@ This will allow to share the cluster later.</p> </div> </div> <div class="sect2"> -<h3 id="integration.tests"><a class="anchor" href="#integration.tests"></a>166.5. Integration Tests</h3> +<h3 id="integration.tests"><a class="anchor" href="#integration.tests"></a>167.5. Integration Tests</h3> <div class="paragraph"> <p>HBase integration/system tests are tests that are beyond HBase unit tests. They are generally long-lasting, sizeable (the test can be asked to 1M rows or 1B rows), targetable (they can take configuration that will point them at the ready-made cluster they are to run against; integration tests do not include cluster start/stop code), and verifying success, integration tests rely on public APIs only; they do not attempt to examine server internals asserting success/fail. @@ -31056,7 +31231,7 @@ For example,</p> </div> </div> <div class="sect3"> -<h4 id="maven.build.commands.integration.tests.mini"><a class="anchor" href="#maven.build.commands.integration.tests.mini"></a>166.5.1. Running integration tests against mini cluster</h4> +<h4 id="maven.build.commands.integration.tests.mini"><a class="anchor" href="#maven.build.commands.integration.tests.mini"></a>167.5.1. Running integration tests against mini cluster</h4> <div class="paragraph"> <p>HBase 0.92 added a <code>verify</code> maven target. Invoking it, for example by doing <code>mvn verify</code>, will run all the phases up to and including the verify phase via the maven <a href="https://maven.apache.org/plugins/maven-failsafe-plugin/";>failsafe @@ -31101,7 +31276,7 @@ To just run <code>IntegrationTestClassXYZ.java</code>, use: mvn </div> </div> <div class="sect3"> -<h4 id="maven.build.commands.integration.tests.distributed"><a class="anchor" href="#maven.build.commands.integration.tests.distributed"></a>166.5.2. Running integration tests against distributed cluster</h4> +<h4 id="maven.build.commands.integration.tests.distributed"><a class="anchor" href="#maven.build.commands.integration.tests.distributed"></a>167.5.2. Running integration tests against distributed cluster</h4> <div class="paragraph"> <p>If you have an already-setup HBase cluster, you can launch the integration tests by invoking the class <code>IntegrationTestsDriver</code>. You may have to run test-compile first. @@ -31138,7 +31313,7 @@ For other deployment options, a ClusterManager can be implemented and plugged in </div> </div> <div class="sect3"> -<h4 id="maven.build.commands.integration.tests.destructive"><a class="anchor" href="#maven.build.commands.integration.tests.destructive"></a>166.5.3. Destructive integration / system tests (ChaosMonkey)</h4> +<h4 id="maven.build.commands.integration.tests.destructive"><a class="anchor" href="#maven.build.commands.integration.tests.destructive"></a>167.5.3. Destructive integration / system tests (ChaosMonkey)</h4> <div class="paragraph"> <p>HBase 0.96 introduced a tool named <code>ChaosMonkey</code>, modeled after <a href="https://netflix.github.io/chaosmonkey/";>same-named tool by Netflix’s Chaos Monkey tool</a>. @@ -31243,14 +31418,14 @@ policy, which is configured with all the available actions. It chose to run <cod </div> </div> <div class="sect3"> -<h4 id="_available_policies"><a class="anchor" href="#_available_policies"></a>166.5.4. Available Policies</h4> +<h4 id="_available_policies"><a class="anchor" href="#_available_policies"></a>167.5.4. Available Policies</h4> <div class="paragraph"> <p>HBase ships with several ChaosMonkey policies, available in the <code>hbase/hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/policies/</code> directory.</p> </div> </div> <div class="sect3"> -<h4 id="chaos.monkey.properties"><a class="anchor" href="#chaos.monkey.properties"></a>166.5.5. Configuring Individual ChaosMonkey Actions</h4> +<h4 id="chaos.monkey.properties"><a class="anchor" href="#chaos.monkey.properties"></a>167.5.5. Configuring Individual ChaosMonkey Actions</h4> <div class="paragraph"> <p>Since HBase version 1.0.0 (<a href="https://issues.apache.org/jira/browse/HBASE-11348";>HBASE-11348</a>), ChaosMonkey integration tests can be configured per test run. @@ -31318,10 +31493,10 @@ properties file, which may be <code>hbase-site.xml</code> or a different propert </div> </div> <div class="sect1"> -<h2 id="developing"><a class="anchor" href="#developing"></a>167. Developer Guidelines</h2> +<h2 id="developing"><a class="anchor" href="#developing"></a>168. Developer Guidelines</h2> <div class="sectionbody"> <div class="sect2"> -<h3 id="_branches"><a class="anchor" href="#_branches"></a>167.1. Branches</h3> +<h3 id="_branches"><a class="anchor" href="#_branches"></a>168.1. Branches</h3> <div class="paragraph"> <p>We use Git for source code management and latest development happens on <code>master</code> branch. There are branches for past major/minor/maintenance releases and important features and bug fixes are often @@ -31329,7 +31504,7 @@ branches for past major/minor/maintenance releases and important features and bu </div> </div> <div class="sect2"> -<h3 id="_release_managers"><a class="anchor" href="#_release_managers"></a>167.2. Release Managers</h3> +<h3 id="_release_managers"><a class="anchor" href="#_release_managers"></a>168.2. Release Managers</h3> <div class="paragraph"> <p>Each maintained release branch has a release manager, who volunteers to coordinate new features and bug fixes are backported to that release. The release managers are <a href="https://hbase.apache.org/team-list.html";>committers</a>. @@ -31385,9 +31560,9 @@ End-of-life releases are not included in this list. </table> </div> <div class="sect2"> -<h3 id="code.standards"><a class="anchor" href="#code.standards"></a>167.3. Code Standards</h3> +<h3 id="code.standards"><a class="anchor" href="#code.standards"></a>168.3. Code Standards</h3> <div class="sect3"> -<h4 id="_interface_classifications"><a class="anchor" href="#_interface_classifications"></a>167.3.1. Interface Classifications</h4> +<h4 id="_interface_classifications"><a class="anchor" href="#_interface_classifications"></a>168.3.1. Interface Classifications</h4> <div class="paragraph"> <p>Interfaces are classified both by audience and by stability level. These labels appear at the head of a class. @@ -31465,7 +31640,7 @@ Mark your new packages if publicly accessible.</p> </div> </div> <div class="sect3"> -<h4 id="common.patch.feedback"><a class="anchor" href="#common.patch.feedback"></a>167.3.2. Code Formatting Conventions</h4> +<h4 id="common.patch.feedback"><a class="anchor" href="#common.patch.feedback"></a>168.3.2. Code Formatting Conventions</h4> <div class="paragraph"> <p>Please adhere to the following guidelines so that your patches can be reviewed more quickly. These guidelines have been developed based upon common feedback on patches from new contributors.</p> @@ -31653,7 +31828,7 @@ If you are creating a new class that needs to implement this interface, do not f </div> </div> <div class="sect3"> -<h4 id="_garbage_collection_conserving_guidelines"><a class="anchor" href="#_garbage_collection_conserving_guidelines"></a>167.3.3. Garbage-Collection Conserving Guidelines</h4> +<h4 id="_garbage_collection_conserving_guidelines"><a class="anchor" href="#_garbage_collection_conserving_guidelines"></a>168.3.3. Garbage-Collection Conserving Guidelines</h4> <div class="paragraph"> <p>The following guidelines were borrowed from <a href="http://engineering.linkedin.com/performance/linkedin-feed-faster-less-jvm-garbage"; class="bare">http://engineering.linkedin.com/performance/linkedin-feed-faster-less-jvm-garbage</a>. Keep them in mind to keep preventable garbage collection to a minimum. Have a look @@ -31685,13 +31860,13 @@ these guidelines.</p> </div> </div> <div class="sect2"> -<h3 id="design.invariants"><a class="anchor" href="#design.invariants"></a>167.4. Invariants</h3> +<h3 id="design.invariants"><a class="anchor" href="#design.invariants"></a>168.4. Invariants</h3> <div class="paragraph"> <p>We don’t have many but what we have we list below. All are subject to challenge of course but until then, please hold to the rules of the road.</p> </div> <div class="sect3"> -<h4 id="design.invariants.zk.data"><a class="anchor" href="#design.invariants.zk.data"></a>167.4.1. No permanent state in ZooKeeper</h4> +<h4 id="design.invariants.zk.data"><a class="anchor" href="#design.invariants.zk.data"></a>168.4.1. No permanent state in ZooKeeper</h4> <div class="paragraph"> <p>ZooKeeper state should transient (treat it like memory). If ZooKeeper state is deleted, hbase should be able to recover and essentially be in the same state.</p> </div> @@ -31723,7 +31898,7 @@ Follow progress on this issue at <a href="https://issues.apache.org/jira/browse/ </div> </div> <div class="sect2"> -<h3 id="run.insitu"><a class="anchor" href="#run.insitu"></a>167.5. Running In-Situ</h3> +<h3 id="run.insitu"><a class="anchor" href="#run.insitu"></a>168.5. Running In-Situ</h3> <div class="paragraph"> <p>If you are developing Apache HBase, frequently it is useful to test your changes against a more-real cluster than what you find in unit tests. In this case, HBase can be run directly from the source in local-mode. @@ -31752,7 +31927,7 @@ Generally, the above command is just a good thing to try running first, if maven </div> </div> <div class="sect2"> -<h3 id="add.metrics"><a class="anchor" href="#add.metrics"></a>167.6. Adding Metrics</h3> +<h3 id="add.metrics"><a class="anchor" href="#add.metrics"></a>168.6. Adding Metrics</h3> <div class="paragraph"> <p>After adding a new feature a developer might want to add metrics. HBase exposes metrics using the Hadoop Metrics 2 system, so adding a new metric involves exposing that metric to the hadoop system. @@ -31762,14 +31937,14 @@ To get an in-depth look at the reasoning and structure of these classes you can To add a metric to an existing MBean follow the short guide below:</p> </div> <div class="sect3"> -<h4 id="_add_metric_name_and_function_to_hadoop_compat_interface"><a class="anchor" href="#_add_metric_name_and_function_to_hadoop_compat_interface"></a>167.6.1. Add Metric name and Function to Hadoop Compat Interface.</h4> +<h4 id="_add_metric_name_and_function_to_hadoop_compat_interface"><a class="anchor" href="#_add_metric_name_and_function_to_hadoop_compat_interface"></a>168.6.1. Add Metric name and Function to Hadoop Compat Interface.</h4> <div class="paragraph"> <p>Inside of the source interface the corresponds to where the metrics are generated (eg MetricsMasterSource for things coming from HMaster) create new static strings for metric name and description. Then add a new method that will be called to add new reading.</p> </div> </div> <div class="sect3"> -<h4 id="_add_the_implementation_to_both_hadoop_1_and_hadoop_2_compat_modules"><a class="anchor" href="#_add_the_implementation_to_both_hadoop_1_and_hadoop_2_compat_modules"></a>167.6.2. Add the Implementation to Both Hadoop 1 and Hadoop 2 Compat modules.</h4> +<h4 id="_add_the_implementation_to_both_hadoop_1_and_hadoop_2_compat_modules"><a class="anchor" href="#_add_the_implementation_to_both_hadoop_1_and_hadoop_2_compat_modules"></a>168.6.2. Add the Implementation to Both Hadoop 1 and Hadoop 2 Compat modules.</h4> <div class="paragraph"> <p>Inside of the implementation of the source (eg. MetricsMasterSourceImpl in the above example) create a new histogram, counter, gauge, or stat in the init method. @@ -31782,7 +31957,7 @@ For this the MetricsAssertHelper is provided.</p> </div> </div> <div class="sect2"> -<h3 id="git.best.practices"><a class="anchor" href="#git.best.practices"></a>167.7. Git Best Practices</h3> +<h3 id="git.best.practices"><a class="anchor" href="#git.best.practices"></a>168.7. Git Best Practices</h3> <div class="dlist"> <dl> <dt class="hdlist1">Avoid git merges.</dt> @@ -31799,7 +31974,7 @@ For this the MetricsAssertHelper is provided.</p> <p>Please contribute to this document if you think of other Git best practices.</p> </div> <div class="sect3"> -<h4 id="__code_rebase_all_git_branches_sh_code"><a class="anchor" href="#__code_rebase_all_git_branches_sh_code"></a>167.7.1. <code>rebase_all_git_branches.sh</code></h4> +<h4 id="__code_rebase_all_git_branches_sh_code"><a class="anchor" href="#__code_rebase_all_git_branches_sh_code"></a>168.7.1. <code>rebase_all_git_branches.sh</code></h4> <div class="paragraph"> <p>The <em>dev-support/rebase_all_git_branches.sh</em> script is provided to help keep your Git repository clean. Use the <code>-h</code> parameter to get usage instructions. @@ -31812,7 +31987,7 @@ The script checks the directory for sub-directory called <em>.git/</em>, before </div> </div> <div class="sect2"> -<h3 id="submitting.patches"><a class="anchor" href="#submitting.patches"></a>167.8. Submitting Patches</h3> +<h3 id="submitting.patches"><a class="anchor" href="#submitting.patches"></a>168.8. Submitting Patches</h3> <div class="paragraph"> <p>If you are new to submitting patches to open source or new to submitting patches to Apache, start by reading the <a href="https://commons.apache.org/patches.html";>On Contributing Patches</a> page from @@ -31822,7 +31997,7 @@ It provides a nice overview that applies equally to the Apache HBase Project. good read to understand development workflow.</p> </div> <div class="sect3"> -<h4 id="submitting.patches.create"><a class="anchor" href="#submitting.patches.create"></a>167.8.1. Create Patch</h4> +<h4 id="submitting.patches.create"><a class="anchor" href="#submitting.patches.create"></a>168.8.1. Create Patch</h4> <div class="paragraph"> <p>Make sure you review <a href="#common.patch.feedback">common.patch.feedback</a> for code style. If your patch @@ -31928,7 +32103,7 @@ Click on <b class="button">Cancel Patch</b> and then on <b class="button">Submit </div> </div> <div class="sect3"> -<h4 id="submitting.patches.tests"><a class="anchor" href="#submitting.patches.tests"></a>167.8.2. Unit Tests</h4> +<h4 id="submitting.patches.tests"><a class="anchor" href="#submitting.patches.tests"></a>168.8.2. Unit Tests</h4> <div class="paragraph"> <p>Always add and/or update relevant unit tests when making the changes. Make sure that new/changed unit tests pass locally before submitting the patch because it is faster @@ -31945,13 +32120,13 @@ See <a href="#hbase.tests">hbase.tests</a> for more information on tests.</p> </div> </div> <div class="sect3"> -<h4 id="_integration_tests"><a class="anchor" href="#_integration_tests"></a>167.8.3. Integration Tests</h4> +<h4 id="_integration_tests"><a class="anchor" href="#_integration_tests"></a>168.8.3. Integration Tests</h4> <div class="paragraph"> <p>Significant new features should provide an integration test in addition to unit tests, suitable for exercising the new feature at different points in its configuration space.</p> </div> </div> <div class="sect3"> -<h4 id="reviewboard"><a class="anchor" href="#reviewboard"></a>167.8.4. ReviewBoard</h4> +<h4 id="reviewboard"><a class="anchor" href="#reviewboard"></a>168.8.4. ReviewBoard</h4> <div class="paragraph"> <p>Patches larger than one screen, or patches that will be tricky to review, should go through <a href="https://reviews.apache.org";>ReviewBoard</a>.</p> </div> @@ -31994,7 +32169,7 @@ This attaches the ReviewBoard to the JIRA, for easy access.</p> </div> </div> <div class="sect3"> -<h4 id="_guide_for_hbase_committers"><a class="anchor" href="#_guide_for_hbase_committers"></a>167.8.5. Guide for HBase Committers</h4> +<h4 id="_guide_for_hbase_committers"><a class="anchor" href="#_guide_for_hbase_committers"></a>168.8.5. Guide for HBase Committers</h4> <div class="sect4"> <h5 id="_new_committers"><a class="anchor" href="#_new_committers"></a>New committers</h5> <div class="paragraph"> @@ -32274,21 +32449,21 @@ A committer should.</p> </div> </div> <div class="sect3"> -<h4 id="_dialog"><a class="anchor" href="#_dialog"></a>167.8.6. Dialog</h4> +<h4 id="_dialog"><a class="anchor" href="#_dialog"></a>168.8.6. Dialog</h4> <div class="paragraph"> <p>Committers should hang out in the #hbase room on irc.freenode.net for real-time discussions. However any substantive discussion (as with any off-list project-related discussion) should be re-iterated in Jira or on the developer list.</p> </div> </div> <div class="sect3"> -<h4 id="_do_not_edit_jira_comments"><a class="anchor" href="#_do_not_edit_jira_comments"></a>167.8.7. Do not edit JIRA comments</h4> +<h4 id="_do_not_edit_jira_comments"><a class="anchor" href="#_do_not_edit_jira_comments"></a>168.8.7. Do not edit JIRA comments</h4> <div class="paragraph"> <p>Misspellings and/or bad grammar is preferable to the disruption a JIRA comment edit causes: See the discussion at <a href="http://search-hadoop.com/?q=%5BReopened%5D+%28HBASE-451%29+Remove+HTableDescriptor+from+HRegionInfo&fc_project=HBase";>Re:(HBASE-451) Remove HTableDescriptor from HRegionInfo</a></p> </div> </div> </div> <div class="sect2"> -<h3 id="thirdparty"><a class="anchor" href="#thirdparty"></a>167.9. The hbase-thirdparty dependency and shading/relocation</h3> +<h3 id="thirdparty"><a class="anchor" href="#thirdparty"></a>168.9. The hbase-thirdparty dependency and shading/relocation</h3> <div class="paragraph"> <p>A new project was created for the release of hbase-2.0.0. It was called <code>hbase-thirdparty</code>. This project exists only to provide the main hbase @@ -32369,7 +32544,7 @@ in the hbase-thirdparty, use the hbase JIRA and mailing lists to post notice.</p </div> </div> <div class="sect2"> -<h3 id="hbase.archetypes.development"><a class="anchor" href="#hbase.archetypes.development"></a>167.10. Development of HBase-related Maven archetypes</h3> +<h3 id="hbase.archetypes.development"><a class="anchor" href="#hbase.archetypes.development"></a>168.10. Development of HBase-related Maven archetypes</h3> <div class="paragraph"> <p>The development of HBase-related Maven archetypes was begun with <a href="https://issues.apache.org/jira/browse/HBASE-14876";>HBASE-14876</a>. @@ -32389,7 +32564,7 @@ For information on unit tests for HBase itself, see <a href="#hbase.tests">hbase </div> </div> <div class="sect1"> -<h2 id="_junit"><a class="anchor" href="#_junit"></a>168. JUnit</h2> +<h2 id="_junit"><a class="anchor" href="#_junit"></a>169. JUnit</h2> <div class="sectionbody"> <div class="paragraph"> <p>HBase uses <a href="http://junit.org";>JUnit</a> for unit tests</p> @@ -32461,7 +32636,7 @@ For an introduction to JUnit, see <a href="https://github.com/junit-team/junit/w </div> </div> <div class="sect1"> -<h2 id="mockito"><a class="anchor" href="#mockito"></a>169. Mockito</h2> +<h2 id="mockito"><a class="anchor" href="#mockito"></a>170. Mockito</h2> <div class="sectionbody"> <div class="paragraph"> <p>Mockito is a mocking framework. @@ -32539,7 +32714,7 @@ Similarly, you can now expand into other operations such as Get, Scan, or Delete </div> </div> <div class="sect1"> -<h2 id="_mrunit"><a class="anchor" href="#_mrunit"></a>170. MRUnit</h2> +<h2 id="_mrunit"><a class="anchor" href="#_mrunit"></a>171. MRUnit</h2> <div class="sectionbody"> <div class="paragraph"> <p><a href="https://mrunit.apache.org/";>Apache MRUnit</a> is a library that allows you to unit-test MapReduce jobs. @@ -32636,7 +32811,7 @@ strValue2 = <span class="string"><span class="delimiter">"</span><span clas </div> </div> <div class="sect1"> -<h2 id="_integration_testing_with_an_hbase_mini_cluster"><a class="anchor" href="#_integration_testing_with_an_hbase_mini_cluster"></a>171. Integration Testing with an HBase Mini-Cluster</h2> +<h2 id="_integration_testing_with_an_hbase_mini_cluster"><a class="anchor" href="#_integration_testing_with_an_hbase_mini_cluster"></a>172. Integration Testing with an HBase Mini-Cluster</h2> <div class="sectionbody"> <div class="paragraph"> <p>HBase ships with HBaseTestingUtility, which makes it easy to write integration tests using a <em class="firstterm">mini-cluster</em>. @@ -32726,7 +32901,7 @@ Starting the mini-cluster takes about 20-30 seconds, but that should be appropri </div> <h1 id="protobuf" class="sect0"><a class="anchor" href="#protobuf"></a>Protobuf in HBase</h1> <div class="sect1"> -<h2 id="_protobuf"><a class="anchor" href="#_protobuf"></a>172. Protobuf</h2> +<h2 id="_protobuf"><a class="anchor" href="#_protobuf"></a>173. Protobuf</h2> <div class="sectionbody"> <div class="paragraph"> <p>HBase uses Google’s <a href="https://developers.google.com/protocol-buffers/";>protobufs</a> wherever @@ -32781,7 +32956,7 @@ practice needs to whither. We’ll explain why in the later <a href="#shaded.protobuf">hbase-2.0.0</a> section.</p> </div> <div class="sect2"> -<h3 id="shaded.protobuf"><a class="anchor" href="#shaded.protobuf"></a>172.1. hbase-2.0.0 and the shading of protobufs (HBASE-15638)</h3> +<h3 id="shaded.protobuf"><a class="anchor" href="#shaded.protobuf"></a>173.1. hbase-2.0.0 and the shading of protobufs (HBASE-15638)</h3> <div class="paragraph"> <p>As of hbase-2.0.0, our protobuf usage gets a little more involved. HBase core protobuf references are offset so as to refer to a private, @@ -32987,7 +33162,7 @@ zookeeper could start dropping sessions if it has to run through a directory of </div> </div> <div class="sect1"> -<h2 id="_using_existing_zookeeper_ensemble"><a class="anchor" href="#_using_existing_zookeeper_ensemble"></a>173. Using existing ZooKeeper ensemble</h2> +<h2 id="_using_existing_zookeeper_ensemble"><a class="anchor" href="#_using_existing_zookeeper_ensemble"></a>174. Using existing ZooKeeper ensemble</h2> <div class="sectionbody"> <div class="paragraph"> <p>To point HBase at an existing ZooKeeper cluster, one that is not managed by HBase, set <code>HBASE_MANAGES_ZK</code> in <em>conf/hbase-env.sh</em> to false</p> @@ -33024,7 +33199,7 @@ Additionally, see the <a href="https://wiki.apache.org/hadoop/ZooKeeper/FAQ#A7";> </div> </div> <div class="sect1"> -<h2 id="zk.sasl.auth"><a class="anchor" href="#zk.sasl.auth"></a>174. SASL Authentication with ZooKeeper</h2> +<h2 id="zk.sasl.auth"><a class="anchor" href="#zk.sasl.auth"></a>175. SASL Authentication with ZooKeeper</h2> <div class="sectionbody"> <div class="paragraph"> <p>Newer releases of Apache HBase (>= 0.92) will support connecting to a ZooKeeper Quorum that supports SASL authentication (which is available in ZooKeeper versions 3.4.0 or later).</p> @@ -33034,7 +33209,7 @@ Additionally, see the <a href="https://wiki.apache.org/hadoop/ZooKeeper/FAQ#A7";> ZooKeeper/HBase mutual authentication (<a href="https://issues.apache.org/jira/browse/HBASE-2418";>HBASE-2418</a>) is required as part of a complete secure HBase configuration (<a href="https://issues.apache.org/jira/browse/HBASE-3025";>HBASE-3025</a>). For simplicity of explication, this section ignores additional configuration required (Secure HDFS and Coprocessor configuration). It’s recommended to begin with an HBase-managed ZooKeeper configuration (as opposed to a standalone ZooKeeper quorum) for ease of learning.</p> </div> <div class="sect2"> -<h3 id="_operating_system_prerequisites"><a class="anchor" href="#_operating_system_prerequisites"></a>174.1. Operating System Prerequisites</h3> +<h3 id="_operating_system_prerequisites"><a class="anchor" href="#_operating_system_prerequisites"></a>175.1. Operating System Prerequisites</h3> <div class="paragraph"> <p>You need to have a working Kerberos KDC setup. For each <code>$HOST</code> that will run a ZooKeeper server, you should have a principle <code>zookeeper/$HOST</code>. @@ -33070,7 +33245,7 @@ The ZooKeeper client and server libraries manage their own ticket refreshment by </div> </div> <div class="sect2"> -<h3 id="_hbase_managed_zookeeper_configuration"><a class="anchor" href="#_hbase_managed_zookeeper_configuration"></a>174.2. HBase-managed ZooKeeper Configuration</h3> +<h3 id="_hbase_managed_zookeeper_configuration"><a class="anchor" href="#_hbase_managed_zookeeper_configuration"></a>175.2. HBase-managed ZooKeeper Configuration</h3> <div class="paragraph"> <p>On each node that will run a zookeeper, a master, or a regionserver, create a <a href="http://docs.oracle.com/javase/7/docs/technotes/guides/security/jgss/tutorials/LoginConfigFile.html";>JAAS</a> configuration file in the conf directory of the node’s <em>HBASE_HOME</em> directory that looks like the following:</p> </div> @@ -33162,7 +33337,7 @@ bin/hbase regionserver start</pre> </div> </div> <div class="sect2"> -<h3 id="_external_zookeeper_configuration"><a class="anchor" href="#_external_zookeeper_configuration"></a>174.3. External ZooKeeper Configuration</h3> +<h3 id="_external_zookeeper_configuration"><a class="anchor" href="#_external_zookeeper_configuration"></a>175.3. External ZooKeeper Configuration</h3> <div class="paragraph"> <p>Add a JAAS configuration file that looks like:</p> </div> @@ -33263,7 +33438,7 @@ bin/hbase regionserver start</pre> </div> </div> <div class="sect2"> -<h3 id="_zookeeper_server_authentication_log_output"><a class="anchor" href="#_zookeeper_server_authentication_log_output"></a>174.4. ZooKeeper Server Authentication Log Output</h3> +<h3 id="_zookeeper_server_authentication_log_output"><a class="anchor" href="#_zookeeper_server_authentication_log_output"></a>175.4. ZooKeeper Server Authentication Log Output</h3> <div class="paragraph"> <p>If the configuration above is successful, you should see something similar to the following in your ZooKeeper server logs:</p> </div> @@ -33285,7 +33460,7 @@ bin/hbase regionserver start</pre> </div> </div> <div class="sect2"> -<h3 id="_zookeeper_client_authentication_log_output"><a class="anchor" href="#_zookeeper_client_authentication_log_output"></a>174.5. ZooKeeper Client Authentication Log Output</h3> +<h3 id="_zookeeper_client_authentication_log_output"><a class="anchor" href="#_zookeeper_client_authentication_log_output"></a>175.5. ZooKeeper Client Authentication Log Output</h3> <div class="paragraph"> <p>On the ZooKeeper client side (HBase master or regionserver), you should see something similar to the following:</p> </div> @@ -33306,7 +33481,7 @@ bin/hbase regionserver start</pre> </div> </div> <div class="sect2"> -<h3 id="_configuration_from_scratch"><a class="anchor" href="#_configuration_from_scratch"></a>174.6. Configuration from Scratch</h3> +<h3 id="_configuration_from_scratch"><a class="anchor" href="#_configuration_from_scratch"></a>175.6. Configuration from Scratch</h3> <div class="paragraph"> <p>This has been tested on the current standard Amazon Linux AMI. First setup KDC and principals as described above. @@ -33332,9 +33507,9 @@ bin/hbase regionserver &</pre> </div> </div> <div class="sect2"> -<h3 id="_future_improvements"><a class="anchor" href="#_future_improvements"></a>174.7. Future improvements</h3> +<h3 id="_future_improvements"><a class="anchor" href="#_future_improvements"></a>175.7. Future improvements</h3> <div class="sect3"> -<h4 id="_fix_target_cached_classpath_txt"><a class="anchor" href="#_fix_target_cached_classpath_txt"></a>174.7.1. Fix target/cached_classpath.txt</h4> +<h4 id="_fix_target_cached_classpath_txt"><a class="anchor" href="#_fix_target_cached_classpath_txt"></a>175.7.1. Fix target/cached_classpath.txt</h4> <div class="paragraph"> <p>You must override the standard hadoop-core jar file from the <code>target/cached_classpath.txt</code> file with the version containing the HADOOP-7070 fix. You can use the following script to do this:</p> @@ -33347,13 +33522,13 @@ mv target/tmp.txt target/cached_classpath.txt</pre> </div> </div> <div class="sect3"> -<h4 id="_set_jaas_configuration_programmatically"><a class="anchor" href="#_set_jaas_configuration_programmatically"></a>174.7.2. Set JAAS configuration programmatically</h4> +<h4 id="_set_jaas_configuration_programmatically"><a class="anchor" href="#_set_jaas_configuration_programmatically"></a>175.7.2. Set JAAS configuration programmatically</h4> <div class="paragraph"> <p>This would avoid the need for a separate Hadoop jar that fixes <a href="https://issues.apache.org/jira/browse/HADOOP-7070";>HADOOP-7070</a>.</p> </div> </div> <div class="sect3"> -<h4 id="_elimination_of_code_kerberos_removehostfromprincipal_code_and_kerberos_removerealmfromprincipal"><a class="anchor" href="#_elimination_of_code_kerberos_removehostfromprincipal_code_and_kerberos_removerealmfromprincipal"></a>174.7.3. Elimination of <code>kerberos.removeHostFromPrincipal</code> and`kerberos.removeRealmFromPrincipal`</h4> +<h4 id="_elimination_of_code_kerberos_removehostfromprincipal_code_and_kerberos_removerealmfromprincipal"><a class="anchor" href="#_elimination_of_code_kerberos_removehostfromprincipal_code_and_kerberos_removerealmfromprincipal"></a>175.7.3. Elimination of <code>kerberos.removeHostFromPrincipal</code> and`kerberos.removeRealmFromPrincipal`</h4> </div> </div> @@ -33361,7 +33536,7 @@ mv target/tmp.txt target/cached_classpath.txt</pre> </div> <h1 id="community" class="sect0"><a class="anchor" href="#community"></a>Community</h1> <div class="sect1"> -<h2 id="_decisions"><a class="anchor" href="#_decisions"></a>175. Decisions</h2> +<h2 id="_decisions"><a class="anchor" href="#_decisions"></a>176. Decisions</h2> <div class="sectionbody"> <div class="paragraph"> <div class="title">Feature Branches</div> @@ -33430,7 +33605,7 @@ We also are currently in violation of this basic tenet — repli </div> </div> <div class="sect1"> -<h2 id="community.roles"><a class="anchor" href="#community.roles"></a>176. Community Roles</h2> +<h2 id="community.roles"><a class="anchor" href="#community.roles"></a>177. Community Roles</h2> <div class="sectionbody"> <div id="owner" class="paragraph"> <div class="title">Component Owner/Lieutenant</div> @@ -33457,7 +33632,7 @@ Owners do not need to be committers.</p> </div> </div> <div class="sect1"> -<h2 id="hbase.commit.msg.format"><a class="anchor" href="#hbase.commit.msg.format"></a>177. Commit Message format</h2> +<h2 id="hbase.commit.msg.format"><a class="anchor" href="#hbase.commit.msg.format"></a>178. Commit Message format</h2> <div class="sectionbody"> <div class="paragraph"> <p>We <a href="http://search-hadoop.com/m/Gwxwl10cFHa1";>agreed</a> to the following Git commit message format:</p> @@ -35718,7 +35893,7 @@ Options: </div> </div> <div class="sect1"> -<h2 id="data.block.encoding.enable"><a class="anchor" href="#data.block.encoding.enable"></a>178. Enable Data Block Encoding</h2> +<h2 id="data.block.encoding.enable"><a class="anchor" href="#data.block.encoding.enable"></a>179. Enable Data Block Encoding</h2> <div class="sectionbody"> <div class="paragraph"> <p>Codecs are built into HBase so no extra configuration is needed. @@ -36634,7 +36809,7 @@ The <code>LocalFileSpanReceiver</code> looks in <em>hbase-site.xml</em> for </div> </div> <div class="sect1"> -<h2 id="tracing.client.modifications"><a class="anchor" href="#tracing.client.modifications"></a>179. Client Modifications</h2> +<h2 id="tracing.client.modifications"><a class="anchor" href="#tracing.client.modifications"></a>180. Client Modifications</h2> <div class="sectionbody"> <div cl
<TRUNCATED>
