http://git-wip-us.apache.org/repos/asf/hbase-site/blob/ecb8d8ba/0.94/book.html ---------------------------------------------------------------------- diff --git a/0.94/book.html b/0.94/book.html index ee5df9e..bf28251 100644 --- a/0.94/book.html +++ b/0.94/book.html @@ -7,38 +7,38 @@ </a> </h2></div><div><p class="copyright">Copyright © 2012 Apache Software Foundation. All Rights Reserved. Apache Hadoop, Hadoop, MapReduce, HDFS, Zookeeper, HBase, and the HBase project logo are trademarks of the Apache Software Foundation. - </p></div><div><div class="revhistory"><table border="1" width="100%" summary="Revision history"><tr><th align="left" valign="top" colspan="2"><b>Revision History</b></th></tr><tr><td align="left">Revision + </p></div><div><div class="revhistory"><table border="1" width="100%" summary="Revision history"><tr><th align="left" valign="top" colspan="2"><b>Revision History</b></th></tr><tr><td align="left">Revision 0.94.27 </td><td align="left"> - 2015-11-03T11:44 + 2015-12-16T16:51 </td></tr></table></div></div><div><div class="abstract" title="Abstract"><p class="title"><b>Abstract</b></p><p>This is the official reference guide of <a class="link" href="http://www.hbase.org"; target="_top">Apache HBase (TM)</a>, a distributed, versioned, column-oriented database built on top of <a class="link" href="http://hadoop.apache.org/"; target="_top">Apache Hadoop</a> and <a class="link" href="http://zookeeper.apache.org/"; target="_top">Apache ZooKeeper</a>. - </p></div></div></div><hr></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="preface"><a href="#preface">Preface</a></span></dt><dt><span class="chapter"><a href="#getting_started">1. Getting Started</a></span></dt><dd><dl><dt><span class="section"><a href="#d1984e77">1.1. Introduction</a></span></dt><dt><span class="section"><a href="#quickstart">1.2. Quick Start</a></span></dt></dl></dd><dt><span class="chapter"><a href="#configuration">2. Apache HBase (TM) Configuration</a></span></dt><dd><dl><dt><span class="section"><a href="#basic.prerequisites">2.1. Basic Prerequisites</a></span></dt><dt><span class="section"><a href="#standalone_dist">2.2. HBase run modes: Standalone and Distributed</a></span></dt><dt><span class="section"><a href="#config.files">2.3. Configuration Files</a></span></dt><dt><span class="section"><a href="#example_config">2.4. Example Configurations</a></span></dt><dt><span class="section"><a href="#important_configurations">2.5. The Important Configurations</a></span></dt></dl></dd><dt><span class="chapter"><a href="#upgrading">3. Upgrading</a></span></dt><dd><dl><dt><span class="section"><a href="#upgrade0.96">3.1. Upgrading from 0.94.x to 0.96.x</a></span></dt><dt><span class="section"><a href="#upgrade0.94">3.2. Upgrading from 0.92.x to 0.94.x</a></span></dt><dt><span class="section"><a href="#upgrade0.92">3.3. Upgrading from 0.90.x to 0.92.x</a></span></dt><dt><span class="section"><a href="#upgrade0.90">3.4. Upgrading to HBase 0.90.x from 0.20.x or 0.89.x</a></span></dt></dl></dd><dt><span class="chapter"><a href="#shell">4. The Apache HBase Shell</a></span></dt><dd><dl><dt><span class="section"><a href="#scripting">4.1. Scripting</a></span></dt><dt><span class="section"><a href="#shell_tricks">4.2. Shell Tricks</a></span></dt></dl></dd><dt><span class="chapter"><a href="#datamodel">5. Data Model</a></span></dt><dd><dl><dt><span class="section"><a href="#conceptual.view">5.1. Conceptual View</a></span> </dt><dt><span class="section"><a href="#physical.view">5.2. Physical View</a></span></dt><dt><span class="section"><a href="#table">5.3. Table</a></span></dt><dt><span class="section"><a href="#row">5.4. Row</a></span></dt><dt><span class="section"><a href="#columnfamily">5.5. Column Family</a></span></dt><dt><span class="section"><a href="#cells">5.6. Cells</a></span></dt><dt><span class="section"><a href="#data_model_operations">5.7. Data Model Operations</a></span></dt><dt><span class="section"><a href="#versions">5.8. Versions</a></span></dt><dt><span class="section"><a href="#dm.sort">5.9. Sort Order</a></span></dt><dt><span class="section"><a href="#dm.column.metadata">5.10. Column Metadata</a></span></dt><dt><span class="section"><a href="#joins">5.11. Joins</a></span></dt><dt><span class="section"><a href="#acid">5.12. ACID</a></span></dt></dl></dd><dt><span class="chapter"><a href="#schema">6. HBase and Schema Design</a></span></dt><dd><dl><dt><span class="section"><a href ="#schema.creation">6.1. + </p></div></div></div><hr></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="preface"><a href="#preface">Preface</a></span></dt><dt><span class="chapter"><a href="#getting_started">1. Getting Started</a></span></dt><dd><dl><dt><span class="section"><a href="#d1984e77">1.1. Introduction</a></span></dt><dt><span class="section"><a href="#quickstart">1.2. Quick Start</a></span></dt></dl></dd><dt><span class="chapter"><a href="#configuration">2. Apache HBase (TM) Configuration</a></span></dt><dd><dl><dt><span class="section"><a href="#basic.prerequisites">2.1. Basic Prerequisites</a></span></dt><dt><span class="section"><a href="#standalone_dist">2.2. HBase run modes: Standalone and Distributed</a></span></dt><dt><span class="section"><a href="#config.files">2.3. Configuration Files</a></span></dt><dt><span class="section"><a href="#example_config">2.4. Example Configurations</a></span></dt><dt><span class="section"><a href="#important_configurations">2.5. The Important Configurations</a></span></dt></dl></dd><dt><span class="chapter"><a href="#upgrading">3. Upgrading</a></span></dt><dd><dl><dt><span class="section"><a href="#upgrade0.96">3.1. Upgrading from 0.94.x to 0.96.x</a></span></dt><dt><span class="section"><a href="#upgrade0.94">3.2. Upgrading from 0.92.x to 0.94.x</a></span></dt><dt><span class="section"><a href="#upgrade0.92">3.3. Upgrading from 0.90.x to 0.92.x</a></span></dt><dt><span class="section"><a href="#upgrade0.90">3.4. Upgrading to HBase 0.90.x from 0.20.x or 0.89.x</a></span></dt></dl></dd><dt><span class="chapter"><a href="#shell">4. The Apache HBase Shell</a></span></dt><dd><dl><dt><span class="section"><a href="#scripting">4.1. Scripting</a></span></dt><dt><span class="section"><a href="#shell_tricks">4.2. Shell Tricks</a></span></dt></dl></dd><dt><span class="chapter"><a href="#datamodel">5. Data Model</a></span></dt><dd><dl><dt><span class="section"><a href="#conceptual.view">5.1. Conceptual View</a></span> </dt><dt><span class="section"><a href="#physical.view">5.2. Physical View</a></span></dt><dt><span class="section"><a href="#table">5.3. Table</a></span></dt><dt><span class="section"><a href="#row">5.4. Row</a></span></dt><dt><span class="section"><a href="#columnfamily">5.5. Column Family</a></span></dt><dt><span class="section"><a href="#cells">5.6. Cells</a></span></dt><dt><span class="section"><a href="#data_model_operations">5.7. Data Model Operations</a></span></dt><dt><span class="section"><a href="#versions">5.8. Versions</a></span></dt><dt><span class="section"><a href="#dm.sort">5.9. Sort Order</a></span></dt><dt><span class="section"><a href="#dm.column.metadata">5.10. Column Metadata</a></span></dt><dt><span class="section"><a href="#joins">5.11. Joins</a></span></dt><dt><span class="section"><a href="#acid">5.12. ACID</a></span></dt></dl></dd><dt><span class="chapter"><a href="#schema">6. HBase and Schema Design</a></span></dt><dd><dl><dt><span class="section"><a href ="#schema.creation">6.1. Schema Creation - </a></span></dt><dt><span class="section"><a href="#number.of.cfs">6.2. + </a></span></dt><dt><span class="section"><a href="#number.of.cfs">6.2. On the number of column families - </a></span></dt><dt><span class="section"><a href="#rowkey.design">6.3. Rowkey Design</a></span></dt><dt><span class="section"><a href="#schema.versions">6.4. + </a></span></dt><dt><span class="section"><a href="#rowkey.design">6.3. Rowkey Design</a></span></dt><dt><span class="section"><a href="#schema.versions">6.4. Number of Versions - </a></span></dt><dt><span class="section"><a href="#supported.datatypes">6.5. + </a></span></dt><dt><span class="section"><a href="#supported.datatypes">6.5. Supported Datatypes - </a></span></dt><dt><span class="section"><a href="#schema.joins">6.6. Joins</a></span></dt><dt><span class="section"><a href="#ttl">6.7. Time To Live (TTL)</a></span></dt><dt><span class="section"><a href="#cf.keep.deleted">6.8. + </a></span></dt><dt><span class="section"><a href="#schema.joins">6.6. Joins</a></span></dt><dt><span class="section"><a href="#ttl">6.7. Time To Live (TTL)</a></span></dt><dt><span class="section"><a href="#cf.keep.deleted">6.8. Keeping Deleted Cells - </a></span></dt><dt><span class="section"><a href="#secondary.indexes">6.9. + </a></span></dt><dt><span class="section"><a href="#secondary.indexes">6.9. Secondary Indexes and Alternate Query Paths - </a></span></dt><dt><span class="section"><a href="#schema.smackdown">6.10. Schema Design Smackdown</a></span></dt><dt><span class="section"><a href="#schema.ops">6.11. Operational and Performance Configuration Options</a></span></dt><dt><span class="section"><a href="#constraints">6.12. Constraints</a></span></dt></dl></dd><dt><span class="chapter"><a href="#mapreduce">7. HBase and MapReduce</a></span></dt><dd><dl><dt><span class="section"><a href="#splitter">7.1. Map-Task Spitting</a></span></dt><dt><span class="section"><a href="#mapreduce.example">7.2. HBase MapReduce Examples</a></span></dt><dt><span class="section"><a href="#mapreduce.htable.access">7.3. Accessing Other HBase Tables in a MapReduce Job</a></span></dt><dt><span class="section"><a href="#mapreduce.specex">7.4. Speculative Execution</a></span></dt></dl></dd><dt><span class="chapter"><a href="#security">8. Secure Apache HBase (TM)</a></span></dt><dd><dl><dt><span class="section"><a href="#hbase.secure.configurati on">8.1. Secure Client Access to Apache HBase</a></span></dt><dt><span class="section"><a href="#hbase.accesscontrol.configuration">8.2. Access Control</a></span></dt><dt><span class="section"><a href="#hbase.secure.bulkload">8.3. Secure Bulk Load</a></span></dt></dl></dd><dt><span class="chapter"><a href="#architecture">9. Architecture</a></span></dt><dd><dl><dt><span class="section"><a href="#arch.overview">9.1. Overview</a></span></dt><dt><span class="section"><a href="#arch.catalog">9.2. Catalog Tables</a></span></dt><dt><span class="section"><a href="#client">9.3. Client</a></span></dt><dt><span class="section"><a href="#client.filter">9.4. Client Request Filters</a></span></dt><dt><span class="section"><a href="#master">9.5. Master</a></span></dt><dt><span class="section"><a href="#regionserver.arch">9.6. RegionServer</a></span></dt><dt><span class="section"><a href="#regions.arch">9.7. Regions</a></span></dt><dt><span class="section"><a href="#arch.bulk.load">9.8. Bulk Loadin g</a></span></dt><dt><span class="section"><a href="#arch.hdfs">9.9. HDFS</a></span></dt></dl></dd><dt><span class="chapter"><a href="#external_apis">10. Apache HBase (TM) External APIs</a></span></dt><dd><dl><dt><span class="section"><a href="#nonjava.jvm">10.1. Non-Java Languages Talking to the JVM</a></span></dt><dt><span class="section"><a href="#rest">10.2. REST</a></span></dt><dt><span class="section"><a href="#thrift">10.3. Thrift</a></span></dt><dt><span class="section"><a href="#c">10.4. C/C++ Apache HBase Client</a></span></dt></dl></dd><dt><span class="chapter"><a href="#performance">11. Apache HBase (TM) Performance Tuning</a></span></dt><dd><dl><dt><span class="section"><a href="#perf.os">11.1. Operating System</a></span></dt><dt><span class="section"><a href="#perf.network">11.2. Network</a></span></dt><dt><span class="section"><a href="#jvm">11.3. Java</a></span></dt><dt><span class="section"><a href="#perf.configurations">11.4. HBase Configurations</a></span></dt><dt ><span class="section"><a href="#perf.zookeeper">11.5. >ZooKeeper</a></span></dt><dt><span class="section"><a >href="#perf.schema">11.6. Schema Design</a></span></dt><dt><span >class="section"><a href="#perf.writing">11.7. Writing to >HBase</a></span></dt><dt><span class="section"><a href="#perf.reading">11.8. >Reading from HBase</a></span></dt><dt><span class="section"><a >href="#perf.deleting">11.9. Deleting from HBase</a></span></dt><dt><span >class="section"><a href="#perf.hdfs">11.10. HDFS</a></span></dt><dt><span >class="section"><a href="#perf.ec2">11.11. Amazon >EC2</a></span></dt><dt><span class="section"><a href="#perf.casestudy">11.12. >Case Studies</a></span></dt></dl></dd><dt><span class="chapter"><a >href="#trouble">12. Troubleshooting and Debugging Apache HBase >(TM)</a></span></dt><dd><dl><dt><span class="section"><a >href="#trouble.general">12.1. General Guidelines</a></span></dt><dt><span >class="section"><a href="#trouble.log">12.2. Logs</a></span></dt><dt><span >class="section" ><a href="#trouble.resources">12.3. Resources</a></span></dt><dt><span >class="section"><a href="#trouble.tools">12.4. Tools</a></span></dt><dt><span >class="section"><a href="#trouble.client">12.5. >Client</a></span></dt><dt><span class="section"><a >href="#trouble.mapreduce">12.6. MapReduce</a></span></dt><dt><span >class="section"><a href="#trouble.namenode">12.7. >NameNode</a></span></dt><dt><span class="section"><a >href="#trouble.network">12.8. Network</a></span></dt><dt><span >class="section"><a href="#trouble.rs">12.9. >RegionServer</a></span></dt><dt><span class="section"><a >href="#trouble.master">12.10. Master</a></span></dt><dt><span >class="section"><a href="#trouble.zookeeper">12.11. >ZooKeeper</a></span></dt><dt><span class="section"><a >href="#trouble.ec2">12.12. Amazon EC2</a></span></dt><dt><span >class="section"><a href="#trouble.versions">12.13. HBase and Hadoop version >issues</a></span></dt><dt><span class="section"><a >href="#trouble.casestudy">12.14. Case Studies</a></span>< /dt></dl></dd><dt><span class="chapter"><a href="#casestudies">13. Apache HBase (TM) Case Studies</a></span></dt><dd><dl><dt><span class="section"><a href="#casestudies.overview">13.1. Overview</a></span></dt><dt><span class="section"><a href="#casestudies.schema">13.2. Schema Design</a></span></dt><dt><span class="section"><a href="#casestudies.perftroub">13.3. Performance/Troubleshooting</a></span></dt></dl></dd><dt><span class="chapter"><a href="#ops_mgt">14. Apache HBase (TM) Operational Management</a></span></dt><dd><dl><dt><span class="section"><a href="#tools">14.1. HBase Tools and Utilities</a></span></dt><dt><span class="section"><a href="#ops.regionmgt">14.2. Region Management</a></span></dt><dt><span class="section"><a href="#node.management">14.3. Node Management</a></span></dt><dt><span class="section"><a href="#hbase_metrics">14.4. HBase Metrics</a></span></dt><dt><span class="section"><a href="#ops.monitoring">14.5. HBase Monitoring</a></span></dt><dt><span class="sec tion"><a href="#cluster_replication">14.6. Cluster Replication</a></span></dt><dt><span class="section"><a href="#ops.backup">14.7. HBase Backup</a></span></dt><dt><span class="section"><a href="#ops.snapshots">14.8. HBase Snapshots</a></span></dt><dt><span class="section"><a href="#ops.capacity">14.9. Capacity Planning</a></span></dt></dl></dd><dt><span class="chapter"><a href="#developer">15. Building and Developing Apache HBase (TM)</a></span></dt><dd><dl><dt><span class="section"><a href="#repos">15.1. Apache HBase Repositories</a></span></dt><dt><span class="section"><a href="#ides">15.2. IDEs</a></span></dt><dt><span class="section"><a href="#build">15.3. Building Apache HBase</a></span></dt><dt><span class="section"><a href="#mvn_repo">15.4. Adding an Apache HBase release to Apache's Maven Repository</a></span></dt><dt><span class="section"><a href="#documentation">15.5. Generating the HBase Reference Guide</a></span></dt><dt><span class="section"><a href="#hbase.org">15.6. U pdating hbase.apache.org</a></span></dt><dt><span class="section"><a href="#hbase.tests">15.7. Tests</a></span></dt><dt><span class="section"><a href="#maven.build.commands">15.8. Maven Build Commands</a></span></dt><dt><span class="section"><a href="#getting.involved">15.9. Getting Involved</a></span></dt><dt><span class="section"><a href="#developing">15.10. Developing</a></span></dt><dt><span class="section"><a href="#submitting.patches">15.11. Submitting Patches</a></span></dt></dl></dd><dt><span class="chapter"><a href="#zookeeper">16. ZooKeeper</a></span></dt><dd><dl><dt><span class="section"><a href="#d1984e11388">16.1. Using existing ZooKeeper ensemble</a></span></dt><dt><span class="section"><a href="#zk.sasl.auth">16.2. SASL Authentication with ZooKeeper</a></span></dt></dl></dd><dt><span class="chapter"><a href="#community">17. Community</a></span></dt><dd><dl><dt><span class="section"><a href="#decisions">17.1. Decisions</a></span></dt><dt><span class="section"><a href=" #community.roles">17.2. Community Roles</a></span></dt></dl></dd><dt><span class="appendix"><a href="#faq">A. FAQ</a></span></dt><dt><span class="appendix"><a href="#hbck.in.depth">B. hbck In Depth</a></span></dt><dd><dl><dt><span class="section"><a href="#d1984e12015">B.1. Running hbck to identify inconsistencies</a></span></dt><dt><span class="section"><a href="#d1984e12031">B.2. Inconsistencies</a></span></dt><dt><span class="section"><a href="#d1984e12050">B.3. Localized repairs</a></span></dt><dt><span class="section"><a href="#d1984e12088">B.4. Region Overlap Repairs</a></span></dt></dl></dd><dt><span class="appendix"><a href="#compression">C. Compression In HBase</a></span></dt><dd><dl><dt><span class="section"><a href="#compression.test">C.1. CompressionTest Tool</a></span></dt><dt><span class="section"><a href="#hbase.regionserver.codecs">C.2. + </a></span></dt><dt><span class="section"><a href="#schema.smackdown">6.10. Schema Design Smackdown</a></span></dt><dt><span class="section"><a href="#schema.ops">6.11. Operational and Performance Configuration Options</a></span></dt><dt><span class="section"><a href="#constraints">6.12. Constraints</a></span></dt></dl></dd><dt><span class="chapter"><a href="#mapreduce">7. HBase and MapReduce</a></span></dt><dd><dl><dt><span class="section"><a href="#splitter">7.1. Map-Task Spitting</a></span></dt><dt><span class="section"><a href="#mapreduce.example">7.2. HBase MapReduce Examples</a></span></dt><dt><span class="section"><a href="#mapreduce.htable.access">7.3. Accessing Other HBase Tables in a MapReduce Job</a></span></dt><dt><span class="section"><a href="#mapreduce.specex">7.4. Speculative Execution</a></span></dt></dl></dd><dt><span class="chapter"><a href="#security">8. Secure Apache HBase (TM)</a></span></dt><dd><dl><dt><span class="section"><a href="#hbase.secure.configurati on">8.1. Secure Client Access to Apache HBase</a></span></dt><dt><span class="section"><a href="#hbase.accesscontrol.configuration">8.2. Access Control</a></span></dt><dt><span class="section"><a href="#hbase.secure.bulkload">8.3. Secure Bulk Load</a></span></dt></dl></dd><dt><span class="chapter"><a href="#architecture">9. Architecture</a></span></dt><dd><dl><dt><span class="section"><a href="#arch.overview">9.1. Overview</a></span></dt><dt><span class="section"><a href="#arch.catalog">9.2. Catalog Tables</a></span></dt><dt><span class="section"><a href="#client">9.3. Client</a></span></dt><dt><span class="section"><a href="#client.filter">9.4. Client Request Filters</a></span></dt><dt><span class="section"><a href="#master">9.5. Master</a></span></dt><dt><span class="section"><a href="#regionserver.arch">9.6. RegionServer</a></span></dt><dt><span class="section"><a href="#regions.arch">9.7. Regions</a></span></dt><dt><span class="section"><a href="#arch.bulk.load">9.8. Bulk Loadin g</a></span></dt><dt><span class="section"><a href="#arch.hdfs">9.9. HDFS</a></span></dt></dl></dd><dt><span class="chapter"><a href="#external_apis">10. Apache HBase (TM) External APIs</a></span></dt><dd><dl><dt><span class="section"><a href="#nonjava.jvm">10.1. Non-Java Languages Talking to the JVM</a></span></dt><dt><span class="section"><a href="#rest">10.2. REST</a></span></dt><dt><span class="section"><a href="#thrift">10.3. Thrift</a></span></dt><dt><span class="section"><a href="#c">10.4. C/C++ Apache HBase Client</a></span></dt></dl></dd><dt><span class="chapter"><a href="#performance">11. Apache HBase (TM) Performance Tuning</a></span></dt><dd><dl><dt><span class="section"><a href="#perf.os">11.1. Operating System</a></span></dt><dt><span class="section"><a href="#perf.network">11.2. Network</a></span></dt><dt><span class="section"><a href="#jvm">11.3. Java</a></span></dt><dt><span class="section"><a href="#perf.configurations">11.4. HBase Configurations</a></span></dt><dt ><span class="section"><a href="#perf.zookeeper">11.5. >ZooKeeper</a></span></dt><dt><span class="section"><a >href="#perf.schema">11.6. Schema Design</a></span></dt><dt><span >class="section"><a href="#perf.writing">11.7. Writing to >HBase</a></span></dt><dt><span class="section"><a href="#perf.reading">11.8. >Reading from HBase</a></span></dt><dt><span class="section"><a >href="#perf.deleting">11.9. Deleting from HBase</a></span></dt><dt><span >class="section"><a href="#perf.hdfs">11.10. HDFS</a></span></dt><dt><span >class="section"><a href="#perf.ec2">11.11. Amazon >EC2</a></span></dt><dt><span class="section"><a href="#perf.casestudy">11.12. >Case Studies</a></span></dt></dl></dd><dt><span class="chapter"><a >href="#trouble">12. Troubleshooting and Debugging Apache HBase >(TM)</a></span></dt><dd><dl><dt><span class="section"><a >href="#trouble.general">12.1. General Guidelines</a></span></dt><dt><span >class="section"><a href="#trouble.log">12.2. Logs</a></span></dt><dt><span >class="section" ><a href="#trouble.resources">12.3. Resources</a></span></dt><dt><span >class="section"><a href="#trouble.tools">12.4. Tools</a></span></dt><dt><span >class="section"><a href="#trouble.client">12.5. >Client</a></span></dt><dt><span class="section"><a >href="#trouble.mapreduce">12.6. MapReduce</a></span></dt><dt><span >class="section"><a href="#trouble.namenode">12.7. >NameNode</a></span></dt><dt><span class="section"><a >href="#trouble.network">12.8. Network</a></span></dt><dt><span >class="section"><a href="#trouble.rs">12.9. >RegionServer</a></span></dt><dt><span class="section"><a >href="#trouble.master">12.10. Master</a></span></dt><dt><span >class="section"><a href="#trouble.zookeeper">12.11. >ZooKeeper</a></span></dt><dt><span class="section"><a >href="#trouble.ec2">12.12. Amazon EC2</a></span></dt><dt><span >class="section"><a href="#trouble.versions">12.13. HBase and Hadoop version >issues</a></span></dt><dt><span class="section"><a >href="#trouble.casestudy">12.14. Case Studies</a></span>< /dt></dl></dd><dt><span class="chapter"><a href="#casestudies">13. Apache HBase (TM) Case Studies</a></span></dt><dd><dl><dt><span class="section"><a href="#casestudies.overview">13.1. Overview</a></span></dt><dt><span class="section"><a href="#casestudies.schema">13.2. Schema Design</a></span></dt><dt><span class="section"><a href="#casestudies.perftroub">13.3. Performance/Troubleshooting</a></span></dt></dl></dd><dt><span class="chapter"><a href="#ops_mgt">14. Apache HBase (TM) Operational Management</a></span></dt><dd><dl><dt><span class="section"><a href="#tools">14.1. HBase Tools and Utilities</a></span></dt><dt><span class="section"><a href="#ops.regionmgt">14.2. Region Management</a></span></dt><dt><span class="section"><a href="#node.management">14.3. Node Management</a></span></dt><dt><span class="section"><a href="#hbase_metrics">14.4. HBase Metrics</a></span></dt><dt><span class="section"><a href="#ops.monitoring">14.5. HBase Monitoring</a></span></dt><dt><span class="sec tion"><a href="#cluster_replication">14.6. Cluster Replication</a></span></dt><dt><span class="section"><a href="#ops.backup">14.7. HBase Backup</a></span></dt><dt><span class="section"><a href="#ops.snapshots">14.8. HBase Snapshots</a></span></dt><dt><span class="section"><a href="#ops.capacity">14.9. Capacity Planning</a></span></dt></dl></dd><dt><span class="chapter"><a href="#developer">15. Building and Developing Apache HBase (TM)</a></span></dt><dd><dl><dt><span class="section"><a href="#repos">15.1. Apache HBase Repositories</a></span></dt><dt><span class="section"><a href="#ides">15.2. IDEs</a></span></dt><dt><span class="section"><a href="#build">15.3. Building Apache HBase</a></span></dt><dt><span class="section"><a href="#mvn_repo">15.4. Adding an Apache HBase release to Apache's Maven Repository</a></span></dt><dt><span class="section"><a href="#documentation">15.5. Generating the HBase Reference Guide</a></span></dt><dt><span class="section"><a href="#hbase.org">15.6. U pdating hbase.apache.org</a></span></dt><dt><span class="section"><a href="#hbase.tests">15.7. Tests</a></span></dt><dt><span class="section"><a href="#maven.build.commands">15.8. Maven Build Commands</a></span></dt><dt><span class="section"><a href="#getting.involved">15.9. Getting Involved</a></span></dt><dt><span class="section"><a href="#developing">15.10. Developing</a></span></dt><dt><span class="section"><a href="#submitting.patches">15.11. Submitting Patches</a></span></dt></dl></dd><dt><span class="chapter"><a href="#zookeeper">16. ZooKeeper</a></span></dt><dd><dl><dt><span class="section"><a href="#d1984e11388">16.1. Using existing ZooKeeper ensemble</a></span></dt><dt><span class="section"><a href="#zk.sasl.auth">16.2. SASL Authentication with ZooKeeper</a></span></dt></dl></dd><dt><span class="chapter"><a href="#community">17. Community</a></span></dt><dd><dl><dt><span class="section"><a href="#decisions">17.1. Decisions</a></span></dt><dt><span class="section"><a href=" #community.roles">17.2. Community Roles</a></span></dt></dl></dd><dt><span class="appendix"><a href="#faq">A. FAQ</a></span></dt><dt><span class="appendix"><a href="#hbck.in.depth">B. hbck In Depth</a></span></dt><dd><dl><dt><span class="section"><a href="#d1984e12015">B.1. Running hbck to identify inconsistencies</a></span></dt><dt><span class="section"><a href="#d1984e12031">B.2. Inconsistencies</a></span></dt><dt><span class="section"><a href="#d1984e12050">B.3. Localized repairs</a></span></dt><dt><span class="section"><a href="#d1984e12088">B.4. Region Overlap Repairs</a></span></dt></dl></dd><dt><span class="appendix"><a href="#compression">C. Compression In HBase</a></span></dt><dd><dl><dt><span class="section"><a href="#compression.test">C.1. CompressionTest Tool</a></span></dt><dt><span class="section"><a href="#hbase.regionserver.codecs">C.2. <code class="varname"> hbase.regionserver.codecs </code> - </a></span></dt><dt><span class="section"><a href="#lzo.compression">C.3. + </a></span></dt><dt><span class="section"><a href="#lzo.compression">C.3. LZO - </a></span></dt><dt><span class="section"><a href="#gzip.compression">C.4. + </a></span></dt><dt><span class="section"><a href="#gzip.compression">C.4. GZIP - </a></span></dt><dt><span class="section"><a href="#snappy.compression">C.5. + </a></span></dt><dt><span class="section"><a href="#snappy.compression">C.5. SNAPPY - </a></span></dt><dt><span class="section"><a href="#changing.compression">C.6. Changing Compression Schemes</a></span></dt></dl></dd><dt><span class="appendix"><a href="#d1984e12294">D. YCSB: The Yahoo! Cloud Serving Benchmark and HBase</a></span></dt><dt><span class="appendix"><a href="#hfilev2">E. HFile format version 2</a></span></dt><dd><dl><dt><span class="section"><a href="#d1984e12311">E.1. Motivation </a></span></dt><dt><span class="section"><a href="#d1984e12324">E.2. HFile format version 1 overview </a></span></dt><dt><span class="section"><a href="#d1984e12370">E.3. + </a></span></dt><dt><span class="section"><a href="#changing.compression">C.6. Changing Compression Schemes</a></span></dt></dl></dd><dt><span class="appendix"><a href="#d1984e12294">D. YCSB: The Yahoo! Cloud Serving Benchmark and HBase</a></span></dt><dt><span class="appendix"><a href="#hfilev2">E. HFile format version 2</a></span></dt><dd><dl><dt><span class="section"><a href="#d1984e12311">E.1. Motivation </a></span></dt><dt><span class="section"><a href="#d1984e12324">E.2. HFile format version 1 overview </a></span></dt><dt><span class="section"><a href="#d1984e12370">E.3. HBase file format with inline blocks (version 2) </a></span></dt></dl></dd><dt><span class="appendix"><a href="#other.info">F. Other Information About HBase</a></span></dt><dd><dl><dt><span class="section"><a href="#other.info.videos">F.1. HBase Videos</a></span></dt><dt><span class="section"><a href="#other.info.pres">F.2. HBase Presentations (Slides)</a></span></dt><dt><span class="section"><a href="#other.info.papers">F.3. HBase Papers</a></span></dt><dt><span class="section"><a href="#other.info.sites">F.4. HBase Sites</a></span></dt><dt><span class="section"><a href="#other.info.books">F.5. HBase Books</a></span></dt><dt><span class="section"><a href="#other.info.books.hadoop">F.6. Hadoop Books</a></span></dt></dl></dd><dt><span class="appendix"><a href="#hbase.history">G. HBase History</a></span></dt><dt><span class="appendix"><a href="#asf">H. HBase and the Apache Software Foundation</a></span></dt><dd><dl><dt><span class="section"><a href="#asf.devprocess">H.1. ASF Development Process</a></span></dt><dt><span class=" section"><a href="#asf.reporting">H.2. ASF Board Reporting</a></span></dt></dl></dd><dt><span class="appendix"><a href="#tracing">I. Enabling Dapper-like Tracing in HBase</a></span></dt><dd><dl><dt><span class="section"><a href="#tracing.spanreceivers">I.1. SpanReceivers</a></span></dt><dt><span class="section"><a href="#tracing.client.modifications">I.2. Client Modifications</a></span></dt></dl></dd><dt><span class="index"><a href="#book_index">Index</a></span></dt></dl></div><div class="list-of-tables"><p><b>List of Tables</b></p><dl><dt>2.1. <a href="#d1984e486">Hadoop version support matrix</a></dt><dt>5.1. <a href="#d1984e3253">Table <code class="varname">webtable</code></a></dt><dt>5.2. <a href="#d1984e3337">ColumnFamily <code class="varname">anchor</code></a></dt><dt>5.3. <a href="#d1984e3376">ColumnFamily <code class="varname">contents</code></a></dt><dt>8.1. <a href="#d1984e4606">Operation To Permission Mapping</a></dt></dl></div><div class="preface" title="Preface"><div cl ass="titlepage"><div><div><h2 class="title"><a name="preface"></a>Preface</h2></div></div></div><p>This is the official reference guide for the <a class="link" href="http://hbase.apache.org/"; target="_top">HBase</a> version it ships with. This document describes HBase version <span class="emphasis"><em>0.94.27</em></span>. @@ -514,7 +514,7 @@ stopping hbase...............</pre><p> Shutdown can take a moment to <span style="color: red"><property> <span style="color: red"><name>hbase.local.dir</name></span> <span style="color: red"><value>${hbase.tmp.dir}/local/</value></span> - <span style="color: red"><description>Directory on the local filesystem to be used + <span style="color: red"><description>Directory on the local filesystem to be used as a local storage. </description></span> </property></span> @@ -591,7 +591,7 @@ stopping hbase...............</pre><p> Shutdown can take a moment to server, getting a cell's value, starting a row update, etc. Default: 14. </description></span> - </property></span> + </property></span> <span style="color: red"><property> <span style="color: red"><name>hbase.bulkload.retries.number</name></span> <span style="color: red"><value>0</value></span> @@ -1023,12 +1023,12 @@ stopping hbase...............</pre><p> Shutdown can take a moment to <span style="color: red"><value>org.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSplitPolicy</value></span> <span style="color: red"><description> A split policy determines when a region should be split. The various other split policies that - are available currently are ConstantSizeRegionSplitPolicy, DisabledRegionSplitPolicy, - DelimitedKeyPrefixRegionSplitPolicy, KeyPrefixRegionSplitPolicy etc. + are available currently are ConstantSizeRegionSplitPolicy, DisabledRegionSplitPolicy, + DelimitedKeyPrefixRegionSplitPolicy, KeyPrefixRegionSplitPolicy etc. </description></span> </property></span> - + <span style="color: red"><property> <span style="color: red"><name>hbase.master.keytab.file</name></span> <span style="color: red"><value></value></span> @@ -1065,7 +1065,7 @@ stopping hbase...............</pre><p> Shutdown can take a moment to </description></span> </property></span> - + <span style="color: red"><property> <span style="color: red"><name>hadoop.policy.file</name></span> <span style="color: red"><value>hbase-policy.xml</value></span> @@ -1085,7 +1085,7 @@ stopping hbase...............</pre><p> Shutdown can take a moment to <span style="color: red"><property> <span style="color: red"><name>hbase.auth.key.update.interval</name></span> <span style="color: red"><value>86400000</value></span> - <span style="color: red"><description>The update interval for master key for authentication tokens + <span style="color: red"><description>The update interval for master key for authentication tokens in servers in milliseconds. Only used when HBase security is enabled. </description></span> </property></span> @@ -1147,7 +1147,7 @@ stopping hbase...............</pre><p> Shutdown can take a moment to <span style="color: red"><value>acl</value></span> <span style="color: red"><description>Root ZNode for access control lists.</description></span> </property></span> - + <span style="color: red"><property> <span style="color: red"><name>hbase.coprocessor.region.classes</name></span> <span style="color: red"><value></value></span> @@ -1171,7 +1171,7 @@ stopping hbase...............</pre><p> Shutdown can take a moment to </description></span> </property></span> - + <span style="color: red"><property> <span style="color: red"><name>hbase.zookeeper.quorum</name></span> <span style="color: red"><value>localhost</value></span> @@ -1210,9 +1210,9 @@ stopping hbase...............</pre><p> Shutdown can take a moment to not fail gracefully if multi-update is invoked (see ZOOKEEPER-1495). </description></span> </property></span> + - - + <span style="color: red"><property> <span style="color: red"><name>hbase.zookeeper.property.initLimit</name></span> <span style="color: red"><value>10</value></span> @@ -1252,7 +1252,7 @@ stopping hbase...............</pre><p> Shutdown can take a moment to standalone and pseudo-distributed. </description></span> </property></span> - + <span style="color: red"><property> <span style="color: red"><name>hbase.rest.port</name></span> <span style="color: red"><value>8080</value></span> @@ -1305,7 +1305,7 @@ stopping hbase...............</pre><p> Shutdown can take a moment to <span style="color: red"><name>hbase.online.schema.update.enable</name></span> <span style="color: red"><value>false</value></span> <span style="color: red"><description> - Set true to enable online schema changes. This is an experimental feature. + Set true to enable online schema changes. This is an experimental feature. There are known issues modifying table schemas at the same time a region split is happening so your table needs to be quiescent or else you have to be running with splits disabled. @@ -2206,14 +2206,14 @@ htable.put(put); consider the complete row to be deleted.</p><p>For an informative discussion on how deletes and versioning interact, see the thread <a class="link" href="http://comments.gmane.org/gmane.comp.java.hadoop.hbase.user/28421"; target="_top">Put w/ timestamp -> Deleteall -> Put w/ timestamp fails</a> up on the user mailing list.</p><p>Also see <a class="xref" href="#keyvalue" title="9.7.5.4. KeyValue">Section 9.7.5.4, “KeyValue”</a> for more information on the internal KeyValue format. - </p><p>Delete markers are purged during the major compaction of store, - unless the KEEP_DELETED_CELLS is set in the column family. In some - scenarios, users want to keep the deletes for a time and you can set the - delete TTL: hbase.hstore.time.to.purge.deletes in the configuration. - If this delete TTL is not set or set to 0, all delete markers including those - with future timestamp are purged during the later major compaction. - Otherwise, a delete marker is kept until the major compaction after - marker's timestamp + delete TTL. + </p><p>Delete markers are purged during the major compaction of store, + unless the KEEP_DELETED_CELLS is set in the column family. In some + scenarios, users want to keep the deletes for a time and you can set the + delete TTL: hbase.hstore.time.to.purge.deletes in the configuration. + If this delete TTL is not set or set to 0, all delete markers including those + with future timestamp are purged during the later major compaction. + Otherwise, a delete marker is kept until the major compaction after + marker's timestamp + delete TTL. </p></div></div><div class="section" title="5.8.2. Current Limitations"><div class="titlepage"><div><div><h3 class="title"><a name="d1984e3776"></a>5.8.2. Current Limitations</h3></div></div></div><div class="section" title="5.8.2.1. Deletes mask Puts"><div class="titlepage"><div><div><h4 class="title"><a name="d1984e3779"></a>5.8.2.1. Deletes mask Puts</h4></div></div></div><p>Deletes mask puts, even puts that happened after the delete was entered<sup>[<a name="d1984e3784" href="#ftn.d1984e3784" class="footnote">18</a>]</sup>. Remember that a delete writes a tombstone, which only disappears after then next major compaction has run. Suppose you do @@ -2261,31 +2261,31 @@ htable.put(put); adding another, <a class="link" href="http://hadoop-hbase.blogspot.com/2012/01/scanning-in-hbase.html"; target="_top">Scanning in HBase: Prefix Delete Marker</a></p></div><div class="footnote"><p><sup>[<a id="ftn.d1984e3761" href="#d1984e3761" class="para">17</a>] </sup>When HBase does a major compaction, the tombstones are processed to actually remove the dead values, together with the tombstones themselves.</p></div><div class="footnote"><p><sup>[<a id="ftn.d1984e3784" href="#d1984e3784" class="para">18</a>] </sup><a class="link" href="https://issues.apache.org/jira/browse/HBASE-2256"; target="_top">HBASE-2256</a></p></div><div class="footnote"><p><sup>[<a id="ftn.d1984e3795" href="#d1984e3795" class="para">19</a>] </sup>See <span class="emphasis"><em>Garbage Collection</em></span> in <a class="link" href="http://outerthought.org/blog/417-ot.html"; target="_top">Bending - time in HBase</a> </p></div></div></div><div class="chapter" title="Chapter 6. HBase and Schema Design"><div class="titlepage"><div><div><h2 class="title"><a name="schema"></a>Chapter 6. HBase and Schema Design</h2></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="section"><a href="#schema.creation">6.1. + time in HBase</a> </p></div></div></div><div class="chapter" title="Chapter 6. HBase and Schema Design"><div class="titlepage"><div><div><h2 class="title"><a name="schema"></a>Chapter 6. HBase and Schema Design</h2></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="section"><a href="#schema.creation">6.1. Schema Creation - </a></span></dt><dd><dl><dt><span class="section"><a href="#schema.updates">6.1.1. Schema Updates</a></span></dt></dl></dd><dt><span class="section"><a href="#number.of.cfs">6.2. + </a></span></dt><dd><dl><dt><span class="section"><a href="#schema.updates">6.1.1. Schema Updates</a></span></dt></dl></dd><dt><span class="section"><a href="#number.of.cfs">6.2. On the number of column families - </a></span></dt><dd><dl><dt><span class="section"><a href="#number.of.cfs.card">6.2.1. Cardinality of ColumnFamilies</a></span></dt></dl></dd><dt><span class="section"><a href="#rowkey.design">6.3. Rowkey Design</a></span></dt><dd><dl><dt><span class="section"><a href="#timeseries">6.3.1. + </a></span></dt><dd><dl><dt><span class="section"><a href="#number.of.cfs.card">6.2.1. Cardinality of ColumnFamilies</a></span></dt></dl></dd><dt><span class="section"><a href="#rowkey.design">6.3. Rowkey Design</a></span></dt><dd><dl><dt><span class="section"><a href="#timeseries">6.3.1. Monotonically Increasing Row Keys/Timeseries Data - </a></span></dt><dt><span class="section"><a href="#keysize">6.3.2. Try to minimize row and column sizes</a></span></dt><dt><span class="section"><a href="#reverse.timestamp">6.3.3. Reverse Timestamps</a></span></dt><dt><span class="section"><a href="#rowkey.scope">6.3.4. Rowkeys and ColumnFamilies</a></span></dt><dt><span class="section"><a href="#changing.rowkeys">6.3.5. Immutability of Rowkeys</a></span></dt><dt><span class="section"><a href="#rowkey.regionsplits">6.3.6. Relationship Between RowKeys and Region Splits</a></span></dt></dl></dd><dt><span class="section"><a href="#schema.versions">6.4. + </a></span></dt><dt><span class="section"><a href="#keysize">6.3.2. Try to minimize row and column sizes</a></span></dt><dt><span class="section"><a href="#reverse.timestamp">6.3.3. Reverse Timestamps</a></span></dt><dt><span class="section"><a href="#rowkey.scope">6.3.4. Rowkeys and ColumnFamilies</a></span></dt><dt><span class="section"><a href="#changing.rowkeys">6.3.5. Immutability of Rowkeys</a></span></dt><dt><span class="section"><a href="#rowkey.regionsplits">6.3.6. Relationship Between RowKeys and Region Splits</a></span></dt></dl></dd><dt><span class="section"><a href="#schema.versions">6.4. Number of Versions - </a></span></dt><dd><dl><dt><span class="section"><a href="#schema.versions.max">6.4.1. Maximum Number of Versions</a></span></dt><dt><span class="section"><a href="#schema.minversions">6.4.2. + </a></span></dt><dd><dl><dt><span class="section"><a href="#schema.versions.max">6.4.1. Maximum Number of Versions</a></span></dt><dt><span class="section"><a href="#schema.minversions">6.4.2. Minimum Number of Versions - </a></span></dt></dl></dd><dt><span class="section"><a href="#supported.datatypes">6.5. + </a></span></dt></dl></dd><dt><span class="section"><a href="#supported.datatypes">6.5. Supported Datatypes - </a></span></dt><dd><dl><dt><span class="section"><a href="#counters">6.5.1. Counters</a></span></dt></dl></dd><dt><span class="section"><a href="#schema.joins">6.6. Joins</a></span></dt><dt><span class="section"><a href="#ttl">6.7. Time To Live (TTL)</a></span></dt><dt><span class="section"><a href="#cf.keep.deleted">6.8. + </a></span></dt><dd><dl><dt><span class="section"><a href="#counters">6.5.1. Counters</a></span></dt></dl></dd><dt><span class="section"><a href="#schema.joins">6.6. Joins</a></span></dt><dt><span class="section"><a href="#ttl">6.7. Time To Live (TTL)</a></span></dt><dt><span class="section"><a href="#cf.keep.deleted">6.8. Keeping Deleted Cells - </a></span></dt><dt><span class="section"><a href="#secondary.indexes">6.9. + </a></span></dt><dt><span class="section"><a href="#secondary.indexes">6.9. Secondary Indexes and Alternate Query Paths - </a></span></dt><dd><dl><dt><span class="section"><a href="#secondary.indexes.filter">6.9.1. + </a></span></dt><dd><dl><dt><span class="section"><a href="#secondary.indexes.filter">6.9.1. Filter Query - </a></span></dt><dt><span class="section"><a href="#secondary.indexes.periodic">6.9.2. + </a></span></dt><dt><span class="section"><a href="#secondary.indexes.periodic">6.9.2. Periodic-Update Secondary Index - </a></span></dt><dt><span class="section"><a href="#secondary.indexes.dualwrite">6.9.3. + </a></span></dt><dt><span class="section"><a href="#secondary.indexes.dualwrite">6.9.3. Dual-Write Secondary Index - </a></span></dt><dt><span class="section"><a href="#secondary.indexes.summary">6.9.4. + </a></span></dt><dt><span class="section"><a href="#secondary.indexes.summary">6.9.4. Summary Tables - </a></span></dt><dt><span class="section"><a href="#secondary.indexes.coproc">6.9.5. + </a></span></dt><dt><span class="section"><a href="#secondary.indexes.coproc">6.9.5. Coprocessor Secondary Index </a></span></dt></dl></dd><dt><span class="section"><a href="#schema.smackdown">6.10. Schema Design Smackdown</a></span></dt><dd><dl><dt><span class="section"><a href="#schema.smackdown.rowsversions">6.10.1. Rows vs. Versions</a></span></dt><dt><span class="section"><a href="#schema.smackdown.rowscols">6.10.2. Rows vs. Columns</a></span></dt><dt><span class="section"><a href="#schema.smackdown.rowsascols">6.10.3. Rows as Columns</a></span></dt></dl></dd><dt><span class="section"><a href="#schema.ops">6.11. Operational and Performance Configuration Options</a></span></dt><dt><span class="section"><a href="#constraints">6.12. Constraints</a></span></dt></dl></div><p>A good general introduction on the strength and weaknesses modelling on the various non-rdbms datastores is Ian Varley's Master thesis, @@ -2847,12 +2847,12 @@ if (!b) { Master and RegionServer processes are deployed and make them readable only to the user account under which the HBase daemons will run. </p><p> - A Kerberos principal has three parts, with the form + A Kerberos principal has three parts, with the form <code class="code">username/fully.qualified.domain.n...@your-realm.com</code>. We recommend using <code class="code">hbase</code> as the username portion. </p><p> The following is an example of the configuration properties for - Kerberos operation that must be added to the + Kerberos operation that must be added to the <code class="code">hbase-site.xml</code> file on every server machine in the cluster. Required for even the most basic interactions with a secure Hadoop configuration, independent of HBase security. @@ -2885,7 +2885,7 @@ if (!b) { </p><p> Long running daemons with indefinite lifetimes that require client access to HBase can instead be configured to log in from a keytab. For - each host running such daemons, create a keytab with + each host running such daemons, create a keytab with <code class="code">kadmin</code> or <code class="code">kadmin.local</code>. The procedure for creating keytabs for HBase service is the same as for creating keytabs for Hadoop. Those steps are omitted here. Copy the resulting @@ -2896,8 +2896,8 @@ if (!b) { </p><pre class="programlisting"> <property> <name>hbase.security.authentication</name> - <value>kerberos</value> - </property> + <value>kerberos</value> + </property> <property> <name>hbase.security.authorization</name> <value>true</value> @@ -2915,14 +2915,14 @@ if (!b) { <property> <name>hbase.security.authentication</name> <value>kerberos</value> - </property> + </property> </pre><p> The client environment must be logged in to Kerberos from KDC or keytab via the <code class="code">kinit</code> command before communication with the HBase cluster will be possible. </p><p> Be advised that if the <code class="code">hbase.security.authentication</code> - in the client- and server-side site files do not match, the client will + in the client- and server-side site files do not match, the client will not be able to communicate with the cluster. </p><p> Once HBase is configured for secure RPC it is possible to optionally @@ -3001,7 +3001,7 @@ if (!b) { there. </p><p> You must also configure ZooKeeper for secure operation. Changes to ACLs - are synchronized throughout the cluster using ZooKeeper. Secure + are synchronized throughout the cluster using ZooKeeper. Secure authentication to ZooKeeper must be enabled or otherwise it will be possible to subvert HBase access control via direct client access to ZooKeeper. Refer to the section on secure ZooKeeper configuration and @@ -3010,7 +3010,7 @@ if (!b) { With Secure RPC and Access Control enabled, client access to HBase is authenticated and user data is private unless access has been explicitly granted. Access to data can be granted at a table or per - column family basis. + column family basis. </p><p> However, the following items have been left out of the initial implementation for simplicity: @@ -3043,7 +3043,7 @@ Access control mechanisms are mature and fairly standardized in the relational d the table, though it may be changed at table creation time or during an alter operation by setting or changing the OWNER table attribute. Only a single user principal can own a table at a given time. A table owner will - have all permissions over a given table. + have all permissions over a given table. </p></div><div class="section" title="8.2.3. Server-side Configuration for Access Control"><div class="titlepage"><div><div><h3 class="title"><a name="d1984e4744"></a>8.2.3. Server-side Configuration for Access Control</h3></div></div></div><p> Enable the AccessController coprocessor in the cluster configuration and restart HBase. The restart can be a rolling one. Complete the @@ -3062,7 +3062,7 @@ Access control mechanisms are mature and fairly standardized in the relational d org.apache.hadoop.hbase.security.access.AccessController</value> </property> </pre></div><div class="section" title="8.2.4. Shell Enhancements for Access Control"><div class="titlepage"><div><div><h3 class="title"><a name="d1984e4756"></a>8.2.4. Shell Enhancements for Access Control</h3></div></div></div><p> -The HBase shell has been extended to provide simple commands for editing and updating user permissions. The following commands have been added for access control list management: +The HBase shell has been extended to provide simple commands for editing and updating user permissions. The following commands have been added for access control list management: </p> Grant <p> @@ -3070,7 +3070,7 @@ The HBase shell has been extended to provide simple commands for editing and upd grant <user> <permissions> <table> [ <column family> [ <column qualifier> ] ] </pre><p> </p><p> - <code class="code"><permissions></code> is zero or more letters from the set "RWCA": READ('R'), WRITE('W'), CREATE('C'), ADMIN('A'). + <code class="code"><permissions></code> is zero or more letters from the set "RWCA": READ('R'), WRITE('W'), CREATE('C'), ADMIN('A'). </p><p> Note: Grants and revocations of individual permissions on a resource are both accomplished using the <code class="code">grant</code> command. A separate <code class="code">revoke</code> command is also provided by the shell, but this is for fast revocation of all of a user's access rights to a given resource only. </p><p> @@ -3579,7 +3579,7 @@ myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName( </p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem">rowlength <code class="code">------------> 4</code></li><li class="listitem">row <code class="code">-----------------> row1</code></li><li class="listitem">columnfamilylength <code class="code">---> 2</code></li><li class="listitem">columnfamily <code class="code">--------> cf</code></li><li class="listitem">columnqualifier <code class="code">------> attr1</code></li><li class="listitem">timestamp <code class="code">-----------> server time of Put</code></li><li class="listitem">keytype <code class="code">-------------> Put</code></li></ul></div><p> </p><p>Key portion for Put #2: </p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem">rowlength <code class="code">------------> 4</code></li><li class="listitem">row <code class="code">-----------------> row1</code></li><li class="listitem">columnfamilylength <code class="code">---> 2</code></li><li class="listitem">columnfamily <code class="code">--------> cf</code></li><li class="listitem">columnqualifier <code class="code">------> attr2</code></li><li class="listitem">timestamp <code class="code">-----------> server time of Put</code></li><li class="listitem">keytype <code class="code">-------------> Put</code></li></ul></div><p> - + </p></div><p>It is critical to understand that the rowkey, ColumnFamily, and column (aka columnqualifier) are embedded within the KeyValue instance. The longer these identifiers are, the bigger the KeyValue is.</p></div><div class="section" title="9.7.5.5. Compaction"><div class="titlepage"><div><div><h4 class="title"><a name="compaction"></a>9.7.5.5. Compaction</h4></div></div></div><p>There are two types of compactions: minor and major. Minor compactions will usually pick up a couple of the smaller adjacent StoreFiles and rewrite them as one. Minors do not drop deletes or expired cells, only major compactions do this. Sometimes a minor compaction @@ -3823,9 +3823,9 @@ myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName( column value. </p><p><span class="bold"><strong>Syntax:</strong></span> SingleColumnValueExcludeFilter(<compare operator>, '<comparator>', '<family>', '<qualifier>',<latest_version_boolean>, <filterIfColumnMissing_boolean>)</p><p><span class="bold"><strong>Syntax:</strong></span> SingleColumnValueExcludeFilter(<compare operator>, '<comparator>', '<family>', '<qualifier>') </p><p><span class="bold"><strong>Example:</strong></span> "SingleColumnValueExcludeFilter (‘<=’, ‘abc’,‘FamilyA’, ‘Column1’, ‘false’, ‘true’)"</p><p><span class="bold"><strong>Example:</strong></span> "SingleColumnValueExcludeFilter (‘<=’, ‘abc’, ‘FamilyA’, ‘Column1’)" </p></li><li class="listitem"><p><span class="bold"><strong><span class="underline">ColumnRangeFilter</span></strong></span></p><p><span class="bold">< strong>Description:</strong></span> This filter is used for selecting only those keys with columns that are between minColumn and maxColumn. It also takes two boolean variables to indicate whether to include the minColumn and maxColumn or not.</p><p>If you don’t want to set the minColumn or the maxColumn – you can pass in an empty argument.</p><p><span class="bold"><strong>Syntax:</strong></span> ColumnRangeFilter (‘<minColumn>’, <minColumnInclusive_bool>, ‘<maxColumn>’, <maxColumnInclusive_bool>)</p><p><span class="bold"><strong>Example:</strong></span> "ColumnRangeFilter (‘abc’, true, ‘xyz’, false)"</p></li></ol></div></div></div></div><div class="section" title="10.4. C/C++ Apache HBase Client"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="c"></a>10.4. C/C++ Apache HBase Client</h2></div></div></div><p>FB's Chip Turner wrote a pure C/C++ client. <a class="link" href="https://github.com/facebook/native-cpp-hbase-client"; target="_top">Check it out</a>. - </p></div></div><div class="chapter" title="Chapter 11. Apache HBase (TM) Performance Tuning"><div class="titlepage"><div><div><h2 class="title"><a name="performance"></a>Chapter 11. Apache HBase (TM) Performance Tuning</h2></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="section"><a href="#perf.os">11.1. Operating System</a></span></dt><dd><dl><dt><span class="section"><a href="#perf.os.ram">11.1.1. Memory</a></span></dt><dt><span class="section"><a href="#perf.os.64">11.1.2. 64-bit</a></span></dt><dt><span class="section"><a href="#perf.os.swap">11.1.3. Swapping</a></span></dt></dl></dd><dt><span class="section"><a href="#perf.network">11.2. Network</a></span></dt><dd><dl><dt><span class="section"><a href="#perf.network.1switch">11.2.1. Single Switch</a></span></dt><dt><span class="section"><a href="#perf.network.2switch">11.2.2. Multiple Switches</a></span></dt><dt><span class="section"><a href="#perf.network.multirack ">11.2.3. Multiple Racks</a></span></dt><dt><span class="section"><a href="#perf.network.ints">11.2.4. Network Interfaces</a></span></dt></dl></dd><dt><span class="section"><a href="#jvm">11.3. Java</a></span></dt><dd><dl><dt><span class="section"><a href="#gc">11.3.1. The Garbage Collector and Apache HBase</a></span></dt></dl></dd><dt><span class="section"><a href="#perf.configurations">11.4. HBase Configurations</a></span></dt><dd><dl><dt><span class="section"><a href="#perf.number.of.regions">11.4.1. Number of Regions</a></span></dt><dt><span class="section"><a href="#perf.compactions.and.splits">11.4.2. Managing Compactions</a></span></dt><dt><span class="section"><a href="#perf.handlers">11.4.3. <code class="varname">hbase.regionserver.handler.count</code></a></span></dt><dt><span class="section"><a href="#perf.hfile.block.cache.size">11.4.4. <code class="varname">hfile.block.cache.size</code></a></span></dt><dt><span class="section"><a href="#perf.rs.memstore.upperlimit">11.4. 5. <code class="varname">hbase.regionserver.global.memstore.upperLimit</code></a></span></dt><dt><span class="section"><a href="#perf.rs.memstore.lowerlimit">11.4.6. <code class="varname">hbase.regionserver.global.memstore.lowerLimit</code></a></span></dt><dt><span class="section"><a href="#perf.hstore.blockingstorefiles">11.4.7. <code class="varname">hbase.hstore.blockingStoreFiles</code></a></span></dt><dt><span class="section"><a href="#perf.hregion.memstore.block.multiplier">11.4.8. <code class="varname">hbase.hregion.memstore.block.multiplier</code></a></span></dt><dt><span class="section"><a href="#hbase.regionserver.checksum.verify">11.4.9. <code class="varname">hbase.regionserver.checksum.verify</code></a></span></dt></dl></dd><dt><span class="section"><a href="#perf.zookeeper">11.5. ZooKeeper</a></span></dt><dt><span class="section"><a href="#perf.schema">11.6. Schema Design</a></span></dt><dd><dl><dt><span class="section"><a href="#perf.number.of.cfs">11.6.1. Number of Col umn Families</a></span></dt><dt><span class="section"><a href="#perf.schema.keys">11.6.2. Key and Attribute Lengths</a></span></dt><dt><span class="section"><a href="#schema.regionsize">11.6.3. Table RegionSize</a></span></dt><dt><span class="section"><a href="#schema.bloom">11.6.4. Bloom Filters</a></span></dt><dt><span class="section"><a href="#schema.cf.blocksize">11.6.5. ColumnFamily BlockSize</a></span></dt><dt><span class="section"><a href="#cf.in.memory">11.6.6. In-Memory ColumnFamilies</a></span></dt><dt><span class="section"><a href="#perf.compression">11.6.7. Compression</a></span></dt></dl></dd><dt><span class="section"><a href="#perf.writing">11.7. Writing to HBase</a></span></dt><dd><dl><dt><span class="section"><a href="#perf.batch.loading">11.7.1. Batch Loading</a></span></dt><dt><span class="section"><a href="#precreate.regions">11.7.2. + </p></div></div><div class="chapter" title="Chapter 11. Apache HBase (TM) Performance Tuning"><div class="titlepage"><div><div><h2 class="title"><a name="performance"></a>Chapter 11. Apache HBase (TM) Performance Tuning</h2></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="section"><a href="#perf.os">11.1. Operating System</a></span></dt><dd><dl><dt><span class="section"><a href="#perf.os.ram">11.1.1. Memory</a></span></dt><dt><span class="section"><a href="#perf.os.64">11.1.2. 64-bit</a></span></dt><dt><span class="section"><a href="#perf.os.swap">11.1.3. Swapping</a></span></dt></dl></dd><dt><span class="section"><a href="#perf.network">11.2. Network</a></span></dt><dd><dl><dt><span class="section"><a href="#perf.network.1switch">11.2.1. Single Switch</a></span></dt><dt><span class="section"><a href="#perf.network.2switch">11.2.2. Multiple Switches</a></span></dt><dt><span class="section"><a href="#perf.network.multirack ">11.2.3. Multiple Racks</a></span></dt><dt><span class="section"><a href="#perf.network.ints">11.2.4. Network Interfaces</a></span></dt></dl></dd><dt><span class="section"><a href="#jvm">11.3. Java</a></span></dt><dd><dl><dt><span class="section"><a href="#gc">11.3.1. The Garbage Collector and Apache HBase</a></span></dt></dl></dd><dt><span class="section"><a href="#perf.configurations">11.4. HBase Configurations</a></span></dt><dd><dl><dt><span class="section"><a href="#perf.number.of.regions">11.4.1. Number of Regions</a></span></dt><dt><span class="section"><a href="#perf.compactions.and.splits">11.4.2. Managing Compactions</a></span></dt><dt><span class="section"><a href="#perf.handlers">11.4.3. <code class="varname">hbase.regionserver.handler.count</code></a></span></dt><dt><span class="section"><a href="#perf.hfile.block.cache.size">11.4.4. <code class="varname">hfile.block.cache.size</code></a></span></dt><dt><span class="section"><a href="#perf.rs.memstore.upperlimit">11.4. 5. <code class="varname">hbase.regionserver.global.memstore.upperLimit</code></a></span></dt><dt><span class="section"><a href="#perf.rs.memstore.lowerlimit">11.4.6. <code class="varname">hbase.regionserver.global.memstore.lowerLimit</code></a></span></dt><dt><span class="section"><a href="#perf.hstore.blockingstorefiles">11.4.7. <code class="varname">hbase.hstore.blockingStoreFiles</code></a></span></dt><dt><span class="section"><a href="#perf.hregion.memstore.block.multiplier">11.4.8. <code class="varname">hbase.hregion.memstore.block.multiplier</code></a></span></dt><dt><span class="section"><a href="#hbase.regionserver.checksum.verify">11.4.9. <code class="varname">hbase.regionserver.checksum.verify</code></a></span></dt></dl></dd><dt><span class="section"><a href="#perf.zookeeper">11.5. ZooKeeper</a></span></dt><dt><span class="section"><a href="#perf.schema">11.6. Schema Design</a></span></dt><dd><dl><dt><span class="section"><a href="#perf.number.of.cfs">11.6.1. Number of Col umn Families</a></span></dt><dt><span class="section"><a href="#perf.schema.keys">11.6.2. Key and Attribute Lengths</a></span></dt><dt><span class="section"><a href="#schema.regionsize">11.6.3. Table RegionSize</a></span></dt><dt><span class="section"><a href="#schema.bloom">11.6.4. Bloom Filters</a></span></dt><dt><span class="section"><a href="#schema.cf.blocksize">11.6.5. ColumnFamily BlockSize</a></span></dt><dt><span class="section"><a href="#cf.in.memory">11.6.6. In-Memory ColumnFamilies</a></span></dt><dt><span class="section"><a href="#perf.compression">11.6.7. Compression</a></span></dt></dl></dd><dt><span class="section"><a href="#perf.writing">11.7. Writing to HBase</a></span></dt><dd><dl><dt><span class="section"><a href="#perf.batch.loading">11.7.1. Batch Loading</a></span></dt><dt><span class="section"><a href="#precreate.regions">11.7.2. Table Creation: Pre-Creating Regions - </a></span></dt><dt><span class="section"><a href="#def.log.flush">11.7.3. + </a></span></dt><dt><span class="section"><a href="#def.log.flush">11.7.3. Table Creation: Deferred Log Flush </a></span></dt><dt><span class="section"><a href="#perf.hbase.client.autoflush">11.7.4. HBase Client: AutoFlush</a></span></dt><dt><span class="section"><a href="#perf.hbase.client.putwal">11.7.5. HBase Client: Turn off WAL on Puts</a></span></dt><dt><span class="section"><a href="#perf.hbase.client.regiongroup">11.7.6. HBase Client: Group Puts by RegionServer</a></span></dt><dt><span class="section"><a href="#perf.hbase.write.mr.reducer">11.7.7. MapReduce: Skip The Reducer</a></span></dt><dt><span class="section"><a href="#perf.one.region">11.7.8. Anti-Pattern: One Hot Region</a></span></dt></dl></dd><dt><span class="section"><a href="#perf.reading">11.8. Reading from HBase</a></span></dt><dd><dl><dt><span class="section"><a href="#perf.hbase.client.caching">11.8.1. Scan Caching</a></span></dt><dt><span class="section"><a href="#perf.hbase.client.selection">11.8.2. Scan Attribute Selection</a></span></dt><dt><span class="section"><a href="#perf.hbase.client.seek">11.8.3. A void scan seeks</a></span></dt><dt><span class="section"><a href="#perf.hbase.mr.input">11.8.4. MapReduce - Input Splits</a></span></dt><dt><span class="section"><a href="#perf.hbase.client.scannerclose">11.8.5. Close ResultScanners</a></span></dt><dt><span class="section"><a href="#perf.hbase.client.blockcache">11.8.6. Block Cache</a></span></dt><dt><span class="section"><a href="#perf.hbase.client.rowkeyonly">11.8.7. Optimal Loading of Row Keys</a></span></dt><dt><span class="section"><a href="#perf.hbase.read.dist">11.8.8. Concurrency: Monitor Data Spread</a></span></dt><dt><span class="section"><a href="#blooms">11.8.9. Bloom Filters</a></span></dt></dl></dd><dt><span class="section"><a href="#perf.deleting">11.9. Deleting from HBase</a></span></dt><dd><dl><dt><span class="section"><a href="#perf.deleting.queue">11.9.1. Using HBase Tables as Queues</a></span></dt><dt><span class="section"><a href="#perf.deleting.rpc">11.9.2. Delete RPC Behavior</a></span></dt></dl></dd><dt><spa n class="section"><a href="#perf.hdfs">11.10. HDFS</a></span></dt><dd><dl><dt><span class="section"><a href="#perf.hdfs.curr">11.10.1. Current Issues With Low-Latency Reads</a></span></dt><dt><span class="section"><a href="#perf.hdfs.configs.localread">11.10.2. Leveraging local data</a></span></dt><dt><span class="section"><a href="#perf.hdfs.comp">11.10.3. Performance Comparisons of HBase vs. HDFS</a></span></dt></dl></dd><dt><span class="section"><a href="#perf.ec2">11.11. Amazon EC2</a></span></dt><dt><span class="section"><a href="#perf.casestudy">11.12. Case Studies</a></span></dt></dl></div><div class="section" title="11.1. Operating System"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="perf.os"></a>11.1. Operating System</h2></div></div></div><div class="section" title="11.1.1. Memory"><div class="titlepage"><div><div><h3 class="title"><a name="perf.os.ram"></a>11.1.1. Memory</h3></div></div></div><p>RAM, RAM, RAM. Don't starve HBase.</p></div><div class="section" title="11.1.2. 64-bit"><div class="titlepage"><div><div><h3 class="title"><a name="perf.os.64"></a>11.1.2. 64-bit</h3></div></div></div><p>Use a 64-bit platform (and 64-bit JVM).</p></div><div class="section" title="11.1.3. Swapping"><div class="titlepage"><div><div><h3 class="title"><a name="perf.os.swap"></a>11.1.3. Swapping</h3></div></div></div><p>Watch out for swapping. Set swappiness to 0.</p></div></div><div class="section" title="11.2. Network"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="perf.network"></a>11.2. Network</h2></div></div></div><p> Perhaps the most important factor in avoiding network issues degrading Hadoop and HBbase performance is the switching hardware @@ -3928,10 +3928,10 @@ myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName( </p></div><div class="section" title="11.7.2. Table Creation: Pre-Creating Regions"><div class="titlepage"><div><div><h3 class="title"><a name="precreate.regions"></a>11.7.2. Table Creation: Pre-Creating Regions </h3></div></div></div><p> -Tables in HBase are initially created with one region by default. For bulk imports, this means that all clients will write to the same region -until it is large enough to split and become distributed across the cluster. A useful pattern to speed up the bulk import process is to pre-create empty regions. - Be somewhat conservative in this, because too-many regions can actually degrade performance. -</p><p>There are two different approaches to pre-creating splits. The first approach is to rely on the default <code class="code">HBaseAdmin</code> strategy +Tables in HBase are initially created with one region by default. For bulk imports, this means that all clients will write to the same region +until it is large enough to split and become distributed across the cluster. A useful pattern to speed up the bulk import process is to pre-create empty regions. + Be somewhat conservative in this, because too-many regions can actually degrade performance. +</p><p>There are two different approaches to pre-creating splits. The first approach is to rely on the default <code class="code">HBaseAdmin</code> strategy (which is implemented in <code class="code">Bytes.split</code>)... </p><pre class="programlisting"> byte[] startKey = ...; // your lowest keuy @@ -4784,10 +4784,10 @@ org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate w Use the HBase built against Hadoop 2.0 or rebuild your HBase passing the <span class="command"><strong>-Dhadoop.profile=2.0</strong></span> attribute to Maven (See <a class="xref" href="#maven.build.hadoop" title="15.8.3. Building against various hadoop versions.">Section 15.8.3, “Building against various hadoop versions.”</a> for more). </p></div></div><div class="section" title="12.14. Case Studies"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="trouble.casestudy"></a>12.14. Case Studies</h2></div></div></div><p>For Performance and Troubleshooting Case Studies, see <a class="xref" href="#casestudies" title="Chapter 13. Apache HBase (TM) Case Studies">Chapter 13, <i>Apache HBase (TM) Case Studies</i></a>. - </p></div><div class="footnotes"><br><hr width="100" align="left"><div class="footnote"><p><sup>[<a id="ftn.d1984e7978" href="#d1984e7978" class="para">28</a>] </sup>See Getting Answers</p></div></div></div><div class="chapter" title="Chapter 13. Apache HBase (TM) Case Studies"><div class="titlepage"><div><div><h2 class="title"><a name="casestudies"></a>Chapter 13. Apache HBase (TM) Case Studies</h2></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="section"><a href="#casestudies.overview">13.1. Overview</a></span></dt><dt><span class="section"><a href="#casestudies.schema">13.2. Schema Design</a></span></dt><dd><dl><dt><span class="section"><a href="#casestudies.schema.listdata">13.2.1. List Data</a></span></dt></dl></dd><dt><span class="section"><a href="#casestudies.perftroub">13.3. Performance/Troubleshooting</a></span></dt><dd><dl><dt><span class="section"><a href="#casestudies.slownode">13.3.1. Case Study #1 (Perfor mance Issue On A Single Node)</a></span></dt><dt><span class="section"><a href="#casestudies.perf.1">13.3.2. Case Study #2 (Performance Research 2012)</a></span></dt><dt><span class="section"><a href="#casestudies.perf.2">13.3.3. Case Study #3 (Performance Research 2010))</a></span></dt><dt><span class="section"><a href="#casestudies.xceivers">13.3.4. Case Study #4 (xcievers Config)</a></span></dt></dl></dd></dl></div><div class="section" title="13.1. Overview"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="casestudies.overview"></a>13.1. Overview</h2></div></div></div><p>This chapter will describe a variety of performance and troubleshooting case studies that can + </p></div><div class="footnotes"><br><hr width="100" align="left"><div class="footnote"><p><sup>[<a id="ftn.d1984e7978" href="#d1984e7978" class="para">28</a>] </sup>See Getting Answers</p></div></div></div><div class="chapter" title="Chapter 13. Apache HBase (TM) Case Studies"><div class="titlepage"><div><div><h2 class="title"><a name="casestudies"></a>Chapter 13. Apache HBase (TM) Case Studies</h2></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="section"><a href="#casestudies.overview">13.1. Overview</a></span></dt><dt><span class="section"><a href="#casestudies.schema">13.2. Schema Design</a></span></dt><dd><dl><dt><span class="section"><a href="#casestudies.schema.listdata">13.2.1. List Data</a></span></dt></dl></dd><dt><span class="section"><a href="#casestudies.perftroub">13.3. Performance/Troubleshooting</a></span></dt><dd><dl><dt><span class="section"><a href="#casestudies.slownode">13.3.1. Case Study #1 (Perfor mance Issue On A Single Node)</a></span></dt><dt><span class="section"><a href="#casestudies.perf.1">13.3.2. Case Study #2 (Performance Research 2012)</a></span></dt><dt><span class="section"><a href="#casestudies.perf.2">13.3.3. Case Study #3 (Performance Research 2010))</a></span></dt><dt><span class="section"><a href="#casestudies.xceivers">13.3.4. Case Study #4 (xcievers Config)</a></span></dt></dl></dd></dl></div><div class="section" title="13.1. Overview"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="casestudies.overview"></a>13.1. Overview</h2></div></div></div><p>This chapter will describe a variety of performance and troubleshooting case studies that can provide a useful blueprint on diagnosing Apache HBase (TM) cluster issues.</p><p>For more information on Performance and Troubleshooting, see <a class="xref" href="#performance" title="Chapter 11. Apache HBase (TM) Performance Tuning">Chapter 11, <i>Apache HBase (TM) Performance Tuning</i></a> and <a class="xref" href="#trouble" title="Chapter 12. Troubleshooting and Debugging Apache HBase (TM)">Chapter 12, <i>Troubleshooting and Debugging Apache HBase (TM)</i></a>. - </p></div><div class="section" title="13.2. Schema Design"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="casestudies.schema"></a>13.2. Schema Design</h2></div></div></div><div class="section" title="13.2.1. List Data"><div class="titlepage"><div><div><h3 class="title"><a name="casestudies.schema.listdata"></a>13.2.1. List Data</h3></div></div></div><p>The following is an exchange from the user dist-list regarding a fairly common question: - how to handle per-user list data in Apache HBase. + </p></div><div class="section" title="13.2. Schema Design"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="casestudies.schema"></a>13.2. Schema Design</h2></div></div></div><div class="section" title="13.2.1. List Data"><div class="titlepage"><div><div><h3 class="title"><a name="casestudies.schema.listdata"></a>13.2.1. List Data</h3></div></div></div><p>The following is an exchange from the user dist-list regarding a fairly common question: + how to handle per-user list data in Apache HBase. </p><p>*** QUESTION ***</p><p> We're looking at how to store a large amount of (per-user) list data in HBase, and we were trying to figure out what kind of access pattern made @@ -4894,25 +4894,25 @@ disadvantages!) at extremely high throughput, and the only way to really know that would be to try it out. If you don't have time to build it both ways and compare, my advice would be to start with the simplest option (one row per user+value). Start simple and iterate! :) - </p></div></div><div class="section" title="13.3. Performance/Troubleshooting"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="casestudies.perftroub"></a>13.3. Performance/Troubleshooting</h2></div></div></div><div class="section" title="13.3.1. Case Study #1 (Performance Issue On A Single Node)"><div class="titlepage"><div><div><h3 class="title"><a name="casestudies.slownode"></a>13.3.1. Case Study #1 (Performance Issue On A Single Node)</h3></div></div></div><div class="section" title="13.3.1.1. Scenario"><div class="titlepage"><div><div><h4 class="title"><a name="d1984e8865"></a>13.3.1.1. Scenario</h4></div></div></div><p>Following a scheduled reboot, one data node began exhibiting unusual behavior. Routine MapReduce - jobs run against HBase tables which regularly completed in five or six minutes began taking 30 or 40 minutes - to finish. These jobs were consistently found to be waiting on map and reduce tasks assigned to the troubled data node - (e.g., the slow map tasks all had the same Input Split). + </p></div></div><div class="section" title="13.3. Performance/Troubleshooting"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="casestudies.perftroub"></a>13.3. Performance/Troubleshooting</h2></div></div></div><div class="section" title="13.3.1. Case Study #1 (Performance Issue On A Single Node)"><div class="titlepage"><div><div><h3 class="title"><a name="casestudies.slownode"></a>13.3.1. Case Study #1 (Performance Issue On A Single Node)</h3></div></div></div><div class="section" title="13.3.1.1. Scenario"><div class="titlepage"><div><div><h4 class="title"><a name="d1984e8865"></a>13.3.1.1. Scenario</h4></div></div></div><p>Following a scheduled reboot, one data node began exhibiting unusual behavior. Routine MapReduce + jobs run against HBase tables which regularly completed in five or six minutes began taking 30 or 40 minutes + to finish. These jobs were consistently found to be waiting on map and reduce tasks assigned to the troubled data node + (e.g., the slow map tasks all had the same Input Split). The situation came to a head during a distributed copy, when the copy was severely prolonged by the lagging node. </p></div><div class="section" title="13.3.1.2. Hardware"><div class="titlepage"><div><div><h4 class="title"><a name="d1984e8870"></a>13.3.1.2. Hardware</h4></div></div></div><p>Datanodes: </p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem">Two 12-core processors</li><li class="listitem">Six Enerprise SATA disks</li><li class="listitem">24GB of RAM</li><li class="listitem">Two bonded gigabit NICs</li></ul></div><p> </p><p>Network: </p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem">10 Gigabit top-of-rack switches</li><li class="listitem">20 Gigabit bonded interconnects between racks.</li></ul></div><p> - </p></div><div class="section" title="13.3.1.3. Hypotheses"><div class="titlepage"><div><div><h4 class="title"><a name="d1984e8893"></a>13.3.1.3. Hypotheses</h4></div></div></div><div class="section" title="13.3.1.3.1. HBase "Hot Spot" Region"><div class="titlepage"><div><div><h5 class="title"><a name="d1984e8896"></a>13.3.1.3.1. HBase "Hot Spot" Region</h5></div></div></div><p>We hypothesized that we were experiencing a familiar point of pain: a "hot spot" region in an HBase table, - where uneven key-space distribution can funnel a huge number of requests to a single HBase region, bombarding the RegionServer - process and cause slow response time. Examination of the HBase Master status page showed that the number of HBase requests to the - troubled node was almost zero. Further, examination of the HBase logs showed that there were no region splits, compactions, or other region transitions + </p></div><div class="section" title="13.3.1.3. Hypotheses"><div class="titlepage"><div><div><h4 class="title"><a name="d1984e8893"></a>13.3.1.3. Hypotheses</h4></div></div></div><div class="section" title="13.3.1.3.1. HBase "Hot Spot" Region"><div class="titlepage"><div><div><h5 class="title"><a name="d1984e8896"></a>13.3.1.3.1. HBase "Hot Spot" Region</h5></div></div></div><p>We hypothesized that we were experiencing a familiar point of pain: a "hot spot" region in an HBase table, + where uneven key-space distribution can funnel a huge number of requests to a single HBase region, bombarding the RegionServer + process and cause slow response time. Examination of the HBase Master status page showed that the number of HBase requests to the + troubled node was almost zero. Further, examination of the HBase logs showed that there were no region splits, compactions, or other region transitions in progress. This effectively ruled out a "hot spot" as the root cause of the observed slowness. - </p></div><div class="section" title="13.3.1.3.2. HBase Region With Non-Local Data"><div class="titlepage"><div><div><h5 class="title"><a name="d1984e8901"></a>13.3.1.3.2. HBase Region With Non-Local Data</h5></div></div></div><p>Our next hypothesis was that one of the MapReduce tasks was requesting data from HBase that was not local to the datanode, thus - forcing HDFS to request data blocks from other servers over the network. Examination of the datanode logs showed that there were very - few blocks being requested over the network, indicating that the HBase region was correctly assigned, and that the majority of the necessary + </p></div><div class="section" title="13.3.1.3.2. HBase Region With Non-Local Data"><div class="titlepage"><div><div><h5 class="title"><a name="d1984e8901"></a>13.3.1.3.2. HBase Region With Non-Local Data</h5></div></div></div><p>Our next hypothesis was that one of the MapReduce tasks was requesting data from HBase that was not local to the datanode, thus + forcing HDFS to request data blocks from other servers over the network. Examination of the datanode logs showed that there were very + few blocks being requested over the network, indicating that the HBase region was correctly assigned, and that the majority of the necessary data was located on the node. This ruled out the possibility of non-local data causing a slowdown. - </p></div><div class="section" title="13.3.1.3.3. Excessive I/O Wait Due To Swapping Or An Over-Worked Or Failing Hard Disk"><div class="titlepage"><div><div><h5 class="title"><a name="d1984e8906"></a>13.3.1.3.3. Excessive I/O Wait Due To Swapping Or An Over-Worked Or Failing Hard Disk</h5></div></div></div><p>After concluding that the Hadoop and HBase were not likely to be the culprits, we moved on to troubleshooting the datanode's hardware. + </p></div><div class="section" title="13.3.1.3.3. Excessive I/O Wait Due To Swapping Or An Over-Worked Or Failing Hard Disk"><div class="titlepage"><div><div><h5 class="title"><a name="d1984e8906"></a>13.3.1.3.3. Excessive I/O Wait Due To Swapping Or An Over-Worked Or Failing Hard Disk</h5></div></div></div><p>After concluding that the Hadoop and HBase were not likely to be the culprits, we moved on to troubleshooting the datanode's hardware. Java, by design, will periodically scan its entire memory space to do garbage collection. If system memory is heavily overcommitted, the Linux kernel may enter a vicious cycle, using up all of its resources swapping Java heap back and forth from disk to RAM as Java tries to run garbage collection. Further, a failing hard disk will often retry reads and/or writes many times before giving up and returning an error. This can manifest @@ -7103,4 +7103,4 @@ See the HTrace <code class="filename">README</code> for more information on Samp ulimit and nproc - </a></dt></dl></div><div class="indexdiv"><h3>V</h3><dl><dt>Versions, <a class="indexterm" href="#versions">Versions</a></dt></dl></div><div class="indexdiv"><h3>X</h3><dl><dt>xcievers, <a class="indexterm" href="#dfs.datanode.max.xcievers">dfs.datanode.max.xcievers</a></dt></dl></div><div class="indexdiv"><h3>Z</h3><dl><dt>ZooKeeper, <a class="indexterm" href="#zookeeper">ZooKeeper</a></dt></dl></div></div></div></div></body></html> + </a></dt></dl></div><div class="indexdiv"><h3>V</h3><dl><dt>Versions, <a class="indexterm" href="#versions">Versions</a></dt></dl></div><div class="indexdiv"><h3>X</h3><dl><dt>xcievers, <a class="indexterm" href="#dfs.datanode.max.xcievers">dfs.datanode.max.xcievers</a></dt></dl></div><div class="indexdiv"><h3>Z</h3><dl><dt>ZooKeeper, <a class="indexterm" href="#zookeeper">ZooKeeper</a></dt></dl></div></div></div></div></body></html> \ No newline at end of file
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/ecb8d8ba/0.94/book/book.html ---------------------------------------------------------------------- diff --git a/0.94/book/book.html b/0.94/book/book.html index aedf343..8880cba 100644 --- a/0.94/book/book.html +++ b/0.94/book/book.html @@ -12,7 +12,7 @@ </p></div><div><div class="revhistory"><table border="1" width="100%" summary="Revision history"><tr><th align="left" valign="top" colspan="2"><b>Revision History</b></th></tr><tr><td align="left">Revision 0.94.27 </td><td align="left"> - 2015-11-03T11:44 + 2015-12-16T16:51 </td></tr></table></div></div><div><div class="abstract" title="Abstract"><p class="title"><b>Abstract</b></p><p>This is the official reference guide of <a class="link" href="http://www.hbase.org"; target="_top">Apache HBase (TM)</a>, a distributed, versioned, column-oriented database built on top of http://git-wip-us.apache.org/repos/asf/hbase-site/blob/ecb8d8ba/0.94/bulk-loads.html ---------------------------------------------------------------------- diff --git a/0.94/bulk-loads.html b/0.94/bulk-loads.html index ac11e2e..82b4c33 100644 --- a/0.94/bulk-loads.html +++ b/0.94/bulk-loads.html @@ -1,6 +1,6 @@ <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd";> -<!-- Generated by Apache Maven Doxia at 2015-11-03 --> +<!-- Generated by Apache Maven Doxia at 2015-12-16 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> @@ -15,7 +15,7 @@ </style> <link rel="stylesheet" href="./css/print.css" type="text/css" media="print" /> <link rel="shortcut icon" href="/images/favicon.ico" /> - <meta name="Date-Revision-yyyymmdd" content="20151103" /> + <meta name="Date-Revision-yyyymmdd" content="20151216" /> <meta http-equiv="Content-Language" content="en" /> <!--Google Analytics--> <script type="text/javascript"> @@ -190,7 +190,7 @@ <div id="footer"> <div class="xright"> - <span id="publishDate">Last Published: 2015-11-03</span> + <span id="publishDate">Last Published: 2015-12-16</span> | <span id="projectVersion">Version: 0.94.27</span> </div>
