Repository: hbase Updated Branches: refs/heads/master eec15bd17 -> 281648724
HBASE-11932 Docbook html-single build improvements Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/28164872 Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/28164872 Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/28164872 Branch: refs/heads/master Commit: 281648724758f571300e0bc167f16b2bcc023fb3 Parents: eec15bd Author: Misty Stanley-Jones <[email protected]> Authored: Thu Sep 11 09:06:17 2014 +1000 Committer: Misty Stanley-Jones <[email protected]> Committed: Thu Sep 11 09:06:17 2014 +1000 ---------------------------------------------------------------------- pom.xml | 1 + src/main/docbkx/appendix_hfile_format.xml | 28 +++--- src/main/docbkx/book.xml | 113 ++++++++++--------------- src/main/docbkx/community.xml | 5 +- src/main/docbkx/configuration.xml | 60 +++++-------- src/main/docbkx/cp.xml | 16 ++-- src/main/docbkx/ops_mgt.xml | 28 +++--- src/main/docbkx/performance.xml | 63 ++++++-------- src/main/docbkx/preface.xml | 2 +- src/main/docbkx/rpc.xml | 3 +- src/main/docbkx/security.xml | 8 +- src/main/docbkx/troubleshooting.xml | 23 +++-- src/main/docbkx/upgrading.xml | 8 +- src/main/docbkx/zookeeper.xml | 4 +- 14 files changed, 151 insertions(+), 211 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hbase/blob/28164872/pom.xml ---------------------------------------------------------------------- diff --git a/pom.xml b/pom.xml index 91fd6fa..3fbd4af 100644 --- a/pom.xml +++ b/pom.xml @@ -807,6 +807,7 @@ <phase>pre-site</phase> <configuration> <targetDirectory>${basedir}/target/docbkx/</targetDirectory> + <includes>book.xml</includes> <preProcess> <copy todir="target/docbkx/images"> <fileset dir="src/main/site/resources/images/"/> http://git-wip-us.apache.org/repos/asf/hbase/blob/28164872/src/main/docbkx/appendix_hfile_format.xml ---------------------------------------------------------------------- diff --git a/src/main/docbkx/appendix_hfile_format.xml b/src/main/docbkx/appendix_hfile_format.xml index aac58d3..ee43031 100644 --- a/src/main/docbkx/appendix_hfile_format.xml +++ b/src/main/docbkx/appendix_hfile_format.xml @@ -31,18 +31,22 @@ <para>As we will be discussing changes to the HFile format, it is useful to give a short overview of the original (HFile version 1) format.</para> <section xml:id="hfilev1.overview"> <title>Overview of Version 1</title> - <para>An HFile in version 1 format is structured as follows: - <inlinemediaobject> - <imageobject> - <imagedata align="center" valign="middle" fileref="hfile.png" /> - </imageobject> - <textobject> - <phrase>HFile Version 1</phrase> - </textobject> - - </inlinemediaobject> - <footnote><para>Image courtesy of Lars George, <link xlink:href="http://www.larsgeorge.com/2009/10/hbase-architecture-101-storage.html";>hbase-architecture-101-storage.html</link>.</para></footnote> - </para> + <para>An HFile in version 1 format is structured as follows:</para> + <figure> + <title>HFile V1 Format</title> + <mediaobject> + <imageobject> + <imagedata align="center" valign="middle" fileref="hfile.png"/> + </imageobject> + <textobject> + <phrase>HFile Version 1</phrase> + </textobject> + <caption><para>Image courtesy of Lars George, <link + xlink:href="http://www.larsgeorge.com/2009/10/hbase-architecture-101-storage.html"; + >hbase-architecture-101-storage.html</link>.</para></caption> + </mediaobject> + </figure> + </section> <section><title> Block index format in version 1 </title> <para>The block index in version 1 is very straightforward. For each entry, it contains: </para> http://git-wip-us.apache.org/repos/asf/hbase/blob/28164872/src/main/docbkx/book.xml ---------------------------------------------------------------------- diff --git a/src/main/docbkx/book.xml b/src/main/docbkx/book.xml index 683d8ef..00c5041 100644 --- a/src/main/docbkx/book.xml +++ b/src/main/docbkx/book.xml @@ -639,24 +639,19 @@ try { store file, the most recent values are found first.</para> <para>There is a lot of confusion over the semantics of <literal>cell</literal> versions, in - HBase. In particular, a couple questions that often come up are:</para> + HBase. In particular:</para> <itemizedlist> <listitem> - <para>If multiple writes to a cell have the same version, are all versions maintained or - just the last?<footnote> - <para>Currently, only the last written is fetchable.</para> - </footnote></para> + <para>If multiple writes to a cell have the same version, only the last written is + fetchable.</para> </listitem> <listitem> - <para>Is it OK to write cells in a non-increasing version order?<footnote> - <para>Yes</para> - </footnote></para> + <para>It is OK to write cells in a non-increasing version order.</para> </listitem> </itemizedlist> - <para>Below we describe how the version dimension in HBase currently works<footnote> - <para>See <link + <para>Below we describe how the version dimension in HBase currently works. See <link xlink:href="https://issues.apache.org/jira/browse/HBASE-2406";>HBASE-2406</link> for discussion of HBase versions. <link xlink:href="http://outerthought.org/blog/417-ot.html";>Bending time in HBase</link> @@ -665,7 +660,6 @@ try { <emphasis>Overwriting values at existing timestamps</emphasis> mentioned in the article no longer holds in HBase. This section is basically a synopsis of this article by Bruno Dumon.</para> - </footnote>.</para> <section xml:id="versions.ops"> @@ -783,11 +777,10 @@ htable.put(put); xml:id="version.delete"> <title>Delete</title> - <para>There are three different types of internal delete markers <footnote> - <para>See Lars Hofhansl's blog for discussion of his attempt adding another, <link - xlink:href="http://hadoop-hbase.blogspot.com/2012/01/scanning-in-hbase.html";>Scanning - in HBase: Prefix Delete Marker</link></para> - </footnote>: </para> + <para>There are three different types of internal delete markers. See Lars Hofhansl's blog + for discussion of his attempt adding another, <link + xlink:href="http://hadoop-hbase.blogspot.com/2012/01/scanning-in-hbase.html";>Scanning + in HBase: Prefix Delete Marker</link>. </para> <itemizedlist> <listitem> <para>Delete: for a specific version of a column.</para> @@ -808,11 +801,10 @@ htable.put(put); modifies data in place, so for example a delete will not immediately delete (or mark as deleted) the entries in the storage file that correspond to the delete condition. Rather, a so-called <emphasis>tombstone</emphasis> is written, which will mask the - deleted values<footnote> - <para>When HBase does a major compaction, the tombstones are processed to actually - remove the dead values, together with the tombstones themselves.</para> - </footnote>. If the version you specified when deleting a row is larger than the version - of any value in the row, then you can consider the complete row to be deleted.</para> + deleted values. When HBase does a major compaction, the tombstones are processed to + actually remove the dead values, together with the tombstones themselves. If the version + you specified when deleting a row is larger than the version of any value in the row, + then you can consider the complete row to be deleted.</para> <para>For an informative discussion on how deletes and versioning interact, see the thread <link xlink:href="http://comments.gmane.org/gmane.comp.java.hadoop.hbase.user/28421";>Put w/ timestamp -> Deleteall -> Put w/ timestamp fails</link> up on the user mailing @@ -846,10 +838,8 @@ htable.put(put); <title>Deletes mask Puts</title> <para>Deletes mask puts, even puts that happened after the delete - was entered<footnote> - <para><link - xlink:href="https://issues.apache.org/jira/browse/HBASE-2256";>HBASE-2256</link></para> - </footnote>. Remember that a delete writes a tombstone, which only + was entered. See <link xlink:href="https://issues.apache.org/jira/browse/HBASE-2256"; + >HBASE-2256</link>. Remember that a delete writes a tombstone, which only disappears after then next major compaction has run. Suppose you do a delete of everything <= T. After this you do a new put with a timestamp <= T. This put, even if it happened after the delete, @@ -868,14 +858,12 @@ htable.put(put); <title>Major compactions change query results</title> <para><quote>...create three cell versions at t1, t2 and t3, with a maximum-versions - setting of 2. So when getting all versions, only the values at t2 and t3 will be - returned. But if you delete the version at t2 or t3, the one at t1 will appear again. - Obviously, once a major compaction has run, such behavior will not be the case anymore...<footnote> - <para>See <emphasis>Garbage Collection</emphasis> in <link - xlink:href="http://outerthought.org/blog/417-ot.html";>Bending time in - HBase</link> - </para> - </footnote></quote></para> + setting of 2. So when getting all versions, only the values at t2 and t3 will be + returned. But if you delete the version at t2 or t3, the one at t1 will appear again. + Obviously, once a major compaction has run, such behavior will not be the case + anymore...</quote> (See <emphasis>Garbage Collection</emphasis> in <link + xlink:href="http://outerthought.org/blog/417-ot.html";>Bending time in + HBase</link>.)</para> </section> </section> </section> @@ -2020,14 +2008,13 @@ rs.close(); </section> <!-- client.filter --> <section xml:id="master"><title>Master</title> - <para><code>HMaster</code> is the implementation of the Master Server. The Master server - is responsible for monitoring all RegionServer instances in the cluster, and is - the interface for all metadata changes. In a distributed cluster, the Master typically runs on the <xref linkend="arch.hdfs.nn" /><footnote> - <para>J Mohamed Zahoor goes into some more detail on the Master Architecture in this blog posting, <link - xlink:href="http://blog.zahoor.in/2012/08/hbase-hmaster-architecture/";>HBase HMaster Architecture - </link>.</para> - </footnote> - </para> + <para><code>HMaster</code> is the implementation of the Master Server. The Master server is + responsible for monitoring all RegionServer instances in the cluster, and is the interface + for all metadata changes. In a distributed cluster, the Master typically runs on the <xref + linkend="arch.hdfs.nn"/>. J Mohamed Zahoor goes into some more detail on the Master + Architecture in this blog posting, <link + xlink:href="http://blog.zahoor.in/2012/08/hbase-hmaster-architecture/";>HBase HMaster + Architecture </link>.</para> <section xml:id="master.startup"><title>Startup Behavior</title> <para>If run in a multi-Master environment, all Masters compete to run the cluster. If the active Master loses its lease in ZooKeeper (or the Master shuts down), then then the remaining Masters jostle to @@ -2469,17 +2456,16 @@ rs.close(); physical RAM, and is likely to be less than the total available RAM due to other memory requirements and system constraints. </para> - <para>You can see how much memory -- onheap and offheap/direct -- a RegionServer is configured to use - and how much it is using at any one time by looking at the - <emphasis>Server Metrics: Memory</emphasis> tab in the UI. - It can also be gotten via JMX. In particular the direct - memory currently used by the server can be found on the - <varname>java.nio.type=BufferPool,name=direct</varname> - bean. - <footnote><para>Terracotta has a <link xlink:href="http://terracotta.org/documentation/4.0/bigmemorygo/configuration/storage-options";>good write up</link> on using offheap memory in java. - It is for their product BigMemory but alot of the issues noted apply - in general to any attempt at going offheap. Check it out.</para></footnote> - </para> + <para>You can see how much memory -- onheap and offheap/direct -- a RegionServer is + configured to use and how much it is using at any one time by looking at the + <emphasis>Server Metrics: Memory</emphasis> tab in the UI. It can also be gotten + via JMX. In particular the direct memory currently used by the server can be found + on the <varname>java.nio.type=BufferPool,name=direct</varname> bean. Terracotta has + a <link + xlink:href="http://terracotta.org/documentation/4.0/bigmemorygo/configuration/storage-options"; + >good write up</link> on using offheap memory in java. It is for their product + BigMemory but alot of the issues noted apply in general to any attempt at going + offheap. Check it out.</para> </note> <note xml:id="hbase.bucketcache.percentage.in.combinedcache"><title>hbase.bucketcache.percentage.in.combinedcache</title> <para>This is a pre-HBase 1.0 configuration removed because it @@ -2613,12 +2599,10 @@ rs.close(); </itemizedlist> <para>If the <varname>hbase.hlog.split.skip.errors</varname> optionset to <literal>false</literal>, the default, the exception will be propagated and the - split will be logged as failed.<footnote> - <para>See <link + split will be logged as failed. See <link xlink:href="https://issues.apache.org/jira/browse/HBASE-2958";>HBASE-2958 When hbase.hlog.split.skip.errors is set to false, we fail the split but thats it</link>. We need to do more than just fail split if this flag is set.</para> - </footnote></para> <section> <title>How EOFExceptions are treated when splitting a crashed RegionServers' @@ -2628,11 +2612,9 @@ rs.close(); <varname>hbase.hlog.split.skip.errors</varname> is set to <literal>false</literal>. An EOFException while reading the last log in the set of files to split is likely, because the RegionServer is likely to be in the process of - writing a record at the time of a crash. <footnote> - <para>For background, see <link + writing a record at the time of a crash. For background, see <link xlink:href="https://issues.apache.org/jira/browse/HBASE-2643";>HBASE-2643 Figure how to deal with eof splitting logs</link></para> - </footnote></para> </section> </section> @@ -3042,9 +3024,9 @@ ctime = Sat Jun 23 11:13:40 PDT 2012 </listitem> <listitem><para>Third replica is written on the same rack as the second, but on a different node chosen randomly</para> </listitem> - <listitem><para>Subsequent replicas are written on random nodes on the cluster -<footnote><para>See <emphasis>Replica Placement: The First Baby Steps</emphasis> on this page: <link xlink:href="http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html";>HDFS Architecture</link></para></footnote> - </para></listitem> + <listitem><para>Subsequent replicas are written on random nodes on the cluster. See <emphasis>Replica Placement: The First Baby Steps</emphasis> on this page: <link + xlink:href="http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html"; + >HDFS Architecture</link></para></listitem> </orderedlist> <para> Thus, HBase eventually achieves locality for a region after a flush or a compaction. @@ -5166,8 +5148,7 @@ This option should not normally be used, and it is not in <code>-fixAll</code>. <imageobject> <imagedata fileref="data_block_no_encoding.png" width="800"/> </imageobject> - <textobject><para></para> - </textobject> + <caption><para>A ColumnFamily with no encoding></para></caption> </mediaobject> </figure> <para>Here is the same data with prefix data encoding.</para> @@ -5177,8 +5158,7 @@ This option should not normally be used, and it is not in <code>-fixAll</code>. <imageobject> <imagedata fileref="data_block_prefix_encoding.png" width="800"/> </imageobject> - <textobject><para></para> - </textobject> + <caption><para>A ColumnFamily with prefix encoding</para></caption> </mediaobject> </figure> </listitem> @@ -5202,8 +5182,7 @@ This option should not normally be used, and it is not in <code>-fixAll</code>. <imageobject> <imagedata fileref="data_block_diff_encoding.png" width="800"/> </imageobject> - <textobject><para></para> - </textobject> + <caption><para>A ColumnFamily with diff encoding</para></caption> </mediaobject> </figure> </listitem> http://git-wip-us.apache.org/repos/asf/hbase/blob/28164872/src/main/docbkx/community.xml ---------------------------------------------------------------------- diff --git a/src/main/docbkx/community.xml b/src/main/docbkx/community.xml index 9bdaa39..813f356 100644 --- a/src/main/docbkx/community.xml +++ b/src/main/docbkx/community.xml @@ -40,12 +40,9 @@ committer will add it for you. Thereafter you can file issues against your feature branch in Apache HBase JIRA. Your code you keep elsewhere -- it should be public so it can be observed -- and you can update dev mailing list on progress. When the feature is ready for commit, 3 - +1s from committers will get your feature merged<footnote> - <para>See <link + +1s from committers will get your feature merged. See <link xlink:href="http://search-hadoop.com/m/asM982C5FkS1";>HBase, mail # dev - Thoughts about large feature dev branches</link></para> - </footnote> - </para> </section> <section xml:id="patchplusonepolicy"> http://git-wip-us.apache.org/repos/asf/hbase/blob/28164872/src/main/docbkx/configuration.xml ---------------------------------------------------------------------- diff --git a/src/main/docbkx/configuration.xml b/src/main/docbkx/configuration.xml index 898aa85..0af2b3c 100644 --- a/src/main/docbkx/configuration.xml +++ b/src/main/docbkx/configuration.xml @@ -212,12 +212,10 @@ <term>DNS</term> <listitem> <para>HBase uses the local hostname to self-report its IP address. Both forward and - reverse DNS resolving must work in versions of HBase previous to 0.92.0.<footnote> - <para>The <link + reverse DNS resolving must work in versions of HBase previous to 0.92.0. The <link xlink:href="https://github.com/sujee/hadoop-dns-checker";>hadoop-dns-checker</link> tool can be used to verify DNS is working correctly on the cluster. The project README file provides detailed instructions on usage. </para> - </footnote></para> <para>If your server has multiple network interfaces, HBase defaults to using the interface that the primary hostname resolves to. To override this behavior, set the @@ -306,11 +304,10 @@ running the HBase process is an operating system configuration, rather than an HBase configuration. It is also important to be sure that the settings are changed for the user that actually runs HBase. To see which user started HBase, and that user's ulimit - configuration, look at the first line of the HBase log for that instance.<footnote> - <para>A useful read setting config on you hadoop cluster is Aaron Kimballs' <link - xlink:href="http://www.cloudera.com/blog/2009/03/configuration-parameters-what-can-you-just-ignore/";>Configuration - Parameters: What can you just ignore?</link></para> - </footnote></para> + configuration, look at the first line of the HBase log for that instance. A useful read + setting config on you hadoop cluster is Aaron Kimballs' <link + xlink:href="http://www.cloudera.com/blog/2009/03/configuration-parameters-what-can-you-just-ignore/"; + >Configuration Parameters: What can you just ignore?</link></para> <formalpara xml:id="ulimit_ubuntu"> <title><command>ulimit</command> Settings on Ubuntu</title> <para>To configure <command>ulimit</command> settings on Ubuntu, edit @@ -414,12 +411,8 @@ hadoop - nproc 32000 <entry>HBase-0.92.x</entry> <entry>HBase-0.94.x</entry> <entry>HBase-0.96.x</entry> - <entry>HBase-0.98.x<footnote> - <para>Support for Hadoop 1.x is deprecated.</para> - </footnote></entry> - <entry>HBase-1.0.x<footnote> - <para>Hadoop 1.x is NOT supported</para> - </footnote></entry> + <entry><para>HBase-0.98.x (Support for Hadoop 1.x is deprecated.)</para></entry> + <entry><para>HBase-1.0.x (Hadoop 1.x is NOT supported)</para></entry> </row> </thead> <tbody> @@ -440,11 +433,9 @@ hadoop - nproc 32000 <entry>X</entry> </row> <row> - <entry>Hadoop-1.0.0-1.0.2<footnote> - <para>HBase requires hadoop 1.0.3 at a minimum; there is an issue where we cannot - find KerberosUtil compiling against earlier versions of Hadoop.</para> - </footnote> - </entry> + <entry><para>Hadoop-1.0.0-1.0.2 (HBase requires hadoop 1.0.3 at a minimum; there is an + issue where we cannot find KerberosUtil compiling against earlier versions of + Hadoop.)</para></entry> <entry>X</entry> <entry>X</entry> <entry>X</entry> @@ -494,10 +485,9 @@ hadoop - nproc 32000 <row> <entry>Hadoop-2.2.0 </entry> <entry>X</entry> - <entry>NT <footnote> - <para>To get 0.94.x to run on hadoop 2.2.0, you need to change the hadoop 2 and - protobuf versions in the <filename>pom.xml</filename>: Here is a diff with - pom.xml changes: </para> + <entry><para>NT - To get 0.94.x to run on hadoop 2.2.0, you need to change the hadoop + 2 and protobuf versions in the <filename>pom.xml</filename>: Here is a diff with + pom.xml changes: </para> <programlisting><![CDATA[$ svn diff pom.xml Index: pom.xml =================================================================== @@ -540,8 +530,7 @@ Index: pom.xml </itemizedlist> <para> Building against the hadoop 2 profile by running something like the following command: </para> - <screen language="bourne">$ mvn clean install assembly:single -Dhadoop.profile=2.0 -DskipTests</screen> - </footnote></entry> + <screen language="bourne">$ mvn clean install assembly:single -Dhadoop.profile=2.0 -DskipTests</screen></entry> <entry>S</entry> <entry>S</entry> <entry>NT</entry> @@ -601,11 +590,9 @@ Index: pom.xml <para> As of Apache HBase 0.96.x, Apache Hadoop 1.0.x at least is required. Hadoop 2 is strongly encouraged (faster but also has fixes that help MTTR). We will no longer run properly on older Hadoops such as 0.20.205 or branch-0.20-append. Do not move to Apache - HBase 0.96.x if you cannot upgrade your Hadoop.<footnote> - <para>See <link + HBase 0.96.x if you cannot upgrade your Hadoop.. See <link xlink:href="http://search-hadoop.com/m/7vFVx4EsUb2";>HBase, mail # dev - DISCUSS: Have hbase require at least hadoop 1.0.0 in hbase 0.96.0?</link></para> - </footnote></para> </section> <section @@ -615,13 +602,12 @@ Index: pom.xml <code>sync</code> implementation. DO NOT use Hadoop 0.20.2, Hadoop 0.20.203.0, and Hadoop 0.20.204.0 which DO NOT have this attribute. Currently only Hadoop versions 0.20.205.x or any release in excess of this version -- this includes hadoop-1.0.0 -- have - a working, durable sync <footnote> - <para>The Cloudera blog post <link - xlink:href="http://www.cloudera.com/blog/2012/01/an-update-on-apache-hadoop-1-0/";>An - update on Apache Hadoop 1.0</link> by Charles Zedlweski has a nice exposition on how - all the Hadoop versions relate. Its worth checking out if you are having trouble - making sense of the Hadoop version morass. </para> - </footnote>. Sync has to be explicitly enabled by setting + a working, durable sync. The Cloudera blog post <link + xlink:href="http://www.cloudera.com/blog/2012/01/an-update-on-apache-hadoop-1-0/";>An + update on Apache Hadoop 1.0</link> by Charles Zedlweski has a nice exposition on how all + the Hadoop versions relate. Its worth checking out if you are having trouble making sense + of the Hadoop version morass. </para> + <para>Sync has to be explicitly enabled by setting <varname>dfs.support.append</varname> equal to true on both the client side -- in <filename>hbase-site.xml</filename> -- and on the serverside in <filename>hdfs-site.xml</filename> (The sync facility HBase needs is a subset of the @@ -713,9 +699,7 @@ Index: pom.xml <para>Distributed mode can be subdivided into distributed but all daemons run on a single node -- a.k.a <emphasis>pseudo-distributed</emphasis>-- and <emphasis>fully-distributed</emphasis> where the daemons are spread across all nodes in - the cluster <footnote> - <para>The pseudo-distributed vs fully-distributed nomenclature comes from Hadoop.</para> - </footnote>.</para> + the cluster. The pseudo-distributed vs fully-distributed nomenclature comes from Hadoop.</para> <para>Pseudo-distributed mode can run against the local filesystem or it can run against an instance of the <emphasis>Hadoop Distributed File System</emphasis> (HDFS). http://git-wip-us.apache.org/repos/asf/hbase/blob/28164872/src/main/docbkx/cp.xml ---------------------------------------------------------------------- diff --git a/src/main/docbkx/cp.xml b/src/main/docbkx/cp.xml index 4ddc8c1..8624309 100644 --- a/src/main/docbkx/cp.xml +++ b/src/main/docbkx/cp.xml @@ -29,16 +29,14 @@ */ --> <title>Apache HBase Coprocessors</title> - <para> HBase coprocessors are modeled after the coprocessors which are part of Google's BigTable<footnote> - <para><link - xlink:href="http://www.scribd.com/doc/21631448/Dean-Keynote-Ladis2009"; />, pages - 66-67.</para> - </footnote>. Coprocessors function in a similar way to Linux kernel modules. They provide a way - to run server-level code against locally-stored data. The functionality they provide is very + <para> HBase coprocessors are modeled after the coprocessors which are part of Google's BigTable + (<link xlink:href="http://www.scribd.com/doc/21631448/Dean-Keynote-Ladis2009"/>, pages + 66-67.). Coprocessors function in a similar way to Linux kernel modules. They provide a way to + run server-level code against locally-stored data. The functionality they provide is very powerful, but also carries great risk and can have adverse effects on the system, at the level of the operating system. The information in this chapter is primarily sourced and heavily reused from Mingjie Lai's blog post at <link - xlink:href="https://blogs.apache.org/hbase/entry/coprocessor_introduction"; />. </para> + xlink:href="https://blogs.apache.org/hbase/entry/coprocessor_introduction"/>. </para> <para> Coprocessors are not designed to be used by end users of HBase, but by HBase developers who need to add specialized functionality to HBase. One example of the use of coprocessors is @@ -418,10 +416,10 @@ coprocessors=[AggregateImplementation] <imageobject> <imagedata fileref="coprocessor_stats.png" width="100%"/> </imageobject> - <textobject> + <caption> <para>The Coprocessor Metrics UI shows statistics about time spent executing a given coprocessor, including min, max, average, and 90th, 95th, and 99th percentile.</para> - </textobject> + </caption> </mediaobject> </figure> </section> http://git-wip-us.apache.org/repos/asf/hbase/blob/28164872/src/main/docbkx/ops_mgt.xml ---------------------------------------------------------------------- diff --git a/src/main/docbkx/ops_mgt.xml b/src/main/docbkx/ops_mgt.xml index 7328f67..21045bd 100644 --- a/src/main/docbkx/ops_mgt.xml +++ b/src/main/docbkx/ops_mgt.xml @@ -757,11 +757,9 @@ false <para>Without this facility, decommissioning mulitple nodes may be non-optimal because regions that are being drained from one region server may be moved to other regionservers that are also draining. Marking RegionServers to be in the draining state prevents this - from happening<footnote> - <para>See this <link - xlink:href="http://inchoate-clatter.blogspot.com/2012/03/hbase-ops-automation.html";>blog - post</link> for more details.</para> - </footnote>. </para> + from happening. See this <link + xlink:href="http://inchoate-clatter.blogspot.com/2012/03/hbase-ops-automation.html";>blog + post</link> for more details.</para> </section> <section @@ -1206,9 +1204,9 @@ $ for i in `cat conf/regionservers|sort`; do ./bin/graceful_stop.sh --restart -- <imageobject> <imagedata fileref="bc_basic.png" width="100%"/> </imageobject> - <textobject> + <caption> <para>Shows the cache implementation</para> - </textobject> + </caption> </mediaobject> </figure> <figure> @@ -1217,9 +1215,9 @@ $ for i in `cat conf/regionservers|sort`; do ./bin/graceful_stop.sh --restart -- <imageobject> <imagedata fileref="bc_config.png" width="100%"/> </imageobject> - <textobject> + <caption> <para>Shows all cache configuration options.</para> - </textobject> + </caption> </mediaobject> </figure> <figure> @@ -1228,9 +1226,9 @@ $ for i in `cat conf/regionservers|sort`; do ./bin/graceful_stop.sh --restart -- <imageobject> <imagedata fileref="bc_stats.png" width="100%"/> </imageobject> - <textobject> + <caption> <para>Shows statistics about the performance of the cache.</para> - </textobject> + </caption> </mediaobject> </figure> <figure> @@ -1242,9 +1240,9 @@ $ for i in `cat conf/regionservers|sort`; do ./bin/graceful_stop.sh --restart -- <imageobject> <imagedata fileref="bc_l2_buckets.png" width="100%"/> </imageobject> - <textobject> + <caption> <para>Shows information about the L1 and L2 caches.</para> - </textobject> + </caption> </mediaobject> </figure> <para>This is not an exhaustive list of all the screens and reports available. Have a look in @@ -1305,10 +1303,10 @@ $ for i in `cat conf/regionservers|sort`; do ./bin/graceful_stop.sh --restart -- <imageobject> <imagedata fileref="replication_overview.png" /> </imageobject> - <textobject> + <caption> <para>Illustration of the replication architecture in HBase, as described in the prior text.</para> - </textobject> + </caption> </mediaobject> </figure> http://git-wip-us.apache.org/repos/asf/hbase/blob/28164872/src/main/docbkx/performance.xml ---------------------------------------------------------------------- diff --git a/src/main/docbkx/performance.xml b/src/main/docbkx/performance.xml index 1940663..e7c0fc7 100644 --- a/src/main/docbkx/performance.xml +++ b/src/main/docbkx/performance.xml @@ -146,8 +146,7 @@ xml:id="gcpause"> <title>Long GC pauses</title> - <para - xml:id="mslab">In his presentation, <link + <para xml:id="mslab">In his presentation, <link xlink:href="http://www.slideshare.net/cloudera/hbase-hug-presentation";>Avoiding Full GCs with MemStore-Local Allocation Buffers</link>, Todd Lipcon describes two cases of stop-the-world garbage collections common in HBase, especially during loading; CMS failure @@ -158,16 +157,16 @@ Todd added an experimental facility, <indexterm><primary>MSLAB</primary></indexterm>, that must be explicitly enabled in Apache HBase 0.90.x (Its defaulted to be on in Apache 0.92.x HBase). See <code>hbase.hregion.memstore.mslab.enabled</code> to true in your - <classname>Configuration</classname>. See the cited slides for background and detail<footnote> - <para>The latest jvms do better regards fragmentation so make sure you are running a - recent release. Read down in the message, <link - xlink:href="http://osdir.com/ml/hotspot-gc-use/2011-11/msg00002.html";>Identifying - concurrent mode failures caused by fragmentation</link>.</para> - </footnote>. Be aware that when enabled, each MemStore instance will occupy at least an - MSLAB instance of memory. If you have thousands of regions or lots of regions each with - many column families, this allocation of MSLAB may be responsible for a good portion of - your heap allocation and in an extreme case cause you to OOME. Disable MSLAB in this case, - or lower the amount of memory it uses or float less regions per server. </para> + <classname>Configuration</classname>. See the cited slides for background and detail. + The latest jvms do better regards fragmentation so make sure you are running a recent + release. Read down in the message, <link + xlink:href="http://osdir.com/ml/hotspot-gc-use/2011-11/msg00002.html";>Identifying + concurrent mode failures caused by fragmentation</link>. Be aware that when enabled, + each MemStore instance will occupy at least an MSLAB instance of memory. If you have + thousands of regions or lots of regions each with many column families, this allocation of + MSLAB may be responsible for a good portion of your heap allocation and in an extreme case + cause you to OOME. Disable MSLAB in this case, or lower the amount of memory it uses or + float less regions per server. </para> <para>If you have a write-heavy workload, check out <link xlink:href="https://issues.apache.org/jira/browse/HBASE-8163";>HBASE-8163 MemStoreChunkPool: An improvement for JAVA GC when using MSLAB</link>. It describes @@ -916,24 +915,20 @@ htable.close(); latencies.</para> <para><link xlink:href="http://en.wikipedia.org/wiki/Bloom_filter";>Bloom filters</link> were developed - over in <link - xlink:href="https://issues.apache.org/jira/browse/HBASE-1200";>HBase-1200 Add - bloomfilters</link>.<footnote> - <para>For description of the development process -- why static blooms rather than dynamic + over in <link xlink:href="https://issues.apache.org/jira/browse/HBASE-1200";>HBase-1200 Add + bloomfilters</link>. For description of the development process -- why static blooms rather than dynamic -- and for an overview of the unique properties that pertain to blooms in HBase, as well as possible future directions, see the <emphasis>Development Process</emphasis> section of the document <link xlink:href="https://issues.apache.org/jira/secure/attachment/12444007/Bloom_Filters_in_HBase.pdf";>BloomFilters in HBase</link> attached to <link - xlink:href="https://issues.apache.org/jira/browse/HBASE-1200";>HBase-1200</link>.</para> - </footnote><footnote> - <para>The bloom filters described here are actually version two of blooms in HBase. In + xlink:href="https://issues.apache.org/jira/browse/HBASE-1200";>HBase-1200</link>. The bloom filters described here are actually version two of blooms in HBase. In versions up to 0.19.x, HBase had a dynamic bloom option based on work done by the <link xlink:href="http://www.one-lab.org";>European Commission One-Lab Project 034819</link>. The core of the HBase bloom work was later pulled up into Hadoop to implement org.apache.hadoop.io.BloomMapFile. Version 1 of HBase blooms never worked that well. Version 2 is a rewrite from scratch though again it starts with the one-lab work.</para> - </footnote></para> + <para>See also <xref linkend="schema.bloom" />. </para> @@ -1047,11 +1042,9 @@ htable.close(); possible for the DFSClient to take a "short circuit" and read directly from the disk instead of going through the DataNode when the data is local. What this means for HBase is that the RegionServers can read directly off their machine's disks instead of having to open a socket - to talk to the DataNode, the former being generally much faster<footnote> - <para>See JD's <link + to talk to the DataNode, the former being generally much faster. See JD's <link xlink:href="http://files.meetup.com/1350427/hug_ebay_jdcryans.pdf";>Performance - Talk</link></para> - </footnote>. Also see <link + Talk</link>. Also see <link xlink:href="http://search-hadoop.com/m/zV6dKrLCVh1";>HBase, mail # dev - read short circuit</link> thread for more discussion around short circuit reads. </para> <para>To enable "short circuit" reads, it will depend on your version of Hadoop. The original @@ -1081,19 +1074,17 @@ htable.close(); </description> </property>]]></programlisting> <para>Be careful about permissions for the directory that hosts the shared domain socket; - dfsclient will complain if open to other than the hbase user. <footnote> - <para>If you are running on an old Hadoop, one that is without <link - xlink:href="https://issues.apache.org/jira/browse/HDFS-347";>HDFS-347</link> but that - has <link - xlink:href="https://issues.apache.org/jira/browse/HDFS-2246";>HDFS-2246</link>, you - must set two configurations. First, the hdfs-site.xml needs to be amended. Set the - property <varname>dfs.block.local-path-access.user</varname> to be the - <emphasis>only</emphasis> user that can use the shortcut. This has to be the user that - started HBase. Then in hbase-site.xml, set - <varname>dfs.client.read.shortcircuit</varname> to be <varname>true</varname> - </para> - </footnote> + dfsclient will complain if open to other than the hbase user. </para> + <para>If you are running on an old Hadoop, one that is without <link + xlink:href="https://issues.apache.org/jira/browse/HDFS-347";>HDFS-347</link> but that has + <link xlink:href="https://issues.apache.org/jira/browse/HDFS-2246";>HDFS-2246</link>, you + must set two configurations. First, the hdfs-site.xml needs to be amended. Set the property + <varname>dfs.block.local-path-access.user</varname> to be the <emphasis>only</emphasis> + user that can use the shortcut. This has to be the user that started HBase. Then in + hbase-site.xml, set <varname>dfs.client.read.shortcircuit</varname> to be + <varname>true</varname> </para> + <para> Services -- at least the HBase RegionServers -- will need to be restarted in order to pick up the new configurations. </para> <note http://git-wip-us.apache.org/repos/asf/hbase/blob/28164872/src/main/docbkx/preface.xml ---------------------------------------------------------------------- diff --git a/src/main/docbkx/preface.xml b/src/main/docbkx/preface.xml index 582d04a..a8f6895 100644 --- a/src/main/docbkx/preface.xml +++ b/src/main/docbkx/preface.xml @@ -43,7 +43,7 @@ <title>About This Guide</title> <para>This reference guide is a work in progress. The source for this guide can be found in the <filename>src/main/docbkx</filename> directory of the HBase source. This reference - guide is marked up using <link xlink:href="http://www.docbook.com/";>DocBook</link> from + guide is marked up using <link xlink:href="http://www.docbook.org/";>DocBook</link> from which the the finished guide is generated as part of the 'site' build target. Run <programlisting language="bourne">mvn site</programlisting> to generate this documentation. Amendments and improvements to the documentation are welcomed. Click <link http://git-wip-us.apache.org/repos/asf/hbase/blob/28164872/src/main/docbkx/rpc.xml ---------------------------------------------------------------------- diff --git a/src/main/docbkx/rpc.xml b/src/main/docbkx/rpc.xml index 5745a33..2e5dd5f 100644 --- a/src/main/docbkx/rpc.xml +++ b/src/main/docbkx/rpc.xml @@ -104,7 +104,8 @@ <section> <title><preamble></title> - <para><programlisting><MAGIC 4 byte integer> <1 byte RPC Format Version> <1 byte auth type><footnote><para> We need the auth method spec. here so the connection header is encoded if auth enabled.</para></footnote></programlisting></para> + <programlisting><MAGIC 4 byte integer> <1 byte RPC Format Version> <1 byte auth type></programlisting> + <para> We need the auth method spec. here so the connection header is encoded if auth enabled.</para> <para>E.g.: HBas0x000x50 -- 4 bytes of MAGIC -- âHBasâ -- plus one-byte of version, 0 in this case, and one byte, 0x50 (SIMPLE). of an auth type.</para> http://git-wip-us.apache.org/repos/asf/hbase/blob/28164872/src/main/docbkx/security.xml ---------------------------------------------------------------------- diff --git a/src/main/docbkx/security.xml b/src/main/docbkx/security.xml index 4fe5aa0..c74af9b 100644 --- a/src/main/docbkx/security.xml +++ b/src/main/docbkx/security.xml @@ -32,11 +32,9 @@ <section xml:id="hbase.secure.configuration"> <title>Secure Client Access to Apache HBase</title> - <para>Newer releases of Apache HBase (>= 0.92) support optional SASL authentication of clients<footnote> - <para>See also Matteo Bertozzi's article on <link + <para>Newer releases of Apache HBase (>= 0.92) support optional SASL authentication of clients. See also Matteo Bertozzi's article on <link xlink:href="http://www.cloudera.com/blog/2012/09/understanding-user-authentication-and-authorization-in-apache-hbase/";>Understanding User Authentication and Authorization in Apache HBase</link>.</para> - </footnote>.</para> <para>This describes how to set up Apache HBase and clients for connection to secure HBase resources.</para> @@ -339,11 +337,9 @@ grant 'rest_server', 'RWCA' <section xml:id="hbase.secure.simpleconfiguration"> <title>Simple User Access to Apache HBase</title> - <para>Newer releases of Apache HBase (>= 0.92) support optional SASL authentication of clients<footnote> - <para>See also Matteo Bertozzi's article on <link + <para>Newer releases of Apache HBase (>= 0.92) support optional SASL authentication of clients. See also Matteo Bertozzi's article on <link xlink:href="http://www.cloudera.com/blog/2012/09/understanding-user-authentication-and-authorization-in-apache-hbase/";>Understanding User Authentication and Authorization in Apache HBase</link>.</para> - </footnote>.</para> <para>This describes how to set up Apache HBase and clients for simple user access to HBase resources.</para> http://git-wip-us.apache.org/repos/asf/hbase/blob/28164872/src/main/docbkx/troubleshooting.xml ---------------------------------------------------------------------- diff --git a/src/main/docbkx/troubleshooting.xml b/src/main/docbkx/troubleshooting.xml index ec6c1cf..f48e26f 100644 --- a/src/main/docbkx/troubleshooting.xml +++ b/src/main/docbkx/troubleshooting.xml @@ -233,19 +233,16 @@ export SERVER_GC_OPTS="$SERVER_GC_OPTS -XX:NewSize=64m -XX:MaxNewSize=64m" <section xml:id="trouble.resources.lists"> <title>Mailing Lists</title> - <para>Ask a question on the <link - xlink:href="http://hbase.apache.org/mail-lists.html";>Apache HBase mailing lists</link>. - The 'dev' mailing list is aimed at the community of developers actually building Apache - HBase and for features currently under development, and 'user' is generally used for - questions on released versions of Apache HBase. Before going to the mailing list, make sure - your question has not already been answered by searching the mailing list archives first. - Use <xref - linkend="trouble.resources.searchhadoop" />. Take some time crafting your question<footnote> - <para>See <link - xlink:href="http://www.mikeash.com/getting_answers.html";>Getting Answers</link></para> - </footnote>; a quality question that includes all context and exhibits evidence the author - has tried to find answers in the manual and out on lists is more likely to get a prompt - response. </para> + <para>Ask a question on the <link xlink:href="http://hbase.apache.org/mail-lists.html";>Apache + HBase mailing lists</link>. The 'dev' mailing list is aimed at the community of developers + actually building Apache HBase and for features currently under development, and 'user' is + generally used for questions on released versions of Apache HBase. Before going to the + mailing list, make sure your question has not already been answered by searching the mailing + list archives first. Use <xref linkend="trouble.resources.searchhadoop"/>. Take some time + crafting your question. See <link xlink:href="http://www.mikeash.com/getting_answers.html"; + >Getting Answers</link> for ideas on crafting good questions. A quality question that + includes all context and exhibits evidence the author has tried to find answers in the + manual and out on lists is more likely to get a prompt response. </para> </section> <section xml:id="trouble.resources.irc"> http://git-wip-us.apache.org/repos/asf/hbase/blob/28164872/src/main/docbkx/upgrading.xml ---------------------------------------------------------------------- diff --git a/src/main/docbkx/upgrading.xml b/src/main/docbkx/upgrading.xml index 2ba0143..b0a248b 100644 --- a/src/main/docbkx/upgrading.xml +++ b/src/main/docbkx/upgrading.xml @@ -71,12 +71,10 @@ out the jars of one version and replace them with the jars of another, compatible version and all will just work. Unless otherwise specified, HBase point versions are binary compatible. You can safely do rolling upgrades between binary compatible - versions; i.e. across point versions: e.g. from 0.94.5 to 0.94.6<footnote> - <para>See <link + versions; i.e. across point versions: e.g. from 0.94.5 to 0.94.6. See <link xlink:href="http://search-hadoop.com/m/bOOvwHGW981/Does+compatibility+between+versions+also+mean+binary+compatibility%253F&subj=Re+Does+compatibility+between+versions+also+mean+binary+compatibility+";>Does compatibility between versions also mean binary compatibility?</link> discussion on the hbaes dev mailing list. </para> - </footnote>. </para> </section> <section xml:id="hbase.rolling.restart"> @@ -506,12 +504,10 @@ Successfully completed Log splitting to change this (The 'normal'/default value is 64MB (67108864)). Run the script <filename>bin/set_meta_memstore_size.rb</filename>. This will make the necessary edit to your <varname>.META.</varname> schema. Failure to run this change will make for - a slow cluster <footnote> - <para> See <link + a slow cluster. See <link xlink:href="https://issues.apache.org/jira/browse/HBASE-3499";>HBASE-3499 Users upgrading to 0.90.0 need to have their .META. table updated with the right MEMSTORE_SIZE</link> </para> - </footnote> . </para> </section> </chapter> http://git-wip-us.apache.org/repos/asf/hbase/blob/28164872/src/main/docbkx/zookeeper.xml ---------------------------------------------------------------------- diff --git a/src/main/docbkx/zookeeper.xml b/src/main/docbkx/zookeeper.xml index 9f34a8f..206ccf5 100644 --- a/src/main/docbkx/zookeeper.xml +++ b/src/main/docbkx/zookeeper.xml @@ -52,12 +52,10 @@ <varname>hbase.zookeeper.property.clientPort</varname> property. For all default values used by HBase, including ZooKeeper configuration, see <xref linkend="hbase_default_configurations" />. Look for the - <varname>hbase.zookeeper.property</varname> prefix <footnote> - <para>For the full list of ZooKeeper configurations, see ZooKeeper's + <varname>hbase.zookeeper.property</varname> prefix. For the full list of ZooKeeper configurations, see ZooKeeper's <filename>zoo.cfg</filename>. HBase does not ship with a <filename>zoo.cfg</filename> so you will need to browse the <filename>conf</filename> directory in an appropriate ZooKeeper download.</para> - </footnote></para> <para>You must at least list the ensemble servers in <filename>hbase-site.xml</filename> using the <varname>hbase.zookeeper.quorum</varname> property. This property defaults to a single
