Added: hbase/hbase.apache.org/trunk/book/apks03.html URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/apks03.html?rev=1616896&view=auto ============================================================================== --- hbase/hbase.apache.org/trunk/book/apks03.html (added) +++ hbase/hbase.apache.org/trunk/book/apks03.html Fri Aug 8 22:19:16 2014 @@ -0,0 +1,64 @@ +<html><head> + <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> + <title>K.3. RPC</title><link rel="stylesheet" type="text/css" href="${baserdir}/src/main/site/resources/css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL Stylesheets V1.78.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="hbase.rpc.html" title="Appendix K. 0.95 RPC Specification"><link rel="prev" href="apks02.html" title="K.2. TODO"><link rel="next" href="apks04.html" title="K.4. Notes"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">K.3. RPC</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="apks02.html">Prev</a> </td><th width="60%" align="center">Appendix K. 0.95 RPC Specification</th><td width="20%" align="right"> <a accesskey="n" href="apks04.html">Next</a></td></tr></table><hr></div><script typ e="text/javascript"> + var disqus_shortname = 'hbase'; // required: replace example with your forum shortname + var disqus_url = 'http://hbase.apache.org/book/.html'; + </script><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d4029e22394"></a>K.3. RPC</h2></div></div></div><p>The client will send setup information on connection establish. Thereafter, the client + invokes methods against the remote server sending a protobuf Message and receiving a + protobuf Message in response. Communication is synchronous. All back and forth is + preceded by an int that has the total length of the request/response. Optionally, + Cells(KeyValues) can be passed outside of protobufs in follow-behind Cell blocks + (because <a class="link" href="https://docs.google.com/document/d/1WEtrq-JTIUhlnlnvA0oYRLp0F8MKpEBeBSCFcQiacdw/edit#"; target="_top">we + can’t protobuf megabytes of KeyValues</a> or Cells). These CellBlocks are encoded + and optionally compressed.</p><p></p><p>For more detail on the protobufs involved, see the <a class="link" href="http://svn.apache.org/viewvc/hbase/trunk/hbase-protocol/src/main/protobuf/RPC.proto?view=markup"; target="_top">RPC.proto</a> + file in trunk.</p><div class="section"><div class="titlepage"><div><div><h3 class="title"><a name="d4029e22408"></a>K.3.1. Connection Setup</h3></div></div></div><p>Client initiates connection.</p><div class="section"><div class="titlepage"><div><div><h4 class="title"><a name="d4029e22413"></a>K.3.1.1. Client</h4></div></div></div><p>On connection setup, client sends a preamble followed by a connection header. </p><div class="section"><div class="titlepage"><div><div><h5 class="title"><a name="d4029e22418"></a>K.3.1.1.1. <preamble></h5></div></div></div><pre class="programlisting"><MAGIC 4 byte integer> <1 byte RPC Format Version> <1 byte auth type><a href="#ftn.d4029e22424" class="footnote" name="d4029e22424"><sup class="footnote">[35]</sup></a></pre><p>E.g.: HBas0x000x50 -- 4 bytes of MAGIC -- ‘HBas’ -- plus one-byte of + version, 0 in this case, and one byte, 0x50 (SIMPLE). of an auth + type.</p></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a name="d4029e22429"></a>K.3.1.1.2. <Protobuf ConnectionHeader Message></h5></div></div></div><p>Has user info, and “protocol”, as well as the encoders and compression the + client will use sending CellBlocks. CellBlock encoders and compressors are + for the life of the connection. CellBlock encoders implement + org.apache.hadoop.hbase.codec.Codec. CellBlocks may then also be compressed. + Compressors implement org.apache.hadoop.io.compress.CompressionCodec. This + protobuf is written using writeDelimited so is prefaced by a pb varint with + its serialized length</p></div></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a name="d4029e22435"></a>K.3.1.2. Server</h4></div></div></div><p>After client sends preamble and connection header, server does NOT respond if + successful connection setup. No response means server is READY to accept + requests and to give out response. If the version or authentication in the + preamble is not agreeable or the server has trouble parsing the preamble, it + will throw a org.apache.hadoop.hbase.ipc.FatalConnectionException explaining the + error and will then disconnect. If the client in the connection header -- i.e. + the protobuf’d Message that comes after the connection preamble -- asks for for + a Service the server does not support or a codec the server does not have, again + we throw a FatalConnectionException with explanation.</p></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a name="d4029e22440"></a>K.3.2. Request</h3></div></div></div><p>After a Connection has been set up, client makes requests. Server responds.</p><p>A request is made up of a protobuf RequestHeader followed by a protobuf Message + parameter. The header includes the method name and optionally, metadata on the + optional CellBlock that may be following. The parameter type suits the method being + invoked: i.e. if we are doing a getRegionInfo request, the protobuf Message param + will be an instance of GetRegionInfoRequest. The response will be a + GetRegionInfoResponse. The CellBlock is optionally used ferrying the bulk of the RPC + data: i.e Cells/KeyValues.</p><div class="section"><div class="titlepage"><div><div><h4 class="title"><a name="d4029e22447"></a>K.3.2.1. Request Parts</h4></div></div></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a name="d4029e22450"></a>K.3.2.1.1. <Total Length></h5></div></div></div><p>The request is prefaced by an int that holds the total length of what + follows.</p></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a name="d4029e22455"></a>K.3.2.1.2. <Protobuf RequestHeader Message></h5></div></div></div><p>Will have call.id, trace.id, and method name, etc. including optional + Metadata on the Cell block IFF one is following. Data is protobuf’d inline + in this pb Message or optionally comes in the following CellBlock</p></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a name="d4029e22460"></a>K.3.2.1.3. <Protobuf Param Message></h5></div></div></div><p>If the method being invoked is getRegionInfo, if you study the Service + descriptor for the client to regionserver protocol, you will find that the + request sends a GetRegionInfoRequest protobuf Message param in this + position.</p></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a name="d4029e22465"></a>K.3.2.1.4. <CellBlock></h5></div></div></div><p>An encoded and optionally compressed Cell block.</p></div></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a name="d4029e22472"></a>K.3.3. Response</h3></div></div></div><p>Same as Request, it is a protobuf ResponseHeader followed by a protobuf Message + response where the Message response type suits the method invoked. Bulk of the data + may come in a following CellBlock.</p><div class="section"><div class="titlepage"><div><div><h4 class="title"><a name="d4029e22477"></a>K.3.3.1. Response Parts</h4></div></div></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a name="d4029e22480"></a>K.3.3.1.1. <Total Length></h5></div></div></div><p>The response is prefaced by an int that holds the total length of what + follows.</p></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a name="d4029e22485"></a>K.3.3.1.2. <Protobuf ResponseHeader Message></h5></div></div></div><p>Will have call.id, etc. Will include exception if failed processing. + Optionally includes metadata on optional, IFF there is a CellBlock + following.</p></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a name="d4029e22490"></a>K.3.3.1.3. <Protobuf Response Message></h5></div></div></div><p>Return or may be nothing if exception. If the method being invoked is + getRegionInfo, if you study the Service descriptor for the client to + regionserver protocol, you will find that the response sends a + GetRegionInfoResponse protobuf Message param in this position.</p></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a name="d4029e22495"></a>K.3.3.1.4. <CellBlock></h5></div></div></div><p>An encoded and optionally compressed Cell block.</p></div></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a name="d4029e22502"></a>K.3.4. Exceptions</h3></div></div></div><p>There are two distinct types. There is the request failed which is encapsulated + inside the response header for the response. The connection stays open to receive + new requests. The second type, the FatalConnectionException, kills the + connection.</p><p>Exceptions can carry extra information. See the ExceptionResponse protobuf type. + It has a flag to indicate do-no-retry as well as other miscellaneous payload to help + improve client responsiveness.</p></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a name="d4029e22509"></a>K.3.5. CellBlocks</h3></div></div></div><p>These are not versioned. Server can do the codec or it cannot. If new version of a + codec with say, tighter encoding, then give it a new class name. Codecs will live on + the server for all time so old clients can connect.</p></div><div class="footnotes"><br><hr style="width:100; text-align:left;margin-left: 0"><div id="ftn.d4029e22424" class="footnote"><p><a href="#d4029e22424" class="para"><sup class="para">[35] </sup></a> We need the auth method spec. here so the connection header is encoded if auth enabled.</p></div></div></div><div id="disqus_thread"></div><script type="text/javascript"> + /* * * DON'T EDIT BELOW THIS LINE * * */ + (function() { + var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true; + dsq.src = 'http://' + disqus_shortname + '.disqus.com/embed.js'; + (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq); + })(); +</script><noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript";>comments powered by Disqus.</a></noscript><a href="http://disqus.com"; class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="apks02.html">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="hbase.rpc.html">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="apks04.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">K.2. TODO </td><td width="20%" align="center"><a accesskey="h" href="book.html">Home</a></td><td width="40%" align="right" valign="top"> K.4. Notes</td></tr></table></div></body></html> \ No newline at end of file
Added: hbase/hbase.apache.org/trunk/book/apks04.html URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/apks04.html?rev=1616896&view=auto ============================================================================== --- hbase/hbase.apache.org/trunk/book/apks04.html (added) +++ hbase/hbase.apache.org/trunk/book/apks04.html Fri Aug 8 22:19:16 2014 @@ -0,0 +1,43 @@ +<html><head> + <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> + <title>K.4. Notes</title><link rel="stylesheet" type="text/css" href="${baserdir}/src/main/site/resources/css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL Stylesheets V1.78.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="hbase.rpc.html" title="Appendix K. 0.95 RPC Specification"><link rel="prev" href="apks03.html" title="K.3. RPC"><link rel="next" href="book_index.html" title="Index"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">K.4. Notes</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="apks03.html">Prev</a> </td><th width="60%" align="center">Appendix K. 0.95 RPC Specification</th><td width="20%" align="right"> <a accesskey="n" href="book_index.html">Next</a></td></tr></table><hr></div><script ty pe="text/javascript"> + var disqus_shortname = 'hbase'; // required: replace example with your forum shortname + var disqus_url = 'http://hbase.apache.org/book/.html'; + </script><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d4029e22514"></a>K.4. Notes</h2></div></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a name="d4029e22517"></a>K.4.1. Constraints</h3></div></div></div><p>In some part, current wire-format -- i.e. all requests and responses preceeded by + a length -- has been dictated by current server non-async architecture.</p></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a name="d4029e22522"></a>K.4.2. One fat pb request or header+param</h3></div></div></div><p>We went with pb header followed by pb param making a request and a pb header + followed by pb response for now. Doing header+param rather than a single protobuf + Message with both header and param content:</p><p> + </p><div class="orderedlist"><ol class="orderedlist" type="1"><li class="listitem"><p>Is closer to what we currently have</p></li><li class="listitem"><p>Having a single fat pb requires extra copying putting the already pb’d + param into the body of the fat request pb (and same making + result)</p></li><li class="listitem"><p>We can decide whether to accept the request or not before we read the + param; for example, the request might be low priority. As is, we read + header+param in one go as server is currently implemented so this is a + TODO.</p></li></ol></div><p> + </p><p>The advantages are minor. If later, fat request has clear advantage, can roll out + a v2 later.</p></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a name="rpc.configs"></a>K.4.3. RPC Configurations</h3></div></div></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a name="d4029e22545"></a>K.4.3.1. CellBlock Codecs</h4></div></div></div><p>To enable a codec other than the default <code class="classname">KeyValueCodec</code>, + set <code class="varname">hbase.client.rpc.codec</code> to the name of the Codec class to + use. Codec must implement hbase's <code class="classname">Codec</code> Interface. After + connection setup, all passed cellblocks will be sent with this codec. The server + will return cellblocks using this same codec as long as the codec is on the + servers' CLASSPATH (else you will get + <code class="classname">UnsupportedCellCodecException</code>).</p><p>To change the default codec, set + <code class="varname">hbase.client.default.rpc.codec</code>. </p><p>To disable cellblocks completely and to go pure protobuf, set the default to + the empty String and do not specify a codec in your Configuration. So, set + <code class="varname">hbase.client.default.rpc.codec</code> to the empty string and do + not set <code class="varname">hbase.client.rpc.codec</code>. This will cause the client to + connect to the server with no codec specified. If a server sees no codec, it + will return all responses in pure protobuf. Running pure protobuf all the time + will be slower than running with cellblocks. </p></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a name="d4029e22575"></a>K.4.3.2. Compression</h4></div></div></div><p>Uses hadoops compression codecs. To enable compressing of passed CellBlocks, + set <code class="varname">hbase.client.rpc.compressor</code> to the name of the Compressor + to use. Compressor must implement Hadoops' CompressionCodec Interface. After + connection setup, all passed cellblocks will be sent compressed. The server will + return cellblocks compressed using this same compressor as long as the + compressor is on its CLASSPATH (else you will get + <code class="classname">UnsupportedCompressionCodecException</code>).</p></div></div></div><div id="disqus_thread"></div><script type="text/javascript"> + /* * * DON'T EDIT BELOW THIS LINE * * */ + (function() { + var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true; + dsq.src = 'http://' + disqus_shortname + '.disqus.com/embed.js'; + (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq); + })(); +</script><noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript";>comments powered by Disqus.</a></noscript><a href="http://disqus.com"; class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="apks03.html">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="hbase.rpc.html">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="book_index.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">K.3. RPC </td><td width="20%" align="center"><a accesskey="h" href="book.html">Home</a></td><td width="40%" align="right" valign="top"> Index</td></tr></table></div></body></html> \ No newline at end of file Added: hbase/hbase.apache.org/trunk/book/appendix_contributing_to_documentation.html URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/appendix_contributing_to_documentation.html?rev=1616896&view=auto ============================================================================== --- hbase/hbase.apache.org/trunk/book/appendix_contributing_to_documentation.html (added) +++ hbase/hbase.apache.org/trunk/book/appendix_contributing_to_documentation.html Fri Aug 8 22:19:16 2014 @@ -0,0 +1,24 @@ +<html><head> + <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> + <title>Appendix A. Contributing to Documentation</title><link rel="stylesheet" type="text/css" href="${baserdir}/src/main/site/resources/css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL Stylesheets V1.78.1"><link rel="home" href="appendix_contributing_to_documentation.html" title="Appendix A. Contributing to Documentation"><link rel="next" href="apas02.html" title="A.2. Contributing to Documentation or Other Strings"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">Appendix A. Contributing to Documentation</th></tr><tr><td width="20%" align="left"> </td><th width="60%" align="center"> </th><td width="20%" align="right"> <a accesskey="n" href="apas02.html">Next</a></td></tr></table><hr></div><script type="text/javascript"> + var disqus_shortname = 'hbase'; // required: replace example with your forum shortname + var disqus_url = 'http://hbase.apache.org/book/appendix_contributing_to_documentation.html'; + </script><div class="appendix"><div class="titlepage"><div><div><h2 class="title"><a name="appendix_contributing_to_documentation"></a>Appendix A. Contributing to Documentation</h2></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl class="toc"><dt><span class="section"><a href="appendix_contributing_to_documentation.html#d9580e47">A.1. Getting Access to the Wiki</a></span></dt><dt><span class="section"><a href="apas02.html">A.2. Contributing to Documentation or Other Strings</a></span></dt><dt><span class="section"><a href="apas03.html">A.3. Editing the HBase Website</a></span></dt><dt><span class="section"><a href="apas04.html">A.4. Editing the HBase Reference Guide</a></span></dt><dt><span class="section"><a href="apas05.html">A.5. Auto-Generated Content</a></span></dt><dt><span class="section"><a href="apas06.html">A.6. Multi-Page and Single-Page Output</a></span></dt><dt><span class="section"><a href="apas07.html">A.7. Images in the HBase Referen ce Guide</a></span></dt><dt><span class="section"><a href="apas08.html">A.8. Adding a New Chapter to the HBase Reference Guide</a></span></dt><dt><span class="section"><a href="apas09.html">A.9. Docbook Common Issues</a></span></dt></dl></div><p>The Apache HBase project welcomes contributions to all aspects of the project, including + the documentation. In HBase, documentation includes the following areas, and probably some + others:</p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p>The <a class="link" href="http://hbase.apache.org/book.html"; target="_top">HBase Reference + Guide</a> (this book)</p></li><li class="listitem"><p>The <a class="link" href="http://hbase.apache.org/"; target="_top">HBase website</a>e</p></li><li class="listitem"><p>The <a class="link" href="http://wiki.apache.org/hadoop/Hbase"; target="_top">HBase + Wiki</a></p></li><li class="listitem"><p>API documentation</p></li><li class="listitem"><p>Command-line utility output and help text</p></li><li class="listitem"><p>Web UI strings, explicit help text, context-sensitive strings, and others</p></li><li class="listitem"><p>Log messages</p></li><li class="listitem"><p>Comments in source files, configuration files, and others</p></li><li class="listitem"><p>Localization of any of the above into target languages other than English</p></li></ul></div><p>No matter which area you want to help out with, the first step is almost always to + download (typically by cloning the Git repository) and familiarize yourself with the HBase + source code. The only exception in the list above is the HBase Wiki, which is edited online. + For information on downloading and building the source, see <a class="xref" href="">???</a>.</p><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d9580e47"></a>A.1. Getting Access to the Wiki</h2></div></div></div><p>The HBase Wiki is not well-maintained and much of its content has been moved into the + HBase Reference Guide (this guide). However, some pages on the Wiki are well maintained, + and it would be great to have some volunteers willing to help out with the Wiki. To + request access to the Wiki, register a new account at <a class="link" href="https://wiki.apache.org/hadoop/Hbase?action=newaccount"; target="_top">https://wiki.apache.org/hadoop/Hbase?action=newaccount</a>. Contact one of the + HBase committers, who can either give you access or refer you to someone who can.</p></div></div><div id="disqus_thread"></div><script type="text/javascript"> + /* * * DON'T EDIT BELOW THIS LINE * * */ + (function() { + var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true; + dsq.src = 'http://' + disqus_shortname + '.disqus.com/embed.js'; + (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq); + })(); +</script><noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript";>comments powered by Disqus.</a></noscript><a href="http://disqus.com"; class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"> </td><td width="20%" align="center"> </td><td width="40%" align="right"> <a accesskey="n" href="apas02.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top"> </td><td width="20%" align="center"> </td><td width="40%" align="right" valign="top"> A.2. Contributing to Documentation or Other Strings</td></tr></table></div></body></html> \ No newline at end of file Added: hbase/hbase.apache.org/trunk/book/ch01s14.html URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/ch01s14.html?rev=1616896&view=auto ============================================================================== --- hbase/hbase.apache.org/trunk/book/ch01s14.html (added) +++ hbase/hbase.apache.org/trunk/book/ch01s14.html Fri Aug 8 22:19:16 2014 @@ -0,0 +1,85 @@ +<html><head> + <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> + <title>1.14. HBase and HDFS</title><link rel="stylesheet" type="text/css" href="${baserdir}/src/main/site/resources/css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL Stylesheets V1.78.1"><link rel="home" href="troubleshooting.html" title="Chapter 1. Troubleshooting and Debugging Apache HBase"><link rel="up" href="troubleshooting.html" title="Chapter 1. Troubleshooting and Debugging Apache HBase"><link rel="prev" href="trouble.versions.html" title="1.13. HBase and Hadoop version issues"><link rel="next" href="trouble.tests.html" title="1.15. Running unit or integration tests"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">1.14. HBase and HDFS</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="trouble.versions.html">Prev</a> </td><th width="60%" align="center" > </th><td width="20%" align="right"> <a accesskey="n" >href="trouble.tests.html">Next</a></td></tr></table><hr></div><script >type="text/javascript"> + var disqus_shortname = 'hbase'; // required: replace example with your forum shortname + var disqus_url = 'http://hbase.apache.org/book/.html'; + </script><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d226e1050"></a>1.14. HBase and HDFS</h2></div></div></div><p>General configuration guidance for Apache HDFS is out of the scope of this guide. Refer to + the documentation available at <a class="link" href="http://hadoop.apache.org/"; target="_top">http://hadoop.apache.org/</a> for extensive + information about configuring HDFS. This section deals with HDFS in terms of HBase. </p><p>In most cases, HBase stores its data in Apache HDFS. This includes the HFiles containing + the data, as well as the write-ahead logs (WALs) which store data before it is written to the + HFiles and protect against RegionServer crashes. HDFS provides reliability and protection to + data in HBase because it is distributed. To operate with the most efficiency, HBase needs data + to be available locally. Therefore, it is a good practice to run an HDFS datanode on each + RegionServer.</p><div class="variablelist"><p class="title"><b>Important Information and Guidelines for HBase and HDFS</b></p><dl class="variablelist"><dt><span class="term">HBase is a client of HDFS.</span></dt><dd><p>HBase is an HDFS client, using the HDFS <code class="code">DFSClient</code> class, and references + to this class appear in HBase logs with other HDFS client log messages.</p></dd><dt><span class="term">Configuration is necessary in multiple places.</span></dt><dd><p>Some HDFS configurations relating to HBase need to be done at the HDFS (server) side. + Others must be done within HBase (at the client side). Other settings need + to be set at both the server and client side. + </p></dd><dt><span class="term">Write errors which affect HBase may be logged in the HDFS logs rather than HBase logs.</span></dt><dd><p>When writing, HDFS pipelines communications from one datanode to another. HBase + communicates to both the HDFS namenode and datanode, using the HDFS client classes. + Communication problems between datanodes are logged in the HDFS logs, not the HBase + logs.</p><p>HDFS writes are always local when possible. HBase RegionServers should not + experience many write errors, because they write the local datanode. If the datanode + cannot replicate the blocks, the errors are logged in HDFS, not in the HBase + RegionServer logs.</p></dd><dt><span class="term">HBase communicates with HDFS using two different ports.</span></dt><dd><p>HBase communicates with datanodes using the <code class="code">ipc.Client</code> interface and + the <code class="code">DataNode</code> class. References to these will appear in HBase logs. Each of + these communication channels use a different port (50010 and 50020 by default). The + ports are configured in the HDFS configuration, via the + <code class="code">dfs.datanode.address</code> and <code class="code">dfs.datanode.ipc.address</code> + parameters.</p></dd><dt><span class="term">Errors may be logged in HBase, HDFS, or both.</span></dt><dd><p>When troubleshooting HDFS issues in HBase, check logs in both places for errors.</p></dd><dt><span class="term">HDFS takes a while to mark a node as dead. You can configure HDFS to avoid using stale + datanodes.</span></dt><dd><p>By default, HDFS does not mark a node as dead until it is unreachable for 630 + seconds. In Hadoop 1.1 and Hadoop 2.x, this can be alleviated by enabling checks for + stale datanodes, though this check is disabled by default. You can enable the check for + reads and writes separately, via <code class="code">dfs.namenode.avoid.read.stale.datanode</code> and + <code class="code">dfs.namenode.avoid.write.stale.datanode settings</code>. A stale datanode is one + that has not been reachable for <code class="code">dfs.namenode.stale.datanode.interval</code> + (default is 30 seconds). Stale datanodes are avoided, and marked as the last possible + target for a read or write operation. For configuration details, see the HDFS + documentation.</p></dd><dt><span class="term">Settings for HDFS retries and timeouts are important to HBase.</span></dt><dd><p>You can configure settings for various retries and timeouts. Always refer to the + HDFS documentation for current recommendations and defaults. Some of the settings + important to HBase are listed here. Defaults are current as of Hadoop 2.3. Check the + Hadoop documentation for the most current values and recommendations.</p><div class="variablelist"><p class="title"><b>Retries</b></p><dl class="variablelist"><dt><span class="term"><code class="code">ipc.client.connect.max.retries</code> (default: 10)</span></dt><dd><p>The number of times a client will attempt to establish a connection with the + server. This value sometimes needs to be increased. You can specify different + setting for the maximum number of retries if a timeout occurs. For SASL + connections, the number of retries is hard-coded at 15 and cannot be + configured.</p></dd><dt><span class="term"><code class="code">ipc.client.connect.max.retries.on.timeouts</code> (default: 45)</span></dt><dd><p>The number of times a client will attempt to establish a connection + with the server in the event of a timeout. If some retries are due to timeouts and + some are due to other reasons, this counter is added to + <code class="code">ipc.client.connect.max.retries</code>, so the maximum number of retries for + all reasons could be the combined value.</p></dd><dt><span class="term"><code class="code">dfs.client.block.write.retries</code> (default: 3)</span></dt><dd><p>How many times the client attempts to write to the datanode. After the + number of retries is reached, the client reconnects to the namenode to get a new + location of a datanode. You can try increasing this value.</p></dd></dl></div><div class="variablelist"><p class="title"><b>HDFS Heartbeats</b></p><p>HDFS heartbeats are entirely on the HDFS side, between the namenode and datanodes.</p><dl class="variablelist"><dt><span class="term"><code class="code">dfs.heartbeat.interval</code> (default: 3)</span></dt><dd><p>The interval at which a node heartbeats.</p></dd><dt><span class="term"><code class="code">dfs.namenode.heartbeat.recheck-interval</code> (default: 300000)</span></dt><dd><p>The interval of time between heartbeat checks. The total time before a node is + marked as stale is determined by the following formula, which works out to 10 + minutes and 30 seconds:</p><pre class="screen"> 2 * (dfs.namenode.heartbeat.recheck-interval) + 10 * 1000 * (dfs.heartbeat.interval)</pre></dd><dt><span class="term"><code class="code">dfs.namenode.stale.datanode.interval</code> (default: 3000)</span></dt><dd><p>How long (in milliseconds) a node can go without a heartbeat before it is + determined to be stale, if the other options to do with stale datanodes are + configured (off by default).</p></dd></dl></div></dd></dl></div><div class="variablelist"><p class="title"><b>Connection Timeouts</b></p><p>Connection timeouts occur between the client (HBASE) and the HDFS datanode. They may + occur when establishing a connection, attempting to read, or attempting to write. The two + settings below are used in combination, and affect connections between the DFSClient and the + datanode, the ipc.cClient and the datanode, and communication between two datanodes. </p><dl class="variablelist"><dt><span class="term"><code class="code">dfs.client.socket-timeout</code> (default: 60000)</span></dt><dd><p>The amount of time before a client connection times out when establishing a + connection or reading. The value is expressed in milliseconds, so the default is 60 + seconds.</p></dd><dt><span class="term"><code class="code">dfs.datanode.socket.write.timeout</code> (default: 480000)</span></dt><dd><p>The amount of time before a write operation times out. The default is 8 + minutes, expressed as milliseconds.</p></dd></dl></div><div class="variablelist"><p class="title"><b>Typical Error Logs</b></p><p>The following types of errors are often seen in the logs.</p><dl class="variablelist"><dt><span class="term"><code class="code">INFO HDFS.DFSClient: Failed to connect to /xxx50010, add to deadNodes and + continue java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel + to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending + remote=/region-server-1:50010]</code></span></dt><dd><p>All datanodes for a block are dead, and recovery is not possible. Here is the + sequence of events that leads to this error:</p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p>The client attempts to connect to a dead datanode.</p></li><li class="listitem"><p>The connection fails, so the client moves down the list of datanodes and tries + the next one. It also fails.</p></li><li class="listitem"><p>When the client exhausts its entire list, it sleeps for 3 seconds and requests a + new list. It is very likely to receive the exact same list as before, in which case + the error occurs again.</p></li></ul></div></dd><dt><span class="term"><code class="code">INFO org.apache.hadoop.HDFS.DFSClient: Exception in createBlockOutputStream + java.net.SocketTimeoutException: 69000 millis timeout while waiting for channel to be + ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/ + xxx:50010]</code></span></dt><dd><p>This type of error indicates a write issue. In this case, the master wants to split + the log. It does not have a local datanode so it tries to connect to a remote datanode, + but the datanode is dead.</p><p>In this situation, there will be three retries (by default). If all retries fail, a + message like the following is logged:</p><pre class="screen"> +WARN HDFS.DFSClient: DataStreamer Exception: java.io.IOException: Unable to create new block + </pre><p>If the operation was an attempt to split the log, the following type of message may + also appear:</p><pre class="screen"> +FATAL wal.HLogSplitter: WriterThread-xxx Got while writing log entry to log + </pre></dd></dl></div></div><div id="disqus_thread"></div><script type="text/javascript"> + /* * * DON'T EDIT BELOW THIS LINE * * */ + (function() { + var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true; + dsq.src = 'http://' + disqus_shortname + '.disqus.com/embed.js'; + (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq); + })(); +</script><noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript";>comments powered by Disqus.</a></noscript><a href="http://disqus.com"; class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="trouble.versions.html">Prev</a> </td><td width="20%" align="center"> </td><td width="40%" align="right"> <a accesskey="n" href="trouble.tests.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">1.13. HBase and Hadoop version issues </td><td width="20%" align="center"><a accesskey="h" href="troubleshooting.html">Home</a></td><td width="40%" align="right" valign="top"> 1.15. Running unit or integration tests</td></tr></table></div></body></html> \ No newline at end of file Added: hbase/hbase.apache.org/trunk/book/ch01s18.html URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/ch01s18.html?rev=1616896&view=auto ============================================================================== --- hbase/hbase.apache.org/trunk/book/ch01s18.html (added) +++ hbase/hbase.apache.org/trunk/book/ch01s18.html Fri Aug 8 22:19:16 2014 @@ -0,0 +1,19 @@ +<html><head> + <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> + <title>1.18. Operating System Specific Issues</title><link rel="stylesheet" type="text/css" href="${baserdir}/src/main/site/resources/css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL Stylesheets V1.78.1"><link rel="home" href="troubleshooting.html" title="Chapter 1. Troubleshooting and Debugging Apache HBase"><link rel="up" href="troubleshooting.html" title="Chapter 1. Troubleshooting and Debugging Apache HBase"><link rel="prev" href="trouble.crypto.html" title="1.17. Cryptographic Features"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">1.18. Operating System Specific Issues</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="trouble.crypto.html">Prev</a> </td><th width="60%" align="center"> </th><td width="20%" align="right"> </td></tr></table><hr></d iv><script type="text/javascript"> + var disqus_shortname = 'hbase'; // required: replace example with your forum shortname + var disqus_url = 'http://hbase.apache.org/book/.html'; + </script><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d226e1300"></a>1.18. Operating System Specific Issues</h2></div></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a name="d226e1303"></a>1.18.1. Page Allocation Failure</h3></div></div></div><div class="note" style="margin-left: 0.5in; margin-right: 0.5in;"><h3 class="title">Note</h3><p>This issue is known to affect CentOS 6.2 and possibly CentOS 6.5. It may also affect + some versions of Red Hat Enterprise Linux, according to <a class="link" href="https://bugzilla.redhat.com/show_bug.cgi?id=770545"; target="_top">https://bugzilla.redhat.com/show_bug.cgi?id=770545</a>.</p></div><p>Some users have reported seeing the following error:</p><pre class="screen">kernel: java: page allocation failure. order:4, mode:0x20</pre><p>Raising the value of <code class="code">min_free_kbytes</code> was reported to fix this problem. This + parameter is set to a percentage of the amount of RAM on your system, and is described in more + detail at <a class="link" href="http://www.centos.org/docs/5/html/5.1/Deployment_Guide/s3-proc-sys-vm.html"; target="_top">http://www.centos.org/docs/5/html/5.1/Deployment_Guide/s3-proc-sys-vm.html</a>. </p><p>To find the current value on your system, run the following command:</p><pre class="screen">[user@host]<em class="hl-comment" style="color: silver"># </em><strong class="userinput"><code>cat /proc/sys/vm/min_free_kbytes</code></strong></pre><p>Next, raise the value. Try doubling, then quadrupling the value. Note that setting the + value too low or too high could have detrimental effects on your system. Consult your + operating system vendor for specific recommendations.</p><p>Use the following command to modify the value of <code class="code">min_free_kbytes</code>, + substituting <em class="replaceable"><code><value></code></em> with your intended value:</p><pre class="screen">[user@host]<em class="hl-comment" style="color: silver"># </em><strong class="userinput"><code>echo <value> > /proc/sys/vm/min_free_kbytes</code></strong></pre></div></div><div id="disqus_thread"></div><script type="text/javascript"> + /* * * DON'T EDIT BELOW THIS LINE * * */ + (function() { + var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true; + dsq.src = 'http://' + disqus_shortname + '.disqus.com/embed.js'; + (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq); + })(); +</script><noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript";>comments powered by Disqus.</a></noscript><a href="http://disqus.com"; class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="trouble.crypto.html">Prev</a> </td><td width="20%" align="center"> </td><td width="40%" align="right"> </td></tr><tr><td width="40%" align="left" valign="top">1.17. Cryptographic Features </td><td width="20%" align="center"><a accesskey="h" href="troubleshooting.html">Home</a></td><td width="40%" align="right" valign="top"> </td></tr></table></div></body></html> \ No newline at end of file Added: hbase/hbase.apache.org/trunk/book/ch15s14.html URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/ch15s14.html?rev=1616896&view=auto ============================================================================== --- hbase/hbase.apache.org/trunk/book/ch15s14.html (added) +++ hbase/hbase.apache.org/trunk/book/ch15s14.html Fri Aug 8 22:19:16 2014 @@ -0,0 +1,85 @@ +<html><head> + <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> + <title>15.14. HBase and HDFS</title><link rel="stylesheet" type="text/css" href="${baserdir}/src/main/site/resources/css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL Stylesheets V1.78.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="trouble.html" title="Chapter 15. Troubleshooting and Debugging Apache HBase"><link rel="prev" href="trouble.versions.html" title="15.13. HBase and Hadoop version issues"><link rel="next" href="trouble.tests.html" title="15.15. Running unit or integration tests"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">15.14. HBase and HDFS</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="trouble.versions.html">Prev</a> </td><th width="60%" align="center">Chapter 15. Troubleshooting and Debugging Apache HBase</th><td width="20%" align="right"> <a accesskey="n" href="trouble.tests.html">Next</a></td></tr></table><hr></div><script type="text/javascript"> + var disqus_shortname = 'hbase'; // required: replace example with your forum shortname + var disqus_url = 'http://hbase.apache.org/book/.html'; + </script><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d4029e15588"></a>15.14. HBase and HDFS</h2></div></div></div><p>General configuration guidance for Apache HDFS is out of the scope of this guide. Refer to + the documentation available at <a class="link" href="http://hadoop.apache.org/"; target="_top">http://hadoop.apache.org/</a> for extensive + information about configuring HDFS. This section deals with HDFS in terms of HBase. </p><p>In most cases, HBase stores its data in Apache HDFS. This includes the HFiles containing + the data, as well as the write-ahead logs (WALs) which store data before it is written to the + HFiles and protect against RegionServer crashes. HDFS provides reliability and protection to + data in HBase because it is distributed. To operate with the most efficiency, HBase needs data + to be available locally. Therefore, it is a good practice to run an HDFS datanode on each + RegionServer.</p><div class="variablelist"><p class="title"><b>Important Information and Guidelines for HBase and HDFS</b></p><dl class="variablelist"><dt><span class="term">HBase is a client of HDFS.</span></dt><dd><p>HBase is an HDFS client, using the HDFS <code class="code">DFSClient</code> class, and references + to this class appear in HBase logs with other HDFS client log messages.</p></dd><dt><span class="term">Configuration is necessary in multiple places.</span></dt><dd><p>Some HDFS configurations relating to HBase need to be done at the HDFS (server) side. + Others must be done within HBase (at the client side). Other settings need + to be set at both the server and client side. + </p></dd><dt><span class="term">Write errors which affect HBase may be logged in the HDFS logs rather than HBase logs.</span></dt><dd><p>When writing, HDFS pipelines communications from one datanode to another. HBase + communicates to both the HDFS namenode and datanode, using the HDFS client classes. + Communication problems between datanodes are logged in the HDFS logs, not the HBase + logs.</p><p>HDFS writes are always local when possible. HBase RegionServers should not + experience many write errors, because they write the local datanode. If the datanode + cannot replicate the blocks, the errors are logged in HDFS, not in the HBase + RegionServer logs.</p></dd><dt><span class="term">HBase communicates with HDFS using two different ports.</span></dt><dd><p>HBase communicates with datanodes using the <code class="code">ipc.Client</code> interface and + the <code class="code">DataNode</code> class. References to these will appear in HBase logs. Each of + these communication channels use a different port (50010 and 50020 by default). The + ports are configured in the HDFS configuration, via the + <code class="code">dfs.datanode.address</code> and <code class="code">dfs.datanode.ipc.address</code> + parameters.</p></dd><dt><span class="term">Errors may be logged in HBase, HDFS, or both.</span></dt><dd><p>When troubleshooting HDFS issues in HBase, check logs in both places for errors.</p></dd><dt><span class="term">HDFS takes a while to mark a node as dead. You can configure HDFS to avoid using stale + datanodes.</span></dt><dd><p>By default, HDFS does not mark a node as dead until it is unreachable for 630 + seconds. In Hadoop 1.1 and Hadoop 2.x, this can be alleviated by enabling checks for + stale datanodes, though this check is disabled by default. You can enable the check for + reads and writes separately, via <code class="code">dfs.namenode.avoid.read.stale.datanode</code> and + <code class="code">dfs.namenode.avoid.write.stale.datanode settings</code>. A stale datanode is one + that has not been reachable for <code class="code">dfs.namenode.stale.datanode.interval</code> + (default is 30 seconds). Stale datanodes are avoided, and marked as the last possible + target for a read or write operation. For configuration details, see the HDFS + documentation.</p></dd><dt><span class="term">Settings for HDFS retries and timeouts are important to HBase.</span></dt><dd><p>You can configure settings for various retries and timeouts. Always refer to the + HDFS documentation for current recommendations and defaults. Some of the settings + important to HBase are listed here. Defaults are current as of Hadoop 2.3. Check the + Hadoop documentation for the most current values and recommendations.</p><div class="variablelist"><p class="title"><b>Retries</b></p><dl class="variablelist"><dt><span class="term"><code class="code">ipc.client.connect.max.retries</code> (default: 10)</span></dt><dd><p>The number of times a client will attempt to establish a connection with the + server. This value sometimes needs to be increased. You can specify different + setting for the maximum number of retries if a timeout occurs. For SASL + connections, the number of retries is hard-coded at 15 and cannot be + configured.</p></dd><dt><span class="term"><code class="code">ipc.client.connect.max.retries.on.timeouts</code> (default: 45)</span></dt><dd><p>The number of times a client will attempt to establish a connection + with the server in the event of a timeout. If some retries are due to timeouts and + some are due to other reasons, this counter is added to + <code class="code">ipc.client.connect.max.retries</code>, so the maximum number of retries for + all reasons could be the combined value.</p></dd><dt><span class="term"><code class="code">dfs.client.block.write.retries</code> (default: 3)</span></dt><dd><p>How many times the client attempts to write to the datanode. After the + number of retries is reached, the client reconnects to the namenode to get a new + location of a datanode. You can try increasing this value.</p></dd></dl></div><div class="variablelist"><p class="title"><b>HDFS Heartbeats</b></p><p>HDFS heartbeats are entirely on the HDFS side, between the namenode and datanodes.</p><dl class="variablelist"><dt><span class="term"><code class="code">dfs.heartbeat.interval</code> (default: 3)</span></dt><dd><p>The interval at which a node heartbeats.</p></dd><dt><span class="term"><code class="code">dfs.namenode.heartbeat.recheck-interval</code> (default: 300000)</span></dt><dd><p>The interval of time between heartbeat checks. The total time before a node is + marked as stale is determined by the following formula, which works out to 10 + minutes and 30 seconds:</p><pre class="screen"> 2 * (dfs.namenode.heartbeat.recheck-interval) + 10 * 1000 * (dfs.heartbeat.interval)</pre></dd><dt><span class="term"><code class="code">dfs.namenode.stale.datanode.interval</code> (default: 3000)</span></dt><dd><p>How long (in milliseconds) a node can go without a heartbeat before it is + determined to be stale, if the other options to do with stale datanodes are + configured (off by default).</p></dd></dl></div></dd></dl></div><div class="variablelist"><p class="title"><b>Connection Timeouts</b></p><p>Connection timeouts occur between the client (HBASE) and the HDFS datanode. They may + occur when establishing a connection, attempting to read, or attempting to write. The two + settings below are used in combination, and affect connections between the DFSClient and the + datanode, the ipc.cClient and the datanode, and communication between two datanodes. </p><dl class="variablelist"><dt><span class="term"><code class="code">dfs.client.socket-timeout</code> (default: 60000)</span></dt><dd><p>The amount of time before a client connection times out when establishing a + connection or reading. The value is expressed in milliseconds, so the default is 60 + seconds.</p></dd><dt><span class="term"><code class="code">dfs.datanode.socket.write.timeout</code> (default: 480000)</span></dt><dd><p>The amount of time before a write operation times out. The default is 8 + minutes, expressed as milliseconds.</p></dd></dl></div><div class="variablelist"><p class="title"><b>Typical Error Logs</b></p><p>The following types of errors are often seen in the logs.</p><dl class="variablelist"><dt><span class="term"><code class="code">INFO HDFS.DFSClient: Failed to connect to /xxx50010, add to deadNodes and + continue java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel + to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending + remote=/region-server-1:50010]</code></span></dt><dd><p>All datanodes for a block are dead, and recovery is not possible. Here is the + sequence of events that leads to this error:</p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p>The client attempts to connect to a dead datanode.</p></li><li class="listitem"><p>The connection fails, so the client moves down the list of datanodes and tries + the next one. It also fails.</p></li><li class="listitem"><p>When the client exhausts its entire list, it sleeps for 3 seconds and requests a + new list. It is very likely to receive the exact same list as before, in which case + the error occurs again.</p></li></ul></div></dd><dt><span class="term"><code class="code">INFO org.apache.hadoop.HDFS.DFSClient: Exception in createBlockOutputStream + java.net.SocketTimeoutException: 69000 millis timeout while waiting for channel to be + ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/ + xxx:50010]</code></span></dt><dd><p>This type of error indicates a write issue. In this case, the master wants to split + the log. It does not have a local datanode so it tries to connect to a remote datanode, + but the datanode is dead.</p><p>In this situation, there will be three retries (by default). If all retries fail, a + message like the following is logged:</p><pre class="screen"> +WARN HDFS.DFSClient: DataStreamer Exception: java.io.IOException: Unable to create new block + </pre><p>If the operation was an attempt to split the log, the following type of message may + also appear:</p><pre class="screen"> +FATAL wal.HLogSplitter: WriterThread-xxx Got while writing log entry to log + </pre></dd></dl></div></div><div id="disqus_thread"></div><script type="text/javascript"> + /* * * DON'T EDIT BELOW THIS LINE * * */ + (function() { + var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true; + dsq.src = 'http://' + disqus_shortname + '.disqus.com/embed.js'; + (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq); + })(); +</script><noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript";>comments powered by Disqus.</a></noscript><a href="http://disqus.com"; class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="trouble.versions.html">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="trouble.html">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="trouble.tests.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">15.13. HBase and Hadoop version issues </td><td width="20%" align="center"><a accesskey="h" href="book.html">Home</a></td><td width="40%" align="right" valign="top"> 15.15. Running unit or integration tests</td></tr></table></div></body></html> \ No newline at end of file Added: hbase/hbase.apache.org/trunk/book/ch15s18.html URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/ch15s18.html?rev=1616896&view=auto ============================================================================== --- hbase/hbase.apache.org/trunk/book/ch15s18.html (added) +++ hbase/hbase.apache.org/trunk/book/ch15s18.html Fri Aug 8 22:19:16 2014 @@ -0,0 +1,19 @@ +<html><head> + <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> + <title>15.18. Operating System Specific Issues</title><link rel="stylesheet" type="text/css" href="${baserdir}/src/main/site/resources/css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL Stylesheets V1.78.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="trouble.html" title="Chapter 15. Troubleshooting and Debugging Apache HBase"><link rel="prev" href="trouble.crypto.html" title="15.17. Cryptographic Features"><link rel="next" href="casestudies.html" title="Chapter 16. Apache HBase Case Studies"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">15.18. Operating System Specific Issues</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="trouble.crypto.html">Prev</a> </td><th width="60%" align="center">Chapter 15 . Troubleshooting and Debugging Apache HBase</th><td width="20%" align="right"> <a accesskey="n" href="casestudies.html">Next</a></td></tr></table><hr></div><script type="text/javascript"> + var disqus_shortname = 'hbase'; // required: replace example with your forum shortname + var disqus_url = 'http://hbase.apache.org/book/.html'; + </script><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d4029e15838"></a>15.18. Operating System Specific Issues</h2></div></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a name="d4029e15841"></a>15.18.1. Page Allocation Failure</h3></div></div></div><div class="note" style="margin-left: 0.5in; margin-right: 0.5in;"><h3 class="title">Note</h3><p>This issue is known to affect CentOS 6.2 and possibly CentOS 6.5. It may also affect + some versions of Red Hat Enterprise Linux, according to <a class="link" href="https://bugzilla.redhat.com/show_bug.cgi?id=770545"; target="_top">https://bugzilla.redhat.com/show_bug.cgi?id=770545</a>.</p></div><p>Some users have reported seeing the following error:</p><pre class="screen">kernel: java: page allocation failure. order:4, mode:0x20</pre><p>Raising the value of <code class="code">min_free_kbytes</code> was reported to fix this problem. This + parameter is set to a percentage of the amount of RAM on your system, and is described in more + detail at <a class="link" href="http://www.centos.org/docs/5/html/5.1/Deployment_Guide/s3-proc-sys-vm.html"; target="_top">http://www.centos.org/docs/5/html/5.1/Deployment_Guide/s3-proc-sys-vm.html</a>. </p><p>To find the current value on your system, run the following command:</p><pre class="screen">[user@host]<em class="hl-comment" style="color: silver"># </em><strong class="userinput"><code>cat /proc/sys/vm/min_free_kbytes</code></strong></pre><p>Next, raise the value. Try doubling, then quadrupling the value. Note that setting the + value too low or too high could have detrimental effects on your system. Consult your + operating system vendor for specific recommendations.</p><p>Use the following command to modify the value of <code class="code">min_free_kbytes</code>, + substituting <em class="replaceable"><code><value></code></em> with your intended value:</p><pre class="screen">[user@host]<em class="hl-comment" style="color: silver"># </em><strong class="userinput"><code>echo <value> > /proc/sys/vm/min_free_kbytes</code></strong></pre></div></div><div id="disqus_thread"></div><script type="text/javascript"> + /* * * DON'T EDIT BELOW THIS LINE * * */ + (function() { + var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true; + dsq.src = 'http://' + disqus_shortname + '.disqus.com/embed.js'; + (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq); + })(); +</script><noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript";>comments powered by Disqus.</a></noscript><a href="http://disqus.com"; class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="trouble.crypto.html">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="trouble.html">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="casestudies.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">15.17. Cryptographic Features </td><td width="20%" align="center"><a accesskey="h" href="book.html">Home</a></td><td width="40%" align="right" valign="top"> Chapter 16. Apache HBase Case Studies</td></tr></table></div></body></html> \ No newline at end of file Added: hbase/hbase.apache.org/trunk/book/ld-d2012e1089.html URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/ld-d2012e1089.html?rev=1616896&view=auto ============================================================================== --- hbase/hbase.apache.org/trunk/book/ld-d2012e1089.html (added) +++ hbase/hbase.apache.org/trunk/book/ld-d2012e1089.html Fri Aug 8 22:19:16 2014 @@ -0,0 +1,4 @@ +<html><head> + <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> + <title>Long Description</title><link rel="stylesheet" type="text/css" href="${baserdir}/src/main/site/resources/css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL Stylesheets V1.78.1"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><p>Illustration of the replication architecture in HBase, as described in the prior + text.</p></body></html> \ No newline at end of file Added: hbase/hbase.apache.org/trunk/book/ld-d4029e17132.html URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/ld-d4029e17132.html?rev=1616896&view=auto ============================================================================== --- hbase/hbase.apache.org/trunk/book/ld-d4029e17132.html (added) +++ hbase/hbase.apache.org/trunk/book/ld-d4029e17132.html Fri Aug 8 22:19:16 2014 @@ -0,0 +1,4 @@ +<html><head> + <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> + <title>Long Description</title><link rel="stylesheet" type="text/css" href="${baserdir}/src/main/site/resources/css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL Stylesheets V1.78.1"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><p>Illustration of the replication architecture in HBase, as described in the prior + text.</p></body></html> \ No newline at end of file Added: hbase/hbase.apache.org/trunk/book/ld-d4029e21286.html URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/ld-d4029e21286.html?rev=1616896&view=auto ============================================================================== --- hbase/hbase.apache.org/trunk/book/ld-d4029e21286.html (added) +++ hbase/hbase.apache.org/trunk/book/ld-d4029e21286.html Fri Aug 8 22:19:16 2014 @@ -0,0 +1,3 @@ +<html><head> + <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> + <title>Long Description</title><link rel="stylesheet" type="text/css" href="${baserdir}/src/main/site/resources/css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL Stylesheets V1.78.1"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><p></p></body></html> \ No newline at end of file Added: hbase/hbase.apache.org/trunk/book/ld-d4029e21296.html URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/ld-d4029e21296.html?rev=1616896&view=auto ============================================================================== --- hbase/hbase.apache.org/trunk/book/ld-d4029e21296.html (added) +++ hbase/hbase.apache.org/trunk/book/ld-d4029e21296.html Fri Aug 8 22:19:16 2014 @@ -0,0 +1,3 @@ +<html><head> + <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> + <title>Long Description</title><link rel="stylesheet" type="text/css" href="${baserdir}/src/main/site/resources/css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL Stylesheets V1.78.1"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><p></p></body></html> \ No newline at end of file Added: hbase/hbase.apache.org/trunk/book/ld-d4029e21311.html URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/ld-d4029e21311.html?rev=1616896&view=auto ============================================================================== --- hbase/hbase.apache.org/trunk/book/ld-d4029e21311.html (added) +++ hbase/hbase.apache.org/trunk/book/ld-d4029e21311.html Fri Aug 8 22:19:16 2014 @@ -0,0 +1,3 @@ +<html><head> + <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> + <title>Long Description</title><link rel="stylesheet" type="text/css" href="${baserdir}/src/main/site/resources/css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL Stylesheets V1.78.1"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><p></p></body></html> \ No newline at end of file Added: hbase/hbase.apache.org/trunk/book/repos.html URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/repos.html?rev=1616896&view=auto ============================================================================== --- hbase/hbase.apache.org/trunk/book/repos.html (added) +++ hbase/hbase.apache.org/trunk/book/repos.html Fri Aug 8 22:19:16 2014 @@ -0,0 +1,20 @@ +<html><head> + <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> + <title>18.2. Apache HBase Repositories</title><link rel="stylesheet" type="text/css" href="${baserdir}/src/main/site/resources/css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL Stylesheets V1.78.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="developer.html" title="Chapter 18. Building and Developing Apache HBase"><link rel="prev" href="developer.html" title="Chapter 18. Building and Developing Apache HBase"><link rel="next" href="ides.html" title="18.3. IDEs"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">18.2. Apache HBase Repositories</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="developer.html">Prev</a> </td><th width="60%" align="center">Chapter 18. Building and Developing Apache HBase</ th><td width="20%" align="right"> <a accesskey="n" href="ides.html">Next</a></td></tr></table><hr></div><script type="text/javascript"> + var disqus_shortname = 'hbase'; // required: replace example with your forum shortname + var disqus_url = 'http://hbase.apache.org/book/repos.html'; + </script><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="repos"></a>18.2. Apache HBase Repositories</h2></div></div></div><p>There are two different repositories for Apache HBase: Subversion (SVN) and Git. + GIT is our repository of record for all but the Apache HBase website. + We used to be on SVN. We migrated. See <a class="link" href="https://issues.apache.org/jira/browse/INFRA-7768"; target="_top">Migrade Apache HBase SVN Repos to Git</a>. + Updating hbase.apache.org still requires use of SVN (See <a class="xref" href="hbase.org.html" title="18.7. Updating hbase.apache.org">Section 18.7, “Updating hbase.apache.org”</a>). + See <a class="link" href="http://hbase.apache.org/source-repository.html"; target="_top">Source Code Management</a> + page for contributor and committer links or + seach for HBase on the <a class="link" href="http://git.apache.org/"; target="_top">Apache Git</a> page. + </p></div><div id="disqus_thread"></div><script type="text/javascript"> + /* * * DON'T EDIT BELOW THIS LINE * * */ + (function() { + var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true; + dsq.src = 'http://' + disqus_shortname + '.disqus.com/embed.js'; + (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq); + })(); +</script><noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript";>comments powered by Disqus.</a></noscript><a href="http://disqus.com"; class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="developer.html">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="developer.html">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="ides.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">Chapter 18. Building and Developing Apache HBase </td><td width="20%" align="center"><a accesskey="h" href="book.html">Home</a></td><td width="40%" align="right" valign="top"> 18.3. IDEs</td></tr></table></div></body></html> \ No newline at end of file
