Author: sumit
Date: Mon Dec 7 15:55:08 2015
New Revision: 1718414
URL: http://svn.apache.org/viewvc?rev=1718414&view=rev
Log:
KNOX-625 Fix page link to ui services
Added:
knox/trunk/books/0.7.0/book_ui_service_details.md
- copied unchanged from r1718407,
knox/trunk/books/0.7.0/book_ui_service-details.md
Removed:
knox/trunk/books/0.7.0/book_ui_service-details.md
Modified:
knox/site/books/knox-0-7-0/user-guide.html
knox/trunk/books/0.7.0/book.md
Modified: knox/site/books/knox-0-7-0/user-guide.html
URL:
http://svn.apache.org/viewvc/knox/site/books/knox-0-7-0/user-guide.html?rev=1718414&r1=1718413&r2=1718414&view=diff
==============================================================================
--- knox/site/books/knox-0-7-0/user-guide.html (original)
+++ knox/site/books/knox-0-7-0/user-guide.html Mon Dec 7 15:55:08 2015
@@ -4196,7 +4196,163 @@ curl -ik -b ~/cookiejar.txt -c ~/cookiej
<ul>
<li>In the first cURL request, the quotes are necessary around the URL or
else a command line terminal will not include the <code>&password</code>
query parameter in the request.</li>
<li>This API call does not require any credentials to receive a response
from Knox, but expect to receive 401 responses from each of the services if
none are provided.</li>
-</ul><h2><a id="Limitations">Limitations</a> <a href="#Limitations"><img
src="markbook-section-link.png"/></a></h2><h3><a
id="Secure+Oozie+POST/PUT+Request+Payload+Size+Restriction">Secure Oozie
POST/PUT Request Payload Size Restriction</a> <a
href="#Secure+Oozie+POST/PUT+Request+Payload+Size+Restriction"><img
src="markbook-section-link.png"/></a></h3><p>With one exception there are no
known size limits for requests or responses payloads that pass through the
gateway. The exception involves POST or PUT request payload sizes for Oozie in
a Kerberos secured Hadoop cluster. In this one case there is currently a 4Kb
payload size limit for the first request made to the Hadoop cluster. This is a
result of how the gateway negotiates a trust relationship between itself and
the cluster via SPNego. There is an undocumented configuration setting to
modify this limit’s value if required. In the future this will be made
more easily configuration and at that time it will be documented.</p><
h3><a id="Group+Membership+Propagation">Group Membership Propagation</a> <a
href="#Group+Membership+Propagation"><img
src="markbook-section-link.png"/></a></h3><p>Groups that are acquired via Shiro
Group Lookup and/or Identity Assertion Group Principal Mapping are not
propagated to the Hadoop services. Therefore, groups used for Service Level
Authorization policy may not match those acquired within the cluster via
GroupMappingServiceProvider plugins.</p><h2><a
id="Troubleshooting">Troubleshooting</a> <a href="#Troubleshooting"><img
src="markbook-section-link.png"/></a></h2><h3><a id="Finding+Logs">Finding
Logs</a> <a href="#Finding+Logs"><img
src="markbook-section-link.png"/></a></h3><p>When things aren’t working
the first thing you need to do is examine the diagnostic logs. Depending upon
how you are running the gateway these diagnostic logs will be output to
different locations.</p><h4><a id="java+-jar+bin/gateway.jar">java -jar
bin/gateway.jar</a> <a href="#java+-jar+bin/ga
teway.jar"><img src="markbook-section-link.png"/></a></h4><p>When the gateway
is run this way the diagnostic output is written directly to the console. If
you want to capture that output you will need to redirect the console output to
a file using OS specific techniques.</p>
+</ul><h2><a id="UI+Service+Details">UI Service Details</a> <a
href="#UI+Service+Details"><img src="markbook-section-link.png"/></a></h2><p>In
the sections that follow the integrations for proxying various UIs currently
available out of the box with the gateway will be described. These sections
will include examples that demonstrate how to access each of these services via
the gateway.</p><p>These are the current Hadoop services with built-in support
for their UIs.</p>
+<ul>
+ <li><a href="#Name+Node+UI">Name Node UI</a></li>
+ <li><a href="#Job+History+UI">Job History UI</a></li>
+ <li><a href="#Oozie+UI">Oozie UI</a></li>
+ <li><a href="#HBase+UI">HBase UI</a></li>
+ <li><a href="#Yarn+UI">Yarn UI</a></li>
+ <li><a href="#Spark+UI">Spark UI</a></li>
+</ul><h3><a id="Assumptions">Assumptions</a> <a href="#Assumptions"><img
src="markbook-section-link.png"/></a></h3><p>This section assumes an
environment setup similar to the one in the REST services section <a
href="#Service+Details">Service Details</a></p><h3><a id="Name+Node+UI">Name
Node UI</a> <a href="#Name+Node+UI"><img
src="markbook-section-link.png"/></a></h3><p>The Name Node UI is available on
the same host and port combination that WebHDFS is available on. As mentioned
in the WebHDFS REST service configuration section, the values for the host and
port can be obtained from the following properties in hdfs-site.xml</p>
+<pre><code><property>
+ <name>dfs.namenode.http-address</name>
+ <value>sandbox.hortonworks.com:50070</value>
+</property>
+<property>
+ <name>dfs.https.namenode.https-address</name>
+ <value>sandbox.hortonworks.com:50470</value>
+</property>
+</code></pre><p>The values above need to be reflected in each topology
descriptor file deployed to the gateway. The gateway by default includes a
sample topology descriptor file
<code>{GATEWAY_HOME}/deployments/sandbox.xml</code>. The values in this sample
are configured to work with an installed Sandbox VM.</p>
+<pre><code><service>
+ <role>HDFSUI</role>
+ <url>http://sandbox.hortonworks.com:50070/webhdfs</url>
+</service>
+</code></pre><p>In addition to the service configuration for HDFSUI, the REST
service configuration for WEBHDFS is also required.</p>
+<pre><code><service>
+ <role>NAMENODE</role>
+ <url>hdfs://sandbox.hortonworks.com:8020</url>
+</service>
+<service>
+ <role>WEBHDFS</role>
+ <url>http://sandbox.hortonworks.com:50070/webhdfs</url>
+</service>
+</code></pre><p>By default the gateway is configured to use the HTTP endpoint
for WebHDFS in the Sandbox. This could alternatively be configured to use the
HTTPS endpoint by provided the correct address.</p><h4><a
id="Name+Node+UI+URL+Mapping">Name Node UI URL Mapping</a> <a
href="#Name+Node+UI+URL+Mapping"><img
src="markbook-section-link.png"/></a></h4><p>For Name Node UI URLs, the mapping
of Knox Gateway accessible HDFS UI URLs to direct HDFS UI URLs is:</p>
+<table>
+ <tbody>
+ <tr>
+ <td>Gateway </td>
+
<td><code>https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/hdfs</code>
</td>
+ </tr>
+ <tr>
+ <td>Cluster </td>
+ <td><code>http://{webhdfs-host}:50070/</code> </td>
+ </tr>
+ </tbody>
+</table><p>For example to browse the file system using the NameNode UI the URL
in a web browser would be:</p>
+<pre><code>http://sandbox.hortonworks.com:50070/explorer.html#
+</code></pre><p>And using the gateway to access the same page the URL would be
(where the gateway host:port is ‘localhost:8443’)</p>
+<pre><code>https://localhost:8443/gateway/sandbox/hdfs/explorer.html#
+</code></pre><h3><a id="Job+History+UI">Job History UI</a> <a
href="#Job+History+UI"><img src="markbook-section-link.png"/></a></h3><p>The
Job History UI service can be configured in a topology by adding the following
snippet. The values in this sample are configured to work with an installed
Sandbox VM.</p>
+<pre><code><service>
+ <role>JOBHISTORYUI</role>
+ <url>http://sandbox.hortonworks.com:19888</url>
+</service>
+</code></pre><p>The values for the host and port can be obtained from the
following property in mapred-site.xml</p>
+<pre><code><property>
+ <name>mapreduce.jobhistory.webapp.address</name>
+ <value>sandbox.hortonworks.com:19888</value>
+</property>
+</code></pre><h4><a id="Job+History+UI+URL+Mapping">Job History UI URL
Mapping</a> <a href="#Job+History+UI+URL+Mapping"><img
src="markbook-section-link.png"/></a></h4><p>For Job History UI URLs, the
mapping of Knox Gateway accessible Job History UI URLs to direct Job History UI
URLs is:</p>
+<table>
+ <tbody>
+ <tr>
+ <td>Gateway </td>
+
<td><code>https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/jobhistory</code>
</td>
+ </tr>
+ <tr>
+ <td>Cluster </td>
+ <td><code>http://{jobhistory-host}:19888/jobhistory</code> </td>
+ </tr>
+ </tbody>
+</table><h3><a id="Oozie+UI">Oozie UI</a> <a href="#Oozie+UI"><img
src="markbook-section-link.png"/></a></h3><p>The Oozie UI service can be
configured in a topology by adding the following snippet. The values in this
sample are configured to work with an installed Sandbox VM.</p>
+<pre><code><service>
+ <role>OOZIEUI</role>
+ <url>http://sandbox.hortonworks.com:11000/oozie</url>
+</service>
+</code></pre><p>The value for the URL can be obtained from the following
property in oozie-site.xml</p>
+<pre><code><property>
+ <name>oozie.base.url</name>
+ <value>http://sandbox.hortonworks.com:11000/oozie</value>
+</property>
+</code></pre><h4><a id="Oozie+UI+URL+Mapping">Oozie UI URL Mapping</a> <a
href="#Oozie+UI+URL+Mapping"><img
src="markbook-section-link.png"/></a></h4><p>For Oozie UI URLs, the mapping of
Knox Gateway accessible Oozie UI URLs to direct Oozie UI URLs is:</p>
+<table>
+ <tbody>
+ <tr>
+ <td>Gateway </td>
+
<td><code>https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/oozie/</code>
</td>
+ </tr>
+ <tr>
+ <td>Cluster </td>
+ <td><code>http://{oozie-host}:11000/oozie/</code> </td>
+ </tr>
+ </tbody>
+</table><h3><a id="HBase+UI">HBase UI</a> <a href="#HBase+UI"><img
src="markbook-section-link.png"/></a></h3><p>The HBase UI service can be
configured in a topology by adding the following snippet. The values in this
sample are configured to work with an installed Sandbox VM.</p>
+<pre><code><service>
+ <role>HBASEUI</role>
+ <url>http://sandbox.hortonworks.com:16010</url>
+</service>
+</code></pre><p>The values for the host and port can be obtained from the
following property in hbase-site.xml. Below the hostname of the hbase master is
used since the bindAddress is 0.0.0.0</p>
+<pre><code><property>
+ <name>hbase.master.info.bindAddress</name>
+ <value>0.0.0.0</value>
+</property>
+<property>
+ <name>hbase.master.info.port</name>
+ <value>16010</value>
+</property>
+</code></pre><h4><a id="HBase+UI+URL+Mapping">HBase UI URL Mapping</a> <a
href="#HBase+UI+URL+Mapping"><img
src="markbook-section-link.png"/></a></h4><p>For HBase UI URLs, the mapping of
Knox Gateway accessible HBase UI URLs to direct HBase Master UI URLs is:</p>
+<table>
+ <tbody>
+ <tr>
+ <td>Gateway </td>
+
<td><code>https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/hbase/webui/</code>
</td>
+ </tr>
+ <tr>
+ <td>Cluster </td>
+ <td><code>http://{hbase-master-host}:16010/</code> </td>
+ </tr>
+ </tbody>
+</table><h3><a id="Yarn+UI">Yarn UI</a> <a href="#Yarn+UI"><img
src="markbook-section-link.png"/></a></h3><p>The Yarn UI service can be
configured in a topology by adding the following snippet. The values in this
sample are configured to work with an installed Sandbox VM.</p>
+<pre><code><service>
+ <role>YARNUI</role>
+ <url>http://sandbox.hortonworks.com:8088</url>
+</service>
+</code></pre><p>The values for the host and port can be obtained from the
following property in mapred-site.xml</p>
+<pre><code><property>
+ <name>yarn.resourcemanager.webapp.address</name>
+ <value>sandbox.hortonworks.com:8088</value>
+</property>
+</code></pre><h4><a id="Yarn+UI+URL+Mapping">Yarn UI URL Mapping</a> <a
href="#Yarn+UI+URL+Mapping"><img
src="markbook-section-link.png"/></a></h4><p>For Resource Manager UI URLs, the
mapping of Knox Gateway accessible Resource Manager UI URLs to direct Resource
Manager UI URLs is:</p>
+<table>
+ <tbody>
+ <tr>
+ <td>Gateway </td>
+
<td><code>https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/yarn</code>
</td>
+ </tr>
+ <tr>
+ <td>Cluster </td>
+ <td><code>http://{resource-manager-host}:8088/cluster</code> </td>
+ </tr>
+ </tbody>
+</table><h3><a id="Spark+UI">Spark UI</a> <a href="#Spark+UI"><img
src="markbook-section-link.png"/></a></h3><p>The Spark History UI service can
be configured in a topology by adding the following snippet. The values in this
sample are configured to work with an installed Sandbox VM.</p>
+<pre><code><service>
+ <role>SPARKHISTORYUI</role>
+ <url>http://sandbox.hortonworks.com:18080/</url>
+</service>
+</code></pre><h4><a id="Spark+History+UI+URL+Mapping">Spark History UI URL
Mapping</a> <a href="#Spark+History+UI+URL+Mapping"><img
src="markbook-section-link.png"/></a></h4><p>For Spark History UI URLs, the
mapping of Knox Gateway accessible Spark History UI URLs to direct Spark
History UI URLs is:</p>
+<table>
+ <tbody>
+ <tr>
+ <td>Gateway </td>
+
<td><code>https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/sparkhistory</code>
</td>
+ </tr>
+ <tr>
+ <td>Cluster </td>
+ <td><code>http://{spark-history-host}:18080</code> </td>
+ </tr>
+ </tbody>
+</table><h2><a id="Limitations">Limitations</a> <a href="#Limitations"><img
src="markbook-section-link.png"/></a></h2><h3><a
id="Secure+Oozie+POST/PUT+Request+Payload+Size+Restriction">Secure Oozie
POST/PUT Request Payload Size Restriction</a> <a
href="#Secure+Oozie+POST/PUT+Request+Payload+Size+Restriction"><img
src="markbook-section-link.png"/></a></h3><p>With one exception there are no
known size limits for requests or responses payloads that pass through the
gateway. The exception involves POST or PUT request payload sizes for Oozie in
a Kerberos secured Hadoop cluster. In this one case there is currently a 4Kb
payload size limit for the first request made to the Hadoop cluster. This is a
result of how the gateway negotiates a trust relationship between itself and
the cluster via SPNego. There is an undocumented configuration setting to
modify this limit’s value if required. In the future this will be made
more easily configuration and at that time it will be documented.</
p><h3><a id="Group+Membership+Propagation">Group Membership Propagation</a> <a
href="#Group+Membership+Propagation"><img
src="markbook-section-link.png"/></a></h3><p>Groups that are acquired via Shiro
Group Lookup and/or Identity Assertion Group Principal Mapping are not
propagated to the Hadoop services. Therefore, groups used for Service Level
Authorization policy may not match those acquired within the cluster via
GroupMappingServiceProvider plugins.</p><h2><a
id="Troubleshooting">Troubleshooting</a> <a href="#Troubleshooting"><img
src="markbook-section-link.png"/></a></h2><h3><a id="Finding+Logs">Finding
Logs</a> <a href="#Finding+Logs"><img
src="markbook-section-link.png"/></a></h3><p>When things aren’t working
the first thing you need to do is examine the diagnostic logs. Depending upon
how you are running the gateway these diagnostic logs will be output to
different locations.</p><h4><a id="java+-jar+bin/gateway.jar">java -jar
bin/gateway.jar</a> <a href="#java+-jar+bin
/gateway.jar"><img src="markbook-section-link.png"/></a></h4><p>When the
gateway is run this way the diagnostic output is written directly to the
console. If you want to capture that output you will need to redirect the
console output to a file using OS specific techniques.</p>
<pre><code>java -jar bin/gateway.jar > gateway.log
</code></pre><h4><a id="bin/gateway.sh+start">bin/gateway.sh start</a> <a
href="#bin/gateway.sh+start"><img
src="markbook-section-link.png"/></a></h4><p>When the gateway is run this way
the diagnostic output is written to /var/log/knox/knox.out and
/var/log/knox/knox.err. Typically only knox.out will have content.</p><h3><a
id="Increasing+Logging">Increasing Logging</a> <a
href="#Increasing+Logging"><img
src="markbook-section-link.png"/></a></h3><p>The <code>log4j.properties</code>
files <code>{GATEWAY_HOME}/conf</code> can be used to change the granularity of
the logging done by Knox. The Knox server must be restarted in order for these
changes to take effect. There are various useful loggers pre-populated but
commented out.</p>
<pre><code>log4j.logger.org.apache.hadoop.gateway=DEBUG # Use this logger to
increase the debugging of Apache Knox itself.
Modified: knox/trunk/books/0.7.0/book.md
URL:
http://svn.apache.org/viewvc/knox/trunk/books/0.7.0/book.md?rev=1718414&r1=1718413&r2=1718414&view=diff
==============================================================================
--- knox/trunk/books/0.7.0/book.md (original)
+++ knox/trunk/books/0.7.0/book.md Mon Dec 7 15:55:08 2015
@@ -85,6 +85,7 @@ In general the goals of the gateway are
<<book_gateway-details.md>>
<<book_client-details.md>>
<<book_service-details.md>>
+<<book_ui_service_details.md>>
<<book_limitations.md>>
<<book_troubleshooting.md>>
