Author: sumit
Date: Mon Dec 7 04:15:08 2015
New Revision: 1718240
URL: http://svn.apache.org/viewvc?rev=1718240&view=rev
Log:
KNOX-625 docs for UI service support
Added:
knox/trunk/books/0.7.0/book_ui_service-details.md
- copied, changed from r1714654,
knox/trunk/books/0.7.0/book_service-details.md
Copied: knox/trunk/books/0.7.0/book_ui_service-details.md (from r1714654,
knox/trunk/books/0.7.0/book_service-details.md)
URL:
http://svn.apache.org/viewvc/knox/trunk/books/0.7.0/book_ui_service-details.md?p2=knox/trunk/books/0.7.0/book_ui_service-details.md&p1=knox/trunk/books/0.7.0/book_service-details.md&r1=1714654&r2=1718240&rev=1718240&view=diff
==============================================================================
--- knox/trunk/books/0.7.0/book_service-details.md (original)
+++ knox/trunk/books/0.7.0/book_ui_service-details.md Mon Dec 7 04:15:08 2015
@@ -15,283 +15,209 @@
limitations under the License.
--->
-## Service Details ##
+## UI Service Details ##
-In the sections that follow the integrations currently available out of the
box with the gateway will be described.
-In general these sections will include examples that demonstrate how to access
each of these services via the gateway.
-In many cases this will include both the use of [cURL][curl] as a REST API
client as well as the use of the Knox Client DSL.
-You may notice that there are some minor differences between using the REST
API of a given service via the gateway.
-In general this is necessary in order to achieve the goal of leaking internal
Hadoop cluster details to the client.
-
-Keep in mind that the gateway uses a plugin model for supporting Hadoop
services.
-Check back with the [Apache Knox][site] site for the latest news on plugin
availability.
-You can also create your own custom plugin to extend the capabilities of the
gateway.
-
-These are the current Hadoop services with built-in support.
-
-* #[WebHDFS]
-* #[WebHCat]
-* #[Oozie]
-* #[HBase]
-* #[Hive]
-* #[Yarn]
-* #[Storm]
+In the sections that follow the integrations for proxying various UIs
currently available out of the box with the
+gateway will be described. These sections will include examples that
demonstrate how to access each of these services
+via the gateway.
+
+These are the current Hadoop services with built-in support for their UIs.
+
+* #[Name Node UI]
+* #[Job History UI]
+* #[Oozie UI]
+* #[HBase UI]
+* #[Yarn UI]
+* #[Spark UI]
### Assumptions
-This document assumes a few things about your environment in order to simplify
the examples.
+This section assumes an environment setup similar to the one in the REST
services section #[Service Details]
-* The JVM is executable as simply java.
-* The Apache Knox Gateway is installed and functional.
-* The example commands are executed within the context of the GATEWAY_HOME
current directory.
-The GATEWAY_HOME directory is the directory within the Apache Knox Gateway
installation that contains the README file and the bin, conf and deployments
directories.
-* The [cURL][curl] command line HTTP client utility is installed and
functional.
-* A few examples optionally require the use of commands from a standard Groovy
installation.
-These examples are optional but to try them you will need Groovy
[installed](http://groovy.codehaus.org/Installing+Groovy).
-* The default configuration for all of the samples is setup for use with
Hortonworks' [Sandbox][sandbox] version 2.
-
-### Customization
-
-Using these samples with other Hadoop installations will require changes to
the steps describe here as well as changes to referenced sample scripts.
-This will also likely require changes to the gateway's default configuration.
-In particular host names, ports user names and password may need to be changes
to match your environment.
-These changes may need to be made to gateway configuration and also the Groovy
sample script files in the distribution.
-All of the values that may need to be customized in the sample scripts can be
found together at the top of each of these files.
-
-### cURL
-
-The cURL HTTP client command line utility is used extensively in the examples
for each service.
-In particular this form of the cURL command line is used repeatedly.
-
- curl -i -k -u guest:guest-password ...
-
-The option -i (aka --include) is used to output HTTP response header
information.
-This will be important when the content of the HTTP Location header is
required for subsequent requests.
-
-The option -k (aka --insecure) is used to avoid any issues resulting the use
of demonstration SSL certificates.
-
-The option -u (aka --user) is used to provide the credentials to be used when
the client is challenged by the gateway.
-
-Keep in mind that the samples do not use the cookie features of cURL for the
sake of simplicity.
-Therefore each request via cURL will result in an authentication.
-
-<<service_webhdfs.md>>
-<<service_webhcat.md>>
-<<service_oozie.md>>
-<<service_hbase.md>>
-<<service_hive.md>>
-<<service_yarn.md>>
-<<service_storm.md>>
-<<service_default_ha.md>>
-
-
-### Service Test API
-
-The gateway supports a Service Test API that can be used to test Knox's
ability to connect to each of the different Hadoop services via a simeple HTTP
GET request. To be able to access this API one must add the following line into
the topology for which you wish to run the service test.
-
- <service>
- <role>SERVICE-TEST</role>
- </service>
-
-After adding the above to a topology, you can make a cURL request with the
following structure
-
- curl -i -k
"https://{gateway-hostname}:{gateway-port}/gateway/path/{topology-name}/service-test?username=guest&password=guest-password";
-
-An alternate method of providing credentials:
-
- curl -i -k -u guest:guest-password
https://{gateway-hostname}:{gateway-port}/gateway/path/{topology-name}/service-test
-
-Below is an example response. The gateway is also capable of returning XML if
specified in the request's "Accept" HTTP header.
-
- {
- "serviceTestWrapper": {
- "Tests": {
- "ServiceTest": [
- {
- "serviceName": "WEBHDFS",
- "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/webhdfs/v1/?op=LISTSTATUS";,
- "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
- "httpCode": 200,
- "message": "Request sucessful."
- },
- {
- "serviceName": "WEBHCAT",
- "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/templeton/v1/status";,
- "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
- "httpCode": 200,
- "message": "Request sucessful."
- },
- {
- "serviceName": "WEBHCAT",
- "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/templeton/v1/version";,
- "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
- "httpCode": 200,
- "message": "Request sucessful."
- },
- {
- "serviceName": "WEBHCAT",
- "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/templeton/v1/version/hive";,
- "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
- "httpCode": 200,
- "message": "Request sucessful."
- },
- {
- "serviceName": "WEBHCAT",
- "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/templeton/v1/version/hadoop";,
- "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
- "httpCode": 200,
- "message": "Request sucessful."
- },
- {
- "serviceName": "OOZIE",
- "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/oozie/v1/admin/build-version";,
- "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
- "httpCode": 200,
- "message": "Request sucessful."
- },
- {
- "serviceName": "OOZIE",
- "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/oozie/v1/admin/status";,
- "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
- "httpCode": 200,
- "message": "Request sucessful."
- },
- {
- "serviceName": "OOZIE",
- "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/oozie/versions";,
- "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
- "httpCode": 200,
- "message": "Request sucessful."
- },
- {
- "serviceName": "WEBHBASE",
- "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/hbase/version";,
- "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
- "httpCode": 200,
- "message": "Request sucessful."
- },
- {
- "serviceName": "WEBHBASE",
- "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/hbase/version/cluster";,
- "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
- "httpCode": 200,
- "message": "Request sucessful."
- },
- {
- "serviceName": "WEBHBASE",
- "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/hbase/status/cluster";,
- "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
- "httpCode": 200,
- "message": "Request sucessful."
- },
- {
- "serviceName": "WEBHBASE",
- "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/hbase";,
- "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
- "httpCode": 200,
- "message": "Request sucessful."
- },
- {
- "serviceName": "RESOURCEMANAGER",
- "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/resourcemanager/v1/{topology-name}/info";,
- "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
- "httpCode": 200,
- "message": "Request sucessful."
- },
- {
- "serviceName": "RESOURCEMANAGER",
- "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/resourcemanager/v1/{topology-name}/metrics";,
- "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
- "httpCode": 200,
- "message": "Request sucessful."
- },
- {
- "serviceName": "RESOURCEMANAGER",
- "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/resourcemanager/v1/{topology-name}/apps";,
- "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
- "httpCode": 200,
- "message": "Request sucessful."
- },
- {
- "serviceName": "FALCON",
- "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/admin/stack";,
- "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
- "httpCode": 200,
- "message": "Request sucessful."
- },
- {
- "serviceName": "FALCON",
- "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/admin/version";,
- "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
- "httpCode": 200,
- "message": "Request sucessful."
- },
- {
- "serviceName": "FALCON",
- "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/metadata/lineage/serialize";,
- "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
- "httpCode": 200,
- "message": "Request sucessful."
- },
- {
- "serviceName": "FALCON",
- "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/metadata/lineage/vertices/all";,
- "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
- "httpCode": 200,
- "message": "Request sucessful."
- },
- {
- "serviceName": "FALCON",
- "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/metadata/lineage/edges/all";,
- "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
- "httpCode": 200,
- "message": "Request sucessful."
- },
- {
- "serviceName": "STORM",
- "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/storm/api/v1/cluster/configuration";,
- "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
- "httpCode": 200,
- "message": "Request sucessful."
- },
- {
- "serviceName": "STORM",
- "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/storm/api/v1/cluster/summary";,
- "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
- "httpCode": 200,
- "message": "Request sucessful."
- },
- {
- "serviceName": "STORM",
- "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/storm/api/v1/supervisor/summary";,
- "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
- "httpCode": 200,
- "message": "Request sucessful."
- },
- {
- "serviceName": "STORM",
- "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/storm/api/v1/topology/summary";,
- "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
- "httpCode": 200,
- "message": "Request sucessful."
- }
- ]
- },
- "messages": {
- "message": [
-
- ]
- }
- }
- }
-
-
-We can see that this service-test makes HTTP requests to each of the services
through Knox using the specified topology. The test will only make calls to
those services that have entries within the topology file.
-
-##### Adding and Changing test URLs
-
-URLs for each service are stored in
`{GATEWAY_HOME}/data/services/{service-name}/{service-version}/service.xml`.
Each `<testURL>` element represents a service resource that will be tested if
the service is set up in the topology. You can add or remove these from teh
service.xml files. Just note if you add URL's there is no guarantee in the
order they will be tested. All default URLs have been tested and work on
various clusters. If a new URL is added and doesn't respond in a way the user
expects then it is up to the user to determine whether the URL is correct or
not.
-
-##### Some important things to note:
- - In the first cURL request, the quotes are necessary around the URL or else
a command line terminal will not include the `&password` query parameter in the
request.
- - This API call does not require any credentials to receive a response from
Knox, but expect to receive 401 responses from each of the services if none are
provided.
+### Name Node UI ###
+
+The Name Node UI is available on the same host and port combination that
WebHDFS is available on. As mentioned in the
+WebHDFS REST service configuration section, the values for the host and port
can be obtained from the following
+properties in hdfs-site.xml
+
+ <property>
+ <name>dfs.namenode.http-address</name>
+ <value>sandbox.hortonworks.com:50070</value>
+ </property>
+ <property>
+ <name>dfs.https.namenode.https-address</name>
+ <value>sandbox.hortonworks.com:50470</value>
+ </property>
+
+The values above need to be reflected in each topology descriptor file
deployed to the gateway.
+The gateway by default includes a sample topology descriptor file
`{GATEWAY_HOME}/deployments/sandbox.xml`.
+The values in this sample are configured to work with an installed Sandbox VM.
+
+ <service>
+ <role>HDFSUI</role>
+ <url>http://sandbox.hortonworks.com:50070/webhdfs</url>
+ </service>
+
+In addition to the service configuration for HDFSUI, the REST service
configuration for WEBHDFS is also required.
+
+ <service>
+ <role>NAMENODE</role>
+ <url>hdfs://sandbox.hortonworks.com:8020</url>
+ </service>
+ <service>
+ <role>WEBHDFS</role>
+ <url>http://sandbox.hortonworks.com:50070/webhdfs</url>
+ </service>
+
+By default the gateway is configured to use the HTTP endpoint for WebHDFS in
the Sandbox.
+This could alternatively be configured to use the HTTPS endpoint by provided
the correct address.
+
+#### Name Node UI URL Mapping ####
+
+For Name Node UI URLs, the mapping of Knox Gateway accessible HDFS UI URLs to
direct HDFS UI URLs is:
+
+| ------- |
----------------------------------------------------------------------------- |
+| Gateway |
`https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/hdfs` |
+| Cluster | `http://{webhdfs-host}:50070/`
|
+
+For example to browse the file system using the NameNode UI the URL in a web
browser would be:
+
+ http://sandbox.hortonworks.com:50070/explorer.html#
+
+And using the gateway to access the same page the URL would be (where the
gateway host:port is 'localhost:8443')
+
+ https://localhost:8443/gateway/sandbox/hdfs/explorer.html#
+
+
+### Job History UI ###
+
+The Job History UI service can be configured in a topology by adding the
following snippet. The values in this sample
+are configured to work with an installed Sandbox VM.
+
+ <service>
+ <role>JOBHISTORYUI</role>
+ <url>http://sandbox.hortonworks.com:19888</url>
+ </service>
+
+The values for the host and port can be obtained from the following property
in mapred-site.xml
+
+ <property>
+ <name>mapreduce.jobhistory.webapp.address</name>
+ <value>sandbox.hortonworks.com:19888</value>
+ </property>
+
+
+
+#### Job History UI URL Mapping ####
+
+For Job History UI URLs, the mapping of Knox Gateway accessible Job History UI
URLs to direct Job History UI URLs is:
+
+| ------- |
----------------------------------------------------------------------------- |
+| Gateway |
`https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/jobhistory`
|
+| Cluster | `http://{jobhistory-host}:19888/jobhistory`
|
+
+
+### Oozie UI ###
+
+The Oozie UI service can be configured in a topology by adding the following
snippet. The values in this sample
+are configured to work with an installed Sandbox VM.
+
+ <service>
+ <role>OOZIEUI</role>
+ <url>http://sandbox.hortonworks.com:11000/oozie</url>
+ </service>
+
+The value for the URL can be obtained from the following property in
oozie-site.xml
+
+ <property>
+ <name>oozie.base.url</name>
+ <value>http://sandbox.hortonworks.com:11000/oozie</value>
+ </property>
+
+
+
+#### Oozie UI URL Mapping ####
+
+For Oozie UI URLs, the mapping of Knox Gateway accessible Oozie UI URLs to
direct Oozie UI URLs is:
+
+| ------- |
----------------------------------------------------------------------------- |
+| Gateway |
`https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/oozie/` |
+| Cluster | `http://{oozie-host}:11000/oozie/`
|
+
+
+### HBase UI ###
+
+The HBase UI service can be configured in a topology by adding the following
snippet. The values in this sample
+are configured to work with an installed Sandbox VM.
+
+ <service>
+ <role>HBASEUI</role>
+ <url>http://sandbox.hortonworks.com:16010</url>
+ </service>
+
+The values for the host and port can be obtained from the following property
in hbase-site.xml.
+Below the hostname of the hbase master is used since the bindAddress is 0.0.0.0
+
+ <property>
+ <name>hbase.master.info.bindAddress</name>
+ <value>0.0.0.0</value>
+ </property>
+ <property>
+ <name>hbase.master.info.port</name>
+ <value>16010</value>
+ </property>
+
+#### HBase UI URL Mapping ####
+
+For HBase UI URLs, the mapping of Knox Gateway accessible HBase UI URLs to
direct HBase Master
+UI URLs is:
+
+| ------- |
-------------------------------------------------------------------------------------|
+| Gateway |
`https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/hbase/webui/`
|
+| Cluster | `http://{hbase-master-host}:16010/`
|
+
+### Yarn UI ###
+
+The Yarn UI service can be configured in a topology by adding the following
snippet. The values in this sample
+are configured to work with an installed Sandbox VM.
+
+ <service>
+ <role>YARNUI</role>
+ <url>http://sandbox.hortonworks.com:8088</url>
+ </service>
+
+The values for the host and port can be obtained from the following property
in mapred-site.xml
+
+ <property>
+ <name>yarn.resourcemanager.webapp.address</name>
+ <value>sandbox.hortonworks.com:8088</value>
+ </property>
+
+#### Yarn UI URL Mapping ####
+
+For Resource Manager UI URLs, the mapping of Knox Gateway accessible Resource
Manager UI URLs to direct Resource Manager
+UI URLs is:
+
+| ------- |
----------------------------------------------------------------------------- |
+| Gateway |
`https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/yarn` |
+| Cluster | `http://{resource-manager-host}:8088/cluster`
|
+
+### Spark UI ###
+
+The Spark History UI service can be configured in a topology by adding the
following snippet. The values in this sample
+are configured to work with an installed Sandbox VM.
+
+
+ <service>
+ <role>SPARKHISTORYUI</role>
+ <url>http://sandbox.hortonworks.com:18080/</url>
+ </service>
+
+#### Spark History UI URL Mapping ####
+
+For Spark History UI URLs, the mapping of Knox Gateway accessible Spark
History UI URLs to direct Spark History
+UI URLs is:
+
+| ------- |
----------------------------------------------------------------------------- |
+| Gateway |
`https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/sparkhistory`
|
+| Cluster | `http://{spark-history-host}:18080`
|
