Author: kminder
Date: Wed Jul 22 18:20:28 2015
New Revision: 1692312
URL: http://svn.apache.org/r1692312
Log:
KNOX-549: Test service connections through Knox with Knox CLI
Added:
knox/trunk/books/0.7.0/service_service_test.md
Modified:
knox/site/books/knox-0-7-0/user-guide.html
knox/trunk/books/0.7.0/book_service-details.md
knox/trunk/books/0.7.0/knox_cli.md
Modified: knox/site/books/knox-0-7-0/user-guide.html
URL:
http://svn.apache.org/viewvc/knox/site/books/knox-0-7-0/user-guide.html?rev=1692312&r1=1692311&r2=1692312&view=diff
==============================================================================
--- knox/site/books/knox-0-7-0/user-guide.html (original)
+++ knox/site/books/knox-0-7-0/user-guide.html Wed Jul 22 18:20:28 2015
@@ -707,6 +707,40 @@ ip-10-39-107-209.ec2.internal
<td>Optional; Print extra debug info on failed authentication</td>
</tr>
</tbody>
+</table><h4><a id="Gateway+Service+Test"></a>Gateway Service Test</h4><h5><a
id="`bin/knoxcli.sh+service-test+[--cluster+c]+[--hostname+hostname]+[--port+port]+[--u+username]+[--p+password]+[--d]+[--help]`"></a><code>bin/knoxcli.sh
service-test [--cluster c] [--hostname hostname] [--port port] [--u username]
[--p password] [--d] [--help]</code></h5><p>This will test a topology
configuration’s ability to connect to multiple hadoop services. Each
service found in a topology will be tested with multiple URLs. Results are
printed to the console in JSON format..</p>
+<table>
+ <thead>
+ <tr>
+ <th>argument </th>
+ <th>description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>--cluster </td>
+ <td>Required; name of cluster for which you want to test
authentication</td>
+ </tr>
+ <tr>
+ <td>--hostname </td>
+ <td>Required; hostname of the cluster currently running on the
machine</td>
+ </tr>
+ <tr>
+ <td>--port </td>
+ <td>Optional; port that the cluster is running on. If not supplied CLI
will try to read config files to find the port.</td>
+ </tr>
+ <tr>
+ <td>--u </td>
+ <td>Required; username to authorize against Hadoop services</td>
+ </tr>
+ <tr>
+ <td>--p </td>
+ <td>Required; password to match username</td>
+ </tr>
+ <tr>
+ <td>--d </td>
+ <td>Optional; Print extra debug info on failed authentication</td>
+ </tr>
+ </tbody>
</table><h3><a id="Admin+API"></a>Admin API</h3><p>Access to the administrator
functions of Knox are provided by the Admin REST API.</p><h4><a
id="Admin+API+URL"></a>Admin API URL</h4><p>The URL mapping for the Knox Admin
API is simple:</p>
<table>
<tbody>
@@ -3526,7 +3560,201 @@ curl -ik -b ~/cookiejar.txt -c ~/cookiej
curl -ik -b ~/cookiejar.txt -c ~/cookiejar.txt -u guest:guest-password -H
'x-csrf-token:{token-value}' -X POST \
http://localhost:8744/api/v1/topology/{id}/kill/0
-</code></pre><h2><a id="Limitations"></a>Limitations</h2><h3><a
id="Secure+Oozie+POST/PUT+Request+Payload+Size+Restriction"></a>Secure Oozie
POST/PUT Request Payload Size Restriction</h3><p>With one exception there are
no known size limits for requests or responses payloads that pass through the
gateway. The exception involves POST or PUT request payload sizes for Oozie in
a Kerberos secured Hadoop cluster. In this one case there is currently a 4Kb
payload size limit for the first request made to the Hadoop cluster. This is a
result of how the gateway negotiates a trust relationship between itself and
the cluster via SPNego. There is an undocumented configuration setting to
modify this limit’s value if required. In the future this will be made
more easily configuration and at that time it will be documented.</p><h3><a
id="Group+Membership+Propagation"></a>Group Membership
Propagation</h3><p>Groups that are acquired via Shiro Group Lookup and/or
Identity Assertion Group Princip
al Mapping are not propagated to the Hadoop services. Therefore, groups used
for Service Level Authorization policy may not match those acquired within the
cluster via GroupMappingServiceProvider plugins.</p><h2><a
id="Troubleshooting"></a>Troubleshooting</h2><h3><a
id="Finding+Logs"></a>Finding Logs</h3><p>When things aren’t working the
first thing you need to do is examine the diagnostic logs. Depending upon how
you are running the gateway these diagnostic logs will be output to different
locations.</p><h4><a id="java+-jar+bin/gateway.jar"></a>java -jar
bin/gateway.jar</h4><p>When the gateway is run this way the diagnostic output
is written directly to the console. If you want to capture that output you will
need to redirect the console output to a file using OS specific techniques.</p>
+</code></pre><h3><a id="Service+Test+API"></a>Service Test API</h3><p>The
gateway supports a Service Test API that can be used to test Knox’s
ability to connect to each of the different Hadoop services via a simeple HTTP
GET request. To be able to access this API one must add the following line into
the topology for which you wish to run the service test.</p>
+<pre><code><service>
+ <role>SERVICE-TEST</role>
+</service>
+</code></pre><p>After adding the above to a topology, you can make a cURL
request with the following structure</p>
+<pre><code>curl -i -k
"https://{gateway-hostname}:{gateway-port}/gateway/path/{topology-name}/service-test?username=guest&password=guest-password"
+</code></pre><p>An alternate method of providing credentials:</p>
+<pre><code>curl -i -k -u guest:guest-password
https://{gateway-hostname}:{gateway-port}/gateway/path/{topology-name}/service-test
+</code></pre><p>Below is an example response. The gateway is also capable of
returning XML if specified in the request’s “Accept” HTTP
header.</p>
+<pre><code>{
+ "serviceTestWrapper": {
+ "Tests": {
+ "ServiceTest": [
+ {
+ "serviceName": "WEBHDFS",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/webhdfs/v1/?op=LISTSTATUS",
+ "responseContent": "Content-Length:0,Content-Type:
application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHCAT",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/templeton/v1/status",
+ "responseContent": "Content-Length:0,Content-Type:
application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHCAT",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/templeton/v1/version",
+ "responseContent": "Content-Length:0,Content-Type:
application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHCAT",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/templeton/v1/version/hive",
+ "responseContent": "Content-Length:0,Content-Type:
application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHCAT",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/templeton/v1/version/hadoop",
+ "responseContent": "Content-Length:0,Content-Type:
application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "OOZIE",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/oozie/v1/admin/build-version",
+ "responseContent": "Content-Length:0,Content-Type:
application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "OOZIE",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/oozie/v1/admin/status",
+ "responseContent": "Content-Length:0,Content-Type:
application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "OOZIE",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/oozie/versions",
+ "responseContent": "Content-Length:0,Content-Type:
application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHBASE",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/hbase/version",
+ "responseContent": "Content-Length:0,Content-Type:
application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHBASE",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/hbase/version/cluster",
+ "responseContent": "Content-Length:0,Content-Type:
application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHBASE",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/hbase/status/cluster",
+ "responseContent": "Content-Length:0,Content-Type:
application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHBASE",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/hbase",
+ "responseContent": "Content-Length:0,Content-Type:
application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "RESOURCEMANAGER",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/resourcemanager/v1/{topology-name}/info",
+ "responseContent": "Content-Length:0,Content-Type:
application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "RESOURCEMANAGER",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/resourcemanager/v1/{topology-name}/metrics",
+ "responseContent": "Content-Length:0,Content-Type:
application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "RESOURCEMANAGER",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/resourcemanager/v1/{topology-name}/apps",
+ "responseContent": "Content-Length:0,Content-Type:
application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "FALCON",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/admin/stack",
+ "responseContent": "Content-Length:0,Content-Type:
application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "FALCON",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/admin/version",
+ "responseContent": "Content-Length:0,Content-Type:
application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "FALCON",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/metadata/lineage/serialize",
+ "responseContent": "Content-Length:0,Content-Type:
application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "FALCON",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/metadata/lineage/vertices/all",
+ "responseContent": "Content-Length:0,Content-Type:
application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "FALCON",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/metadata/lineage/edges/all",
+ "responseContent": "Content-Length:0,Content-Type:
application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "STORM",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/storm/api/v1/cluster/configuration",
+ "responseContent": "Content-Length:0,Content-Type:
application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "STORM",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/storm/api/v1/cluster/summary",
+ "responseContent": "Content-Length:0,Content-Type:
application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "STORM",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/storm/api/v1/supervisor/summary",
+ "responseContent": "Content-Length:0,Content-Type:
application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "STORM",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/storm/api/v1/topology/summary",
+ "responseContent": "Content-Length:0,Content-Type:
application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ }
+ ]
+ },
+ "messages": {
+ "message": [
+
+ ]
+ }
+ }
+}
+</code></pre><p>We can see that this service-test makes HTTP requests to each
of the services through Knox using the specified topology. The test will only
make calls to those services that have entries within the topology
file.</p><h5><a id="Adding+and+Changing+test+URLs"></a>Adding and Changing test
URLs</h5><p>URLs for each service are stored in
<code>{GATEWAY_HOME}/data/services/{service-name}/{service-version}/service.xml</code>.
Each <code><testURL></code> element represents a service resource that
will be tested if the service is set up in the topology. You can add or remove
these from teh service.xml files. Just note if you add URL’s there is no
guarantee in the order they will be tested. All default URLs have been tested
and work on various clusters. If a new URL is added and doesn’t respond
in a way the user expects then it is up to the user to determine whether the
URL is correct or not.</p><h5><a id="Some+important+things+to+note:"></a>Some
important th
ings to note:</h5>
+<ul>
+ <li>In the first cURL request, the quotes are necessary around the URL or
else a command line terminal will not include the <code>&password</code>
query parameter in the request.</li>
+ <li>This API call does not require any credentials to receive a response
from Knox, but expect to receive 401 responses from each of the services if
none are provided.</li>
+</ul><h2><a id="Limitations"></a>Limitations</h2><h3><a
id="Secure+Oozie+POST/PUT+Request+Payload+Size+Restriction"></a>Secure Oozie
POST/PUT Request Payload Size Restriction</h3><p>With one exception there are
no known size limits for requests or responses payloads that pass through the
gateway. The exception involves POST or PUT request payload sizes for Oozie in
a Kerberos secured Hadoop cluster. In this one case there is currently a 4Kb
payload size limit for the first request made to the Hadoop cluster. This is a
result of how the gateway negotiates a trust relationship between itself and
the cluster via SPNego. There is an undocumented configuration setting to
modify this limit’s value if required. In the future this will be made
more easily configuration and at that time it will be documented.</p><h3><a
id="Group+Membership+Propagation"></a>Group Membership
Propagation</h3><p>Groups that are acquired via Shiro Group Lookup and/or
Identity Assertion Group Principal Mappi
ng are not propagated to the Hadoop services. Therefore, groups used for
Service Level Authorization policy may not match those acquired within the
cluster via GroupMappingServiceProvider plugins.</p><h2><a
id="Troubleshooting"></a>Troubleshooting</h2><h3><a
id="Finding+Logs"></a>Finding Logs</h3><p>When things aren’t working the
first thing you need to do is examine the diagnostic logs. Depending upon how
you are running the gateway these diagnostic logs will be output to different
locations.</p><h4><a id="java+-jar+bin/gateway.jar"></a>java -jar
bin/gateway.jar</h4><p>When the gateway is run this way the diagnostic output
is written directly to the console. If you want to capture that output you will
need to redirect the console output to a file using OS specific techniques.</p>
<pre><code>java -jar bin/gateway.jar > gateway.log
</code></pre><h4><a id="bin/gateway.sh+start"></a>bin/gateway.sh
start</h4><p>When the gateway is run this way the diagnostic output is written
to /var/log/knox/knox.out and /var/log/knox/knox.err. Typically only knox.out
will have content.</p><h3><a id="Increasing+Logging"></a>Increasing
Logging</h3><p>The <code>log4j.properties</code> files
<code>{GATEWAY_HOME}/conf</code> can be used to change the granularity of the
logging done by Knox. The Knox server must be restarted in order for these
changes to take effect. There are various useful loggers pre-populated but
commented out.</p>
<pre><code>log4j.logger.org.apache.hadoop.gateway=DEBUG # Use this logger to
increase the debugging of Apache Knox itself.
Modified: knox/trunk/books/0.7.0/book_service-details.md
URL:
http://svn.apache.org/viewvc/knox/trunk/books/0.7.0/book_service-details.md?rev=1692312&r1=1692311&r2=1692312&view=diff
==============================================================================
--- knox/trunk/books/0.7.0/book_service-details.md (original)
+++ knox/trunk/books/0.7.0/book_service-details.md Wed Jul 22 18:20:28 2015
@@ -84,3 +84,213 @@ Therefore each request via cURL will res
<<service_storm.md>>
+### Service Test API
+
+The gateway supports a Service Test API that can be used to test Knox's
ability to connect to each of the different Hadoop services via a simeple HTTP
GET request. To be able to access this API one must add the following line into
the topology for which you wish to run the service test.
+
+ <service>
+ <role>SERVICE-TEST</role>
+ </service>
+
+After adding the above to a topology, you can make a cURL request with the
following structure
+
+ curl -i -k
"https://{gateway-hostname}:{gateway-port}/gateway/path/{topology-name}/service-test?username=guest&password=guest-password";
+
+An alternate method of providing credentials:
+
+ curl -i -k -u guest:guest-password
https://{gateway-hostname}:{gateway-port}/gateway/path/{topology-name}/service-test
+
+Below is an example response. The gateway is also capable of returning XML if
specified in the request's "Accept" HTTP header.
+
+ {
+ "serviceTestWrapper": {
+ "Tests": {
+ "ServiceTest": [
+ {
+ "serviceName": "WEBHDFS",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/webhdfs/v1/?op=LISTSTATUS";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHCAT",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/templeton/v1/status";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHCAT",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/templeton/v1/version";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHCAT",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/templeton/v1/version/hive";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHCAT",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/templeton/v1/version/hadoop";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "OOZIE",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/oozie/v1/admin/build-version";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "OOZIE",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/oozie/v1/admin/status";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "OOZIE",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/oozie/versions";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHBASE",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/hbase/version";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHBASE",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/hbase/version/cluster";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHBASE",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/hbase/status/cluster";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHBASE",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/hbase";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "RESOURCEMANAGER",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/resourcemanager/v1/{topology-name}/info";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "RESOURCEMANAGER",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/resourcemanager/v1/{topology-name}/metrics";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "RESOURCEMANAGER",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/resourcemanager/v1/{topology-name}/apps";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "FALCON",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/admin/stack";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "FALCON",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/admin/version";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "FALCON",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/metadata/lineage/serialize";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "FALCON",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/metadata/lineage/vertices/all";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "FALCON",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/metadata/lineage/edges/all";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "STORM",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/storm/api/v1/cluster/configuration";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "STORM",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/storm/api/v1/cluster/summary";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "STORM",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/storm/api/v1/supervisor/summary";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "STORM",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/storm/api/v1/topology/summary";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ }
+ ]
+ },
+ "messages": {
+ "message": [
+
+ ]
+ }
+ }
+ }
+
+
+We can see that this service-test makes HTTP requests to each of the services
through Knox using the specified topology. The test will only make calls to
those services that have entries within the topology file.
+
+##### Adding and Changing test URLs
+
+URLs for each service are stored in
`{GATEWAY_HOME}/data/services/{service-name}/{service-version}/service.xml`.
Each `<testURL>` element represents a service resource that will be tested if
the service is set up in the topology. You can add or remove these from teh
service.xml files. Just note if you add URL's there is no guarantee in the
order they will be tested. All default URLs have been tested and work on
various clusters. If a new URL is added and doesn't respond in a way the user
expects then it is up to the user to determine whether the URL is correct or
not.
+
+##### Some important things to note:
+ - In the first cURL request, the quotes are necessary around the URL or else
a command line terminal will not include the `&password` query parameter in the
request.
+ - This API call does not require any credentials to receive a response from
Knox, but expect to receive 401 responses from each of the services if none are
provided.
Modified: knox/trunk/books/0.7.0/knox_cli.md
URL:
http://svn.apache.org/viewvc/knox/trunk/books/0.7.0/knox_cli.md?rev=1692312&r1=1692311&r2=1692312&view=diff
==============================================================================
--- knox/trunk/books/0.7.0/knox_cli.md (original)
+++ knox/trunk/books/0.7.0/knox_cli.md Wed Jul 22 18:20:28 2015
@@ -110,4 +110,19 @@ This command will test a given topology'
argument | description
---------|-----------
\-\-cluster | Required; name of cluster for which you want to test
authentication
+\-\-d | Optional; Print extra debug info on failed authentication
+
+
+#### Gateway Service Test ####
+##### `bin/knoxcli.sh service-test [--cluster c] [--hostname hostname] [--port
port] [--u username] [--p password] [--d] [--help]` ####
+
+This will test a topology configuration's ability to connect to multiple
hadoop services. Each service found in a topology will be tested with multiple
URLs. Results are printed to the console in JSON format..
+
+argument | description
+---------|-----------
+\-\-cluster | Required; name of cluster for which you want to test
authentication
+\-\-hostname | Required; hostname of the cluster currently running on the
machine
+\-\-port | Optional; port that the cluster is running on. If not
supplied CLI will try to read config files to find the port.
+\-\-u | Required; username to authorize against Hadoop services
+\-\-p | Required; password to match username
\-\-d | Optional; Print extra debug info on failed authentication
\ No newline at end of file
Added: knox/trunk/books/0.7.0/service_service_test.md
URL:
http://svn.apache.org/viewvc/knox/trunk/books/0.7.0/service_service_test.md?rev=1692312&view=auto
==============================================================================
--- knox/trunk/books/0.7.0/service_service_test.md (added)
+++ knox/trunk/books/0.7.0/service_service_test.md Wed Jul 22 18:20:28 2015
@@ -0,0 +1,221 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+--->
+
+### Service Test ###
+
+The gateway supports a Service Test API that can be used to test Knox's
ability to connect to each of the different Hadoop services via a simple HTTP
GET request. To be able to access this API one must add the following entry
into the topology for which you wish to run the service test.
+
+ <service>
+ <role>SERVICE-TEST</role>
+ </service>
+
+After adding the above to a topology, you can make a cURL request with the
following structure
+
+ curl -i -k -u guest:guest-password
https://{gateway-host}:{gateway-port}/{gateway-path}/{topology-name}/service-test
+
+Below is an example response. The gateway is also capable of returning XML if
specified in the request's "Accept" HTTP header.
+
+ {
+ "serviceTestWrapper": {
+ "Tests": {
+ "ServiceTest": [
+ {
+ "serviceName": "WEBHDFS",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/webhdfs/v1/?op=LISTSTATUS";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHCAT",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/templeton/v1/status";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHCAT",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/templeton/v1/version";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHCAT",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/templeton/v1/version/hive";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHCAT",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/templeton/v1/version/hadoop";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "OOZIE",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/oozie/v1/admin/build-version";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "OOZIE",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/oozie/v1/admin/status";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "OOZIE",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/oozie/versions";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHBASE",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/hbase/version";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHBASE",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/hbase/version/cluster";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHBASE",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/hbase/status/cluster";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHBASE",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/hbase";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "RESOURCEMANAGER",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/resourcemanager/v1/{topology-name}/info";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "RESOURCEMANAGER",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/resourcemanager/v1/{topology-name}/metrics";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "RESOURCEMANAGER",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/resourcemanager/v1/{topology-name}/apps";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "FALCON",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/admin/stack";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "FALCON",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/admin/version";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "FALCON",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/metadata/lineage/serialize";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "FALCON",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/metadata/lineage/vertices/all";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "FALCON",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/metadata/lineage/edges/all";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "STORM",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/storm/api/v1/cluster/configuration";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "STORM",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/storm/api/v1/cluster/summary";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "STORM",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/storm/api/v1/supervisor/summary";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "STORM",
+ "requestURL":
"http://{gateway-host}:{gateway-port}/gateway/{topology-name}/storm/api/v1/topology/summary";,
+ "responseContent":
"Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ }
+ ]
+ },
+ "messages": {
+ "message": [
+ ]
+ }
+ }
+ }
+
+
+We can see that this service-test makes HTTP requests to each of the services
through Knox using the specified topology. The test will only make calls to
those services that have entries within the topology file.
+
+##### Adding and Changing test URLs
+
+URLs for each service are stored in
`{GATEWAY_HOME}/data/services/{service-name}/{service-version}/service.xml`.
Each `<testURL>` element represents a service resource that will be tested if
the service is set up in the topology. You can add or remove these from teh
service.xml files. Just note if you add URL's there is no guarantee in the
order they will be tested. All default URLs have been tested and work on
various clusters. If a new URL is added and doesn't respond in a way the user
expects then it is up to the user to determine whether the URL is correct or
not.
+
+##### Some important things to note:
+ - This API call does not require any credentials to receive a response from
Knox, but expect to receive 401 responses from each of the services if none are
provided.
