Author: more
Date: Fri Nov 3 16:54:42 2017
New Revision: 1814225
URL: http://svn.apache.org/viewvc?rev=1814225&view=rev
Log:
KNOX-1100 - Document Topology Discovery and Generation (Phil Zampino via
Sandeep More)
Modified:
knox/site/books/knox-0-14-0/user-guide.html
knox/trunk/books/0.14.0/admin_api.md
knox/trunk/books/0.14.0/config.md
Modified: knox/site/books/knox-0-14-0/user-guide.html
URL:
http://svn.apache.org/viewvc/knox/site/books/knox-0-14-0/user-guide.html?rev=1814225&r1=1814224&r2=1814225&view=diff
==============================================================================
--- knox/site/books/knox-0-14-0/user-guide.html (original)
+++ knox/site/books/knox-0-14-0/user-guide.html Fri Nov 3 16:54:42 2017
@@ -775,7 +775,167 @@ ip-10-39-107-209.ec2.internal
</topology>
</code></pre><h5><a id="Hostmap+Provider+Configuration">Hostmap Provider
Configuration</a> <a href="#Hostmap+Provider+Configuration"><img
src="markbook-section-link.png"/></a></h5><p>Details about each provider
configuration element is enumerated below.</p>
<dl><dt>topology/gateway/provider/role</dt><dd>The role for a Hostmap provider
must always be
<code>hostmap</code>.</dd><dt>topology/gateway/provider/name</dt><dd>The
Hostmap provider supplied out-of-the-box is selected via the name
<code>static</code>.</dd><dt>topology/gateway/provider/enabled</dt><dd>Host
mapping can be enabled or disabled by providing <code>true</code> or
<code>false</code>.</dd><dt>topology/gateway/provider/param</dt><dd>Host
mapping is configured by providing parameters for each external to internal
mapping.</dd><dt>topology/gateway/provider/param/name</dt><dd>The parameter
names represent the external host names associated with the internal host names
provided by the value element. This can be a comma separated list of host names
that all represent the same physical host. When mapping from internal to
external host name the first external host name in the list is
used.</dd><dt>topology/gateway/provider/param/value</dt><dd>The parameter
values represent the int
ernal host names associated with the external host names provider by the name
element. This can be a comma separated list of host names that all represent
the same physical host. When mapping from external to internal host names the
first internal host name in the list is used.</dd>
-</dl><h4><a id="Logging">Logging</a> <a href="#Logging"><img
src="markbook-section-link.png"/></a></h4><p>If necessary you can enable
additional logging by editing the <code>log4j.properties</code> file in the
<code>conf</code> directory. Changing the <code>rootLogger</code> value from
<code>ERROR</code> to <code>DEBUG</code> will generate a large amount of debug
logging. A number of useful, more fine loggers are also provided in the
file.</p><h4><a id="Java+VM+Options">Java VM Options</a> <a
href="#Java+VM+Options"><img src="markbook-section-link.png"/></a></h4><p>TODO
- Java VM options doc.</p><h4><a id="Persisting+the+Master+Secret">Persisting
the Master Secret</a> <a href="#Persisting+the+Master+Secret"><img
src="markbook-section-link.png"/></a></h4><p>The master secret is required to
start the server. This secret is used to access secured artifacts by the
gateway instance. Keystore, trust stores and credential stores are all
protected with the master secret.</p><p>You may persi
st the master secret by supplying the <em>-persist-master</em> switch at
startup. This will result in a warning indicating that persisting the secret is
less secure than providing it at startup. We do make some provisions in order
to protect the persisted password.</p><p>It is encrypted with AES 128 bit
encryption and where possible the file permissions are set to only be
accessible by the user that the gateway is running as.</p><p>After persisting
the secret, ensure that the file at data/security/master has the appropriate
permissions set for your environment. This is probably the most important layer
of defense for master secret. Do not assume that the encryption is sufficient
protection.</p><p>A specific user should be created to run the gateway. This
user will be the only user with permissions for the persisted master
file.</p><p>See the Knox CLI section for descriptions of the command line
utilities related to the master secret.</p><h4><a
id="Management+of+Security+Artifacts">M
anagement of Security Artifacts</a> <a
href="#Management+of+Security+Artifacts"><img
src="markbook-section-link.png"/></a></h4><p>There are a number of artifacts
that are used by the gateway in ensuring the security of wire level
communications, access to protected resources and the encryption of sensitive
data. These artifacts can be managed from outside of the gateway instances or
generated and populated by the gateway instance itself.</p><p>The following is
a description of how this is coordinated with both standalone (development,
demo, etc) gateway instances and instances as part of a cluster of gateways in
mind.</p><p>Upon start of the gateway server we:</p>
+</dl><h4><a id="Simplified+Topology+Descriptors">Simplified Topology
Descriptors</a> <a href="#Simplified+Topology+Descriptors"><img
src="markbook-section-link.png"/></a></h4><p>Simplified descriptors are a means
to facilitate provider configuration sharing and service endpoint discovery.
Rather than editing an XML topology descriptor, it’s possible to create a
simpler YAML (or JSON) descriptor specifying the desired contents of a
topology, which will yield a full topology descriptor and deployment.</p><h5><a
id="Externalized+Provider+Configurations">Externalized Provider
Configurations</a> <a href="#Externalized+Provider+Configurations"><img
src="markbook-section-link.png"/></a></h5><p>Sometimes, the same provider
configuration is applied to multiple Knox topologies. With the provider
configuration externalized from the simple descriptors, a single configuration
can be referenced by multiple topologies. This helps reduce the duplication of
configuration, and the need to updat
e multiple configuration files when a policy change is required. Updating a
provider configuration will trigger an update to all those topologies that
reference it.</p><p>The contents of externalized provider configuration is
identical to the gateway element from a full topology descriptor. The only
difference is that it’s defined in its own XML file in
<code>{GATEWAY_HOME}/conf/shared-providers/</code></p><p><em>Provider
Configuration Example</em></p>
+<pre><code><gateway>
+ <provider>
+ <role>authentication</role>
+ <name>ShiroProvider</name>
+ <enabled>true</enabled>
+ <param>
+ <name>sessionTimeout</name>
+ <value>30</value>
+ </param>
+ <param>
+ <name>main.ldapRealm</name>
+
<value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value>
+ </param>
+ <param>
+ <name>main.ldapContextFactory</name>
+
<value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value>
+ </param>
+ <param>
+ <name>main.ldapRealm.contextFactory</name>
+ <value>$ldapContextFactory</value>
+ </param>
+ <param>
+ <name>main.ldapRealm.userDnTemplate</name>
+
<value>uid={0},ou=people,dc=hadoop,dc=apache,dc=org</value>
+ </param>
+ <param>
+ <name>main.ldapRealm.contextFactory.url</name>
+ <value>ldap://localhost:33389</value>
+ </param>
+ <param>
+
<name>main.ldapRealm.contextFactory.authenticationMechanism</name>
+ <value>simple</value>
+ </param>
+ <param>
+ <name>urls./**</name>
+ <value>authcBasic</value>
+ </param>
+ </provider>
+
+ <provider>
+ <role>identity-assertion</role>
+ <name>Default</name>
+ <enabled>true</enabled>
+ </provider>
+
+ <provider>
+ <role>hostmap</role>
+ <name>static</name>
+ <enabled>true</enabled>
+
<param><name>localhost</name><value>sandbox,sandbox.hortonworks.com</value></param>
+ </provider>
+</gateway>
+</code></pre><h5><a id="Simplified+Descriptor+Files">Simplified Descriptor
Files</a> <a href="#Simplified+Descriptor+Files"><img
src="markbook-section-link.png"/></a></h5><p>Simplified descriptors allow
service URLs to be defined explicitly, just like full topology descriptors.
However, if URLs are omitted for a service, Knox will attempt to discover that
service’s URLs from the Hadoop cluster. Currently, this behavior is only
supported for clusters managed by Ambari. In any case, the simplified
descriptors are much more concise than a full topology
descriptor.</p><p><em>Descriptor Properties</em></p>
+<table>
+ <thead>
+ <tr>
+
<th>property </th>
+ <th>description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>discovery-type</td>
+ <td>The discovery source type. (Currently, the only supported type is
<em>AMBARI</em>).</td>
+ </tr>
+ <tr>
+ <td>discovery-address</td>
+ <td>The endpoint address for the discovery source.</td>
+ </tr>
+ <tr>
+ <td>discovery-user</td>
+ <td>The username with permission to access the discovery source. If
omitted, then Knox will check for an alias named
<em>ambari.discovery.user</em>, and use its value if defined.</td>
+ </tr>
+ <tr>
+ <td>discovery-pwd-alias</td>
+ <td>The alias of the password for the user with permission to access the
discovery source. If omitted, then Knox will check for an alias named
<em>ambari.discovery.password</em>, and use its value if defined.</td>
+ </tr>
+ <tr>
+ <td>provider-config-ref</td>
+ <td>A reference to a provider configuration in
<code>{GATEWAY_HOME}/conf/shared-providers/</code>.</td>
+ </tr>
+ <tr>
+ <td>cluster</td>
+ <td>The name of the cluster from which the topology service endpoints
should be determined.</td>
+ </tr>
+ <tr>
+ <td>services</td>
+ <td>The collection of services to be included in the topology.</td>
+ </tr>
+ </tbody>
+</table><p>Two file formats are supported for two distinct purposes.</p>
+<ul>
+ <li>YAML is intended for the individual hand-editing a simplified descriptor
because of its readability.</li>
+ <li>JSON is intended to be used for <a href="#Admin+API">API</a>
interactions.</li>
+</ul><p>That being said, there is nothing preventing the hand-editing of files
in the JSON format. However, the API will <em>not</em> accept YAML files as
input.</p><p><em>YAML Example</em> (based on the HDP Docker Sandbox)</p>
+<pre><code>---
+# Discovery source config
+discovery-type : AMBARI
+discovery-address : http://sandbox.hortonworks.com:8080
+
+# If this is not specified, the alias ambari.discovery.user is checked for a
username
+discovery-user : maria_dev
+
+# If this is not specified, the default alias ambari.discovery.password is used
+discovery-pwd-alias : sandbox.discovery.password
+
+# Provider config reference, the contents of which will be included in the
resulting topology descriptor
+provider-config-ref : sandbox-providers
+
+# The cluster for which the details should be discovered
+cluster: Sandbox
+
+# The services to declare in the resulting topology descriptor, whose URLs
will be discovered (unless a value is specified)
+services:
+ - name: NAMENODE
+ - name: JOBTRACKER
+ - name: WEBHDFS
+ - name: WEBHCAT
+ - name: OOZIE
+ - name: WEBHBASE
+ - name: HIVE
+ - name: RESOURCEMANAGER
+ - name: KNOXSSO
+ params:
+ knoxsso.cookie.secure.only: true
+ knoxsso.token.ttl: 100000
+ - name: AMBARI
+ urls:
+ - http://sandbox.hortonworks.com:8080
+ - name: AMBARIUI
+ urls:
+ - http://sandbox.hortonworks.com:8080
+</code></pre><p><em>JSON Example</em> (based on the HDP Docker Sandbox)</p>
+<pre><code>{
+ "discovery-type":"AMBARI",
+
"discovery-address":"http://sandbox.hortonworks.com:8080",
+ "discovery-user":"maria_dev",
+ "discovery-pwd-alias":"sandbox.discovery.password",
+ "provider-config-ref":"sandbox-providers",
+ "cluster":"Sandbox",
+ "services":[
+ {"name":"NAMENODE"},
+ {"name":"JOBTRACKER"},
+ {"name":"WEBHDFS"},
+ {"name":"WEBHCAT"},
+ {"name":"OOZIE"},
+ {"name":"WEBHBASE"},
+ {"name":"HIVE"},
+ {"name":"RESOURCEMANAGER"},
+ {"name":"KNOXSSO",
+ "params":{
+ "knoxsso.cookie.secure.only":"true",
+ "knoxsso.token.ttl":"100000"
+ }
+ },
+ {"name":"AMBARI",
"urls":["http://sandbox.hortonworks.com:8080"]},
+ {"name":"AMBARIUI",
"urls":["http://sandbox.hortonworks.com:8080"]}
+ ]
+}
+</code></pre><p>Both of these examples illustrate the specification of
credentials for the interaction with Ambari. If no credentials are specified,
then the default aliases are queried. Use of the default aliases is sufficient
for scenarios where topology discovery will only interact with a single Ambari
instance. For multiple Ambari instances however, it’s most likely that
each will require different sets of credentials. The discovery-user and
discovery-pwd-alias properties exist for this purpose. Note that whether using
the default credential aliases or specifying a custom password alias, these <a
href="#Alias+creation">aliases must be defined</a> prior to any attempt to
deploy a topology using a simplified descriptor.</p><h5><a
id="Deployment+Directories">Deployment Directories</a> <a
href="#Deployment+Directories"><img
src="markbook-section-link.png"/></a></h5><p>Effecting topology changes is as
simple as modifying files in two specific directories.</p><p>The <code>{GATEW
AY_HOME}/conf/shared-providers/</code> directory is the location where Knox
looks for provider configurations. This directory is monitored for changes,
such that modifying a provider configuration file therein will trigger updates
to any referencing simplified descriptors in the
<code>{GATEWAY_HOME}/conf/descriptors/</code> directory. <em>Care should be
taken when deleting these files if there are referencing descriptors; any
subsequent modifications of referencing descriptors will fail when the deleted
provider configuration cannot be found. The references should all be modified
before deleting the provider configuration.</em></p><p>Likewise, the
<code>{GATEWAY_HOME}/conf/descriptors/</code> directory is monitored for
changes, such that adding or modifying a simplified descriptor file in this
directory will trigger the generation and deployment of a topology descriptor.
Deleting a descriptor from this directory will conversely result in the removal
of the previously-generated topol
ogy descriptor, and the associated topology will be undeployed.</p><p>If the
service details for a deployed (generated) topology are changed in the cluster,
then the Knox topology can be updated by ’touch’ing the simplified
descriptor. This will trigger discovery and regeneration/redeployment of the
topology descriptor.</p><p>Note that deleting a generated topology descriptor
from <code>{GATEWAY_HOME}/conf/topologies/</code> is not sufficient for its
removal. If the source descriptor is modified, or Knox is restarted, the
topology descriptor will be regenerated and deployed. Removing generated
topology descriptors should be done by removing the associated simplified
descriptor. For the same reason, editing generated topology descriptors is
strongly discouraged since they can be inadvertently overwritten.</p><p>Another
means by which these topology changes can be effected is the <a
href="#Admin+API">Admin API</a>.</p><h4><a id="Logging">Logging</a> <a
href="#Logging"><img
src="markbook-section-link.png"/></a></h4><p>If necessary you can enable
additional logging by editing the <code>log4j.properties</code> file in the
<code>conf</code> directory. Changing the <code>rootLogger</code> value from
<code>ERROR</code> to <code>DEBUG</code> will generate a large amount of debug
logging. A number of useful, more fine loggers are also provided in the
file.</p><h4><a id="Java+VM+Options">Java VM Options</a> <a
href="#Java+VM+Options"><img src="markbook-section-link.png"/></a></h4><p>TODO
- Java VM options doc.</p><h4><a id="Persisting+the+Master+Secret">Persisting
the Master Secret</a> <a href="#Persisting+the+Master+Secret"><img
src="markbook-section-link.png"/></a></h4><p>The master secret is required to
start the server. This secret is used to access secured artifacts by the
gateway instance. Keystore, trust stores and credential stores are all
protected with the master secret.</p><p>You may persist the master secret by
supplying the <em>-persist-master</e
m> switch at startup. This will result in a warning indicating that persisting
the secret is less secure than providing it at startup. We do make some
provisions in order to protect the persisted password.</p><p>It is encrypted
with AES 128 bit encryption and where possible the file permissions are set to
only be accessible by the user that the gateway is running as.</p><p>After
persisting the secret, ensure that the file at data/security/master has the
appropriate permissions set for your environment. This is probably the most
important layer of defense for master secret. Do not assume that the encryption
is sufficient protection.</p><p>A specific user should be created to run the
gateway. This user will be the only user with permissions for the persisted
master file.</p><p>See the Knox CLI section for descriptions of the command
line utilities related to the master secret.</p><h4><a
id="Management+of+Security+Artifacts">Management of Security Artifacts</a> <a
href="#Management+of+
Security+Artifacts"><img src="markbook-section-link.png"/></a></h4><p>There
are a number of artifacts that are used by the gateway in ensuring the security
of wire level communications, access to protected resources and the encryption
of sensitive data. These artifacts can be managed from outside of the gateway
instances or generated and populated by the gateway instance itself.</p><p>The
following is a description of how this is coordinated with both standalone
(development, demo, etc) gateway instances and instances as part of a cluster
of gateways in mind.</p><p>Upon start of the gateway server we:</p>
<ol>
<li>Look for an identity store at
<code>data/security/keystores/gateway.jks</code>. The identity store contains
the certificate and private key used to represent the identity of the server
for SSL connections and signature creation.
<ul>
@@ -1024,47 +1184,502 @@ keytool -keystore gateway.jks -storepass
<td>Optional; Print extra debug info on failed authentication</td>
</tr>
</tbody>
-</table><h3><a id="Admin+API">Admin API</a> <a href="#Admin+API"><img
src="markbook-section-link.png"/></a></h3><p>Access to the administrator
functions of Knox are provided by the Admin REST API.</p><h4><a
id="Admin+API+URL">Admin API URL</a> <a href="#Admin+API+URL"><img
src="markbook-section-link.png"/></a></h4><p>The URL mapping for the Knox Admin
API is simple:</p>
+</table><h3><a id="Admin+API">Admin API</a> <a href="#Admin+API"><img
src="markbook-section-link.png"/></a></h3><p>Access to the administrator
functions of Knox are provided by the Admin REST API.</p><h4><a
id="Admin+API+URL">Admin API URL</a> <a href="#Admin+API+URL"><img
src="markbook-section-link.png"/></a></h4><p>The URL mapping for the Knox Admin
API is:</p>
<table>
<tbody>
<tr>
- <td>Gateway </td>
+ <td>GatewayAPI </td>
<td><code>https://{gateway-host}:{gateway-port}/{gateway-path}/admin/api/v1</code>
</td>
</tr>
</tbody>
-</table><p>Please note that to access that admin API, the user attempting to
connect must have admin credentials inside of the LDAP Server</p><h5><a
id="API+Documentation">API Documentation</a> <a href="#API+Documentation"><img
src="markbook-section-link.png"/></a></h5><h6><a id="Operations">Operations</a>
<a href="#Operations"><img src="markbook-section-link.png"/></a></h6>
-<ul>
- <li><h6>HTTP GET</h6></li>
-</ul>
-<ol>
- <li><a href="#Server+Version">Server Version</a></li>
- <li><a href="#Topology+Collection">Topology Collection</a></li>
- <li><a href="#Topology">Topology</a></li>
-</ol>
-<ul>
- <li><h6>HTTP PUT</h6></li>
- <li><h6>HTTP DELETE</h6></li>
-</ul><h5><a id="Server+Version">Server Version</a> <a
href="#Server+Version"><img src="markbook-section-link.png"/></a></h5><h6><a
id="Description">Description</a> <a href="#Description"><img
src="markbook-section-link.png"/></a></h6><p>Calls to Knox and returns the
gateway’s current version and the version hash inside of a JSON object.
</p><h6><a id="Example+Request+URL">Example Request URL</a> <a
href="#Example+Request+URL"><img
src="markbook-section-link.png"/></a></h6><p><code>https://{gateway-host}:{gateway-port}/{gateway-path}/admin/api/v1/version</code>
</p><h6><a id="Example+cURL+Request">Example cURL Request</a> <a
href="#Example+cURL+Request"><img
src="markbook-section-link.png"/></a></h6><p><code>curl -u admin:admin-password
-i -k
https://{gateway-host}:{gateway-port}/{gateway-path}/admin/api/v1/version</code></p><h6><a
id="Response">Response</a> <a href="#Response"><img
src="markbook-section-link.png"/></a></h6>
-<pre><code><ServerVersion>
- <version>{version-number}</version>
- <hash>{version-hash}</hash>
-</ServerVersion>
-</code></pre><h5><a id="Topology+Collection">Topology Collection</a> <a
href="#Topology+Collection"><img
src="markbook-section-link.png"/></a></h5><h6><a
id="Description">Description</a> <a href="#Description"><img
src="markbook-section-link.png"/></a></h6><p>Calls to Knox and return an array
of JSON objects that represent the list of deployed topologies currently inside
of the gateway. </p><h6><a id="Example+Request+URL">Example Request URL</a> <a
href="#Example+Request+URL"><img
src="markbook-section-link.png"/></a></h6><p><code>https://{gateway-host}:{gateway-port}/{gateway-path}/admin/api/{api-version}/topologies</code></p><h6><a
id="Example+cURL+Request">Example cURL Request</a> <a
href="#Example+cURL+Request"><img
src="markbook-section-link.png"/></a></h6><p><code>curl -u admin:admin-password
-i -k -H Accept:application/json
https://{gateway-host}:{gateway-port}/{gateway-path}/admin/api/v1/topologies</code></p><h6><a
id="Response">Response</a> <a href="#Response"><img src="mar
kbook-section-link.png"/></a></h6>
-<pre><code>[
- {
-
"href":"https://localhost:8443/gateway/admin/api/v1/topologies/_default",
- "name":"_default",
- "timestamp":"1405633120000",
- "uri":"https://localhost:8443/gateway/_default"
- },
- {
-
"href":"https://localhost:8443/gateway/admin/api/v1/topologies/admin",
- "name":"admin",
- "timestamp":"1406672646000",
- "uri":"https://localhost:8443/gateway/admin"
+</table><p>Please note that to access this API, the user attempting to connect
must have admin credentials configured on the LDAP Server</p><h5><a
id="API+Documentation">API Documentation</a> <a href="#API+Documentation"><img
src="markbook-section-link.png"/></a></h5>
+<table>
+ <thead>
+ <th>Resource</th>
+ <th>Operation</th>
+ <th>Description</th>
+ </thead>
+ <tr>
+ <td>version</td>
+ <td>GET</td>
+ <td>Get the gateway version and the associated version hash</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Request</td>
+ <td><pre>curl -iku admin:admin-password {GatewayAPI}/version -H
Accept:application/json</pre></td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Response</td>
+ <td>
+ <pre>
+{
+ "ServerVersion" : {
+ "version" : "VERSION_ID",
+ "hash" : "VERSION_HASH"
}
-]
-</code></pre><h5><a id="Topology">Topology</a> <a href="#Topology"><img
src="markbook-section-link.png"/></a></h5><h6><a
id="Description">Description</a> <a href="#Description"><img
src="markbook-section-link.png"/></a></h6><p>Calls to Knox and return a JSON
object that represents the requested topology </p><h6><a
id="Example+Request+URL">Example Request URL</a> <a
href="#Example+Request+URL"><img
src="markbook-section-link.png"/></a></h6><p><code>https://{gateway-host}:{gateway-port}/{gateway-path}/admin/api/v1/topologies/{topology-name}</code>
</p><h6><a id="Example+cURL+Request">Example cURL Request</a> <a
href="#Example+cURL+Request"><img
src="markbook-section-link.png"/></a></h6><p><code>curl -u admin:admin-password
-i -k -H Accept:application/json
https://{gateway-host}:{gateway-port}/{gateway-path}/admin/api/v1/topologies/{topology-name}</code></p><h6><a
id="Response">Response</a> <a href="#Response"><img
src="markbook-section-link.png"/></a></h6><p>{ “name”: &ld
quo;admin”, “providers”: [{ “enabled”: true,
“name”: “ShiroProvider”, “params”: {
“sessionTimeout”: “30”, “main.ldapRealm”:
“org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm”,
“main.ldapRealm.userDnTemplate”:
“uid={0},ou=people,dc=hadoop,dc=apache,dc=org”,
“main.ldapRealm.contextFactory.url”: “<a
href="ldap://localhost:33389"";>ldap://localhost:33389";</a>,
”main.ldapRealm.contextFactory.authenticationMechanism“:
”simple“, ”urls./**“: ”authcBasic“ },
”role“: ”authentication“ }, { ”enabled“:
true, ”name“: ”AclsAuthz“, ”params“: {
”knox.acl“: ”admin;*;*“ }, ”role“:
”authorization“ }, { ”enabled“: true,
”name“: ”Defa
ult“, ”params“: {}, ”role“:
”identity-assertion“ }, { ”enabled“: true,
”name“: ”static“, ”params“: {
”localhost“: ”sandbox,sandbox.hortonworks.com“ },
”role“: ”hostmap“ }], ”services“: [{
“name”: null, “params”: {}, “role”:
“KNOX”, “url”: null }], ”timestamp“:
1406672646000, ”uri“: ”<a
href="https://localhost:8443/gateway/admin";>https://localhost:8443/gateway/admin</a>"
}</p><h3><a id="X-Forwarded-*+Headers+Support">X-Forwarded-* Headers
Support</a> <a href="#X-Forwarded-*+Headers+Support"><img
src="markbook-section-link.png"/></a></h3><p>Out-of-the-box Knox provides
support for some X-Forwarded-* headers through the use of a Servlet Filter.
Specifically the headers handled/populated by Knox are:</p>
+} </pre>
+ </td>
+ </tr>
+
+ <tr>
+ <td>topologies</td>
+ <td>GET</td>
+ <td>Get an enumeration of the topologies currently deployed in the
gateway.</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Request</td>
+ <td><pre>curl -iku admin:admin-password {GatewayAPI}/topologies -H
Accept:application/json</pre></td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Response</td>
+ <td>
+ <pre>
+{
+ "topologies" : {
+ "topology" : [ {
+ "name" : "admin",
+ "timestamp" : "1501508536000",
+ "uri" : "https://localhost:8443/gateway/admin";,
+ "href" :
"https://localhost:8443/gateway/admin/api/v1/topologies/admin";
+ }, {
+ "name" : "sandbox",
+ "timestamp" : "1501508536000",
+ "uri" : "https://localhost:8443/gateway/sandbox";,
+ "href" :
"https://localhost:8443/gateway/admin/api/v1/topologies/sandbox";
+ } ]
+ }
+} </pre>
+ </td>
+ </tr>
+
+ <tr>
+ <td>topologies/{id}</td>
+ <td>GET</td>
+ <td>Get a JSON representation of the specified topology</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Request</td>
+ <td><pre>curl -iku admin:admin-password {GatewayAPI}/topologies/admin -H
Accept:application/json</pre></td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Response</td>
+ <td>
+ <pre>
+{
+ "name": "admin",
+ "providers": [{
+ "enabled": true,
+ "name": "ShiroProvider",
+ "params": {
+ "sessionTimeout": "30",
+ "main.ldapRealm": "org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm",
+ "main.ldapRealm.userDnTemplate":
"uid={0},ou=people,dc=hadoop,dc=apache,dc=org",
+ "main.ldapRealm.contextFactory.url": "ldap://localhost:33389";,
+ "main.ldapRealm.contextFactory.authenticationMechanism": "simple",
+ "urls./**": "authcBasic"
+ },
+ "role": "authentication"
+ }, {
+ "enabled": true,
+ "name": "AclsAuthz",
+ "params": {
+ "knox.acl": "admin;*;*"
+ },
+ "role": "authorization"
+ }, {
+ "enabled": true,
+ "name": "Default",
+ "params": {},
+ "role": "identity-assertion"
+ }, {
+ "enabled": true,
+ "name": "static",
+ "params": {
+ "localhost": "sandbox,sandbox.hortonworks.com"
+ },
+ "role": "hostmap"
+ }],
+ "services": [{
+ "name": null,
+ "params": {},
+ "role": "KNOX",
+ "url": null
+ }],
+ "timestamp": 1406672646000,
+ "uri": "https://localhost:8443/gateway/admin";
+} </pre>
+ </td>
+ </tr>
+
+ <tr>
+ <td> </td>
+ <td>PUT</td>
+ <td>Add (and deploy) a topology</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Request</td>
+ <td><pre>curl -iku admin:admin-password {Gateway}/topologies/mytopology \
+ -X PUT \
+ -H Content-Type:application/xml
+ -d "@mytopology.xml"</pre></td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Response</td>
+ <td>
+ <pre>
+<?xml version="1.0" encoding="UTF-8"?>
+<topology>
+ <uri>https://localhost:8443/gateway/mytopology</uri>
+ <name>mytopology</name>
+ <timestamp>1509720338000</timestamp>
+ <gateway>
+ <provider>
+ <role>authentication</role>
+ <name>ShiroProvider</name>
+ <enabled>true</enabled>
+ <param>
+ <name>sessionTimeout</name>
+ <value>30</value>
+ </param>
+ <param>
+ <name>main.ldapRealm</name>
+
<value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value>
+ </param>
+ <param>
+ <name>main.ldapContextFactory</name>
+
<value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value>
+ </param>
+ <param>
+ <name>main.ldapRealm.contextFactory</name>
+ <value>$ldapContextFactory</value>
+ </param>
+ <param>
+ <name>main.ldapRealm.userDnTemplate</name>
+
<value>uid={0},ou=people,dc=hadoop,dc=apache,dc=org</value>
+ </param>
+ <param>
+ <name>main.ldapRealm.contextFactory.url</name>
+ <value>ldap://localhost:33389</value>
+ </param>
+ <param>
+
<name>main.ldapRealm.contextFactory.authenticationMechanism</name>
+ <value>simple</value>
+ </param>
+ <param>
+ <name>urls./**</name>
+ <value>authcBasic</value>
+ </param>
+ </provider>
+ <provider>
+ <role>identity-assertion</role>
+ <name>Default</name>
+ <enabled>true</enabled>
+ </provider>
+ <provider>
+ <role>hostmap</role>
+ <name>static</name>
+ <enabled>true</enabled>
+ <param>
+ <name>localhost</name>
+ <value>sandbox,sandbox.hortonworks.com</value>
+ </param>
+ </provider>
+ </gateway>
+ <service>
+ <role>NAMENODE</role>
+ <url>hdfs://localhost:8020</url>
+ </service>
+ <service>
+ <role>JOBTRACKER</role>
+ <url>rpc://localhost:8050</url>
+ </service>
+ <service>
+ <role>WEBHDFS</role>
+ <url>http://localhost:50070/webhdfs</url>
+ </service>
+ <service>
+ <role>WEBHCAT</role>
+ <url>http://localhost:50111/templeton</url>
+ </service>
+ <service>
+ <role>OOZIE</role>
+ <url>http://localhost:11000/oozie</url>
+ </service>
+ <service>
+ <role>WEBHBASE</role>
+ <url>http://localhost:60080</url>
+ </service>
+ <service>
+ <role>HIVE</role>
+ <url>http://localhost:10001/cliservice</url>
+ </service>
+ <service>
+ <role>RESOURCEMANAGER</role>
+ <url>http://localhost:8088/ws</url>
+ </service>
+</topology></pre>
+ </td>
+ </tr>
+
+ <tr>
+ <td> </td>
+ <td>DELETE</td>
+ <td>Delete (and undeploy) a topology</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Request</td>
+ <td><pre>curl -iku admin:admin-password {Gateway}/topologies/mytopology -X
DELETE</pre></td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Response</td>
+ <td><pre>{ "deleted" : true }</pre></td>
+ </tr>
+
+ <tr>
+ <td>providerconfig</td>
+ <td>GET</td>
+ <td>Get an enumeration of the shared provider configurations currently
deployed to the gateway.</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Request</td>
+ <td><pre>curl -iku admin:admin-password
{GatewayAPI}/providerconfig</pre></td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Response</td>
+ <td>
+ <pre>
+{
+ "href" : "https://localhost:8443/gateway/admin/api/v1/providerconfig";,
+ "items" : [ {
+ "href" :
"https://localhost:8443/gateway/admin/api/v1/providerconfig/myproviders";,
+ "name" : "myproviders.xml"
+ },{
+ "href" :
"https://localhost:8443/gateway/admin/api/v1/providerconfig/sandbox-providers";,
+ "name" : "sandbox-providers.xml"
+ } ]
+} </pre>
+ </td>
+ </tr>
+
+ <tr>
+ <td>providerconfig/{id}</td>
+ <td>GET</td>
+ <td>Get the XML content of the specified shared provider
configuration.</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Request</td>
+ <td><pre>curl -iku admin:admin-password
{GatewayAPI}/providerconfig/sandbox-providers \
+ -H Accept:application/xml</pre></td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Response</td>
+ <td>
+ <pre>
+<gateway>
+ <provider>
+ <role>authentication</role>
+ <name>ShiroProvider</name>
+ <enabled>true</enabled>
+ <param>
+ <name>sessionTimeout</name>
+ <value>30</value>
+ </param>
+ <param>
+ <name>main.ldapRealm</name>
+
<value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value>
+ </param>
+ <param>
+ <name>main.ldapContextFactory</name>
+
<value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value>
+ </param>
+ <param>
+ <name>main.ldapRealm.contextFactory</name>
+ <value>$ldapContextFactory</value>
+ </param>
+ <param>
+ <name>main.ldapRealm.userDnTemplate</name>
+
<value>uid={0},ou=people,dc=hadoop,dc=apache,dc=org</value>
+ </param>
+ <param>
+ <name>main.ldapRealm.contextFactory.url</name>
+ <value>ldap://localhost:33389</value>
+ </param>
+ <param>
+
<name>main.ldapRealm.contextFactory.authenticationMechanism</name>
+ <value>simple</value>
+ </param>
+ <param>
+ <name>urls./**</name>
+ <value>authcBasic</value>
+ </param>
+ </provider>
+
+ <provider>
+ <role>identity-assertion</role>
+ <name>Default</name>
+ <enabled>true</enabled>
+ </provider>
+
+ <provider>
+ <role>hostmap</role>
+ <name>static</name>
+ <enabled>true</enabled>
+ <param>
+ <name>localhost</name>
+ <value>sandbox,sandbox.hortonworks.com</value>
+ </param>
+ </provider>
+</gateway></pre>
+ </td>
+ </tr>
+ </tr>
+ <td> </td>
+ <td>PUT</td>
+ <td>Add a shared provider configuration.</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Request</td>
+ <td><pre>curl -iku admin:admin-password
{GatewayAPI}/providerconfig/sandbox-providers \
+ -X PUT \
+ -H Content-Type:application/xml \
+ -d "@sandbox-providers.xml"</pre></td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Response</td>
+ <td><pre>HTTP 201 Created</pre></td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>DELETE</td>
+ <td>Delete a shared provider configuration</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Request</td>
+ <td><pre>curl -iku admin:admin-password
{GatewayAPI}/providerconfig/sandbox-providers -X DELETE</pre></td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Response</td>
+ <td>
+ <pre>{ "deleted" : "provider config sandbox-providers" }</pre>
+ </td>
+ </tr>
+
+ <tr>
+ <td>descriptors</td>
+ <td>GET</td>
+ <td>Get an enumeration of the simple descriptors currently deployed to the
gateway.</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Request</td>
+ <td><pre>curl -iku admin:admin-password {GatewayAPI}/descriptors -H
Accept:application/json</pre></td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Response</td>
+ <td>
+ <pre>
+{
+ "href" : "https://localhost:8443/gateway/admin/api/v1/descriptors";,
+ "items" : [ {
+ "href" :
"https://localhost:8443/gateway/admin/api/v1/descriptors/docker-sandbox";,
+ "name" : "docker-sandbox.json"
+ }, {
+ "href" :
"https://localhost:8443/gateway/admin/api/v1/descriptors/mytopology";,
+ "name" : "mytopology.yml"
+ } ]
+} </pre>
+ </td>
+ </tr>
+
+ <tr>
+ <td>descriptors/{id}</td>
+ <td>GET</td>
+ <td>Get the content of the specified descriptor.</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Request</td>
+ <td><pre>curl -iku admin:admin-password
{GatewayAPI}/descriptors/docker-sandbox \
+ -H Accept:application/json</pre></td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Response</td>
+ <td>
+ <pre>
+{
+ "discovery-type":"AMBARI",
+ "discovery-address":"http://sandbox.hortonworks.com:8080";,
+ "provider-config-ref":"sandbox-providers",
+ "cluster":"Sandbox",
+ "services":[
+ {"name":"NAMENODE"},
+ {"name":"JOBTRACKER"},
+ {"name":"WEBHDFS"},
+ {"name":"WEBHCAT"},
+ {"name":"OOZIE"},
+ {"name":"WEBHBASE"},
+ {"name":"HIVE"},
+ {"name":"RESOURCEMANAGER"} ]
+} </pre>
+ </td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>PUT</td>
+ <td>Add a simple descriptor (and generate and deploy a full topology
descriptor).</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Request</td>
+ <td><pre>curl -iku admin:admin-password
{GatewayAPI}/descriptors/docker-sandbox \
+ -X PUT \
+ -H Content-Type:application/json \
+ -d "@docker-sandbox.json"</pre></td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Response</td>
+ <td><pre>HTTP 201 Created</pre></td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>DELETE</td>
+ <td>Delete a simple descriptor (and undeploy the associated topology)</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Request</td>
+ <td><pre>curl -iku admin:admin-password
{GatewayAPI}/descriptors/docker-sandbox -X DELETE</pre></td>
+ <tr>
+ </tr>
+ <td> </td>
+ <td>Example Response</td>
+ <td>
+ <pre>{ "deleted" : "descriptor docker-sandbox" }</pre>
+ </td>
+ </tr>
+
+</table><h3><a id="X-Forwarded-*+Headers+Support">X-Forwarded-* Headers
Support</a> <a href="#X-Forwarded-*+Headers+Support"><img
src="markbook-section-link.png"/></a></h3><p>Out-of-the-box Knox provides
support for some X-Forwarded-* headers through the use of a Servlet Filter.
Specifically the headers handled/populated by Knox are:</p>
<ul>
<li>X-Forwarded-For</li>
<li>X-Forwarded-Proto</li>
Modified: knox/trunk/books/0.14.0/admin_api.md
URL:
http://svn.apache.org/viewvc/knox/trunk/books/0.14.0/admin_api.md?rev=1814225&r1=1814224&r2=1814225&view=diff
==============================================================================
--- knox/trunk/books/0.14.0/admin_api.md (original)
+++ knox/trunk/books/0.14.0/admin_api.md Fri Nov 3 16:54:42 2017
@@ -15,140 +15,509 @@
limitations under the License.
--->
-### Admin API ###
+### Admin API
Access to the administrator functions of Knox are provided by the Admin REST
API.
-#### Admin API URL ####
+#### Admin API URL
-The URL mapping for the Knox Admin API is simple:
+The URL mapping for the Knox Admin API is:
| ------- |
-----------------------------------------------------------------------------
|
-| Gateway |
`https://{gateway-host}:{gateway-port}/{gateway-path}/admin/api/v1`
|
+| GatewayAPI |
`https://{gateway-host}:{gateway-port}/{gateway-path}/admin/api/v1`
|
-Please note that to access that admin API, the user attempting to connect must
have admin credentials inside of the LDAP Server
+Please note that to access this API, the user attempting to connect must have
admin credentials configured on the LDAP Server
-##### API Documentation
+##### API Documentation
-###### Operations
-
- * ###### HTTP GET
- 1. #[Server Version]
- 2. #[Topology Collection]
- 3. #[Topology]
- * ###### HTTP PUT
- * ###### HTTP DELETE
-
-##### Server Version
-
-###### Description
-
-Calls to Knox and returns the gateway's current version and the version hash
inside of a JSON object.
-
-###### Example Request URL
-
-`https://{gateway-host}:{gateway-port}/{gateway-path}/admin/api/v1/version`
-
-###### Example cURL Request
-
-`curl -u admin:admin-password -i -k
https://{gateway-host}:{gateway-port}/{gateway-path}/admin/api/v1/version`
-
-###### Response
-
- <ServerVersion>
- <version>{version-number}</version>
- <hash>{version-hash}</hash>
- </ServerVersion>
-
-##### Topology Collection
-
-###### Description
-
-Calls to Knox and return an array of JSON objects that represent the list of
deployed topologies currently inside of the gateway.
-
-###### Example Request URL
-
-`https://{gateway-host}:{gateway-port}/{gateway-path}/admin/api/{api-version}/topologies`
-
-###### Example cURL Request
-
-`curl -u admin:admin-password -i -k -H Accept:application/json
https://{gateway-host}:{gateway-port}/{gateway-path}/admin/api/v1/topologies`
-
-###### Response
-
- [
- {
-
"href":"https://localhost:8443/gateway/admin/api/v1/topologies/_default";,
- "name":"_default",
- "timestamp":"1405633120000",
- "uri":"https://localhost:8443/gateway/_default";
- },
- {
- "href":"https://localhost:8443/gateway/admin/api/v1/topologies/admin";,
- "name":"admin",
- "timestamp":"1406672646000",
- "uri":"https://localhost:8443/gateway/admin";
- }
- ]
-
-
-##### Topology
-
-###### Description
-
-Calls to Knox and return a JSON object that represents the requested topology
-
-###### Example Request URL
-
-`https://{gateway-host}:{gateway-port}/{gateway-path}/admin/api/v1/topologies/{topology-name}`
-
-###### Example cURL Request
-
-`curl -u admin:admin-password -i -k -H Accept:application/json
https://{gateway-host}:{gateway-port}/{gateway-path}/admin/api/v1/topologies/{topology-name}`
-
-###### Response
-
- {
- "name": "admin",
- "providers": [{
- "enabled": true,
- "name": "ShiroProvider",
- "params": {
- "sessionTimeout": "30",
- "main.ldapRealm": "org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm",
- "main.ldapRealm.userDnTemplate":
"uid={0},ou=people,dc=hadoop,dc=apache,dc=org",
- "main.ldapRealm.contextFactory.url": "ldap://localhost:33389";,
- "main.ldapRealm.contextFactory.authenticationMechanism": "simple",
- "urls./**": "authcBasic"
- },
- "role": "authentication"
- }, {
- "enabled": true,
- "name": "AclsAuthz",
- "params": {
- "knox.acl": "admin;*;*"
- },
- "role": "authorization"
- }, {
- "enabled": true,
- "name": "Default",
- "params": {},
- "role": "identity-assertion"
- }, {
- "enabled": true,
- "name": "static",
- "params": {
- "localhost": "sandbox,sandbox.hortonworks.com"
- },
- "role": "hostmap"
- }],
- "services": [{
+<table>
+ <thead>
+ <th>Resource</th>
+ <th>Operation</th>
+ <th>Description</th>
+ </thead>
+ <tr>
+ <td>version</td>
+ <td>GET</td>
+ <td>Get the gateway version and the associated version hash</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Request</td>
+ <td><pre>curl -iku admin:admin-password {GatewayAPI}/version -H
Accept:application/json</pre></td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Response</td>
+ <td>
+ <pre>
+{
+ "ServerVersion" : {
+ "version" : "VERSION_ID",
+ "hash" : "VERSION_HASH"
+ }
+} </pre>
+ </td>
+ </tr>
+
+ <tr>
+ <td>topologies</td>
+ <td>GET</td>
+ <td>Get an enumeration of the topologies currently deployed in the
gateway.</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Request</td>
+ <td><pre>curl -iku admin:admin-password {GatewayAPI}/topologies -H
Accept:application/json</pre></td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Response</td>
+ <td>
+ <pre>
+{
+ "topologies" : {
+ "topology" : [ {
+ "name" : "admin",
+ "timestamp" : "1501508536000",
+ "uri" : "https://localhost:8443/gateway/admin";,
+ "href" :
"https://localhost:8443/gateway/admin/api/v1/topologies/admin";
+ }, {
+ "name" : "sandbox",
+ "timestamp" : "1501508536000",
+ "uri" : "https://localhost:8443/gateway/sandbox";,
+ "href" :
"https://localhost:8443/gateway/admin/api/v1/topologies/sandbox";
+ } ]
+ }
+} </pre>
+ </td>
+ </tr>
+
+ <tr>
+ <td>topologies/{id}</td>
+ <td>GET</td>
+ <td>Get a JSON representation of the specified topology</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Request</td>
+ <td><pre>curl -iku admin:admin-password {GatewayAPI}/topologies/admin -H
Accept:application/json</pre></td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Response</td>
+ <td>
+ <pre>
+{
+ "name": "admin",
+ "providers": [{
+ "enabled": true,
+ "name": "ShiroProvider",
+ "params": {
+ "sessionTimeout": "30",
+ "main.ldapRealm": "org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm",
+ "main.ldapRealm.userDnTemplate":
"uid={0},ou=people,dc=hadoop,dc=apache,dc=org",
+ "main.ldapRealm.contextFactory.url": "ldap://localhost:33389";,
+ "main.ldapRealm.contextFactory.authenticationMechanism": "simple",
+ "urls./**": "authcBasic"
+ },
+ "role": "authentication"
+ }, {
+ "enabled": true,
+ "name": "AclsAuthz",
+ "params": {
+ "knox.acl": "admin;*;*"
+ },
+ "role": "authorization"
+ }, {
+ "enabled": true,
+ "name": "Default",
+ "params": {},
+ "role": "identity-assertion"
+ }, {
+ "enabled": true,
+ "name": "static",
+ "params": {
+ "localhost": "sandbox,sandbox.hortonworks.com"
+ },
+ "role": "hostmap"
+ }],
+ "services": [{
"name": null,
"params": {},
"role": "KNOX",
"url": null
- }],
- "timestamp": 1406672646000,
- "uri": "https://localhost:8443/gateway/admin";
- }
+ }],
+ "timestamp": 1406672646000,
+ "uri": "https://localhost:8443/gateway/admin";
+} </pre>
+ </td>
+ </tr>
+
+ <tr>
+ <td> </td>
+ <td>PUT</td>
+ <td>Add (and deploy) a topology</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Request</td>
+ <td><pre>curl -iku admin:admin-password {Gateway}/topologies/mytopology \
+ -X PUT \
+ -H Content-Type:application/xml
+ -d "@mytopology.xml"</pre></td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Response</td>
+ <td>
+ <pre>
+<?xml version="1.0" encoding="UTF-8"?>
+<topology>
+ <uri>https://localhost:8443/gateway/mytopology</uri>
+ <name>mytopology</name>
+ <timestamp>1509720338000</timestamp>
+ <gateway>
+ <provider>
+ <role>authentication</role>
+ <name>ShiroProvider</name>
+ <enabled>true</enabled>
+ <param>
+ <name>sessionTimeout</name>
+ <value>30</value>
+ </param>
+ <param>
+ <name>main.ldapRealm</name>
+
<value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value>
+ </param>
+ <param>
+ <name>main.ldapContextFactory</name>
+
<value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value>
+ </param>
+ <param>
+ <name>main.ldapRealm.contextFactory</name>
+ <value>$ldapContextFactory</value>
+ </param>
+ <param>
+ <name>main.ldapRealm.userDnTemplate</name>
+
<value>uid={0},ou=people,dc=hadoop,dc=apache,dc=org</value>
+ </param>
+ <param>
+ <name>main.ldapRealm.contextFactory.url</name>
+ <value>ldap://localhost:33389</value>
+ </param>
+ <param>
+
<name>main.ldapRealm.contextFactory.authenticationMechanism</name>
+ <value>simple</value>
+ </param>
+ <param>
+ <name>urls./**</name>
+ <value>authcBasic</value>
+ </param>
+ </provider>
+ <provider>
+ <role>identity-assertion</role>
+ <name>Default</name>
+ <enabled>true</enabled>
+ </provider>
+ <provider>
+ <role>hostmap</role>
+ <name>static</name>
+ <enabled>true</enabled>
+ <param>
+ <name>localhost</name>
+ <value>sandbox,sandbox.hortonworks.com</value>
+ </param>
+ </provider>
+ </gateway>
+ <service>
+ <role>NAMENODE</role>
+ <url>hdfs://localhost:8020</url>
+ </service>
+ <service>
+ <role>JOBTRACKER</role>
+ <url>rpc://localhost:8050</url>
+ </service>
+ <service>
+ <role>WEBHDFS</role>
+ <url>http://localhost:50070/webhdfs</url>
+ </service>
+ <service>
+ <role>WEBHCAT</role>
+ <url>http://localhost:50111/templeton</url>
+ </service>
+ <service>
+ <role>OOZIE</role>
+ <url>http://localhost:11000/oozie</url>
+ </service>
+ <service>
+ <role>WEBHBASE</role>
+ <url>http://localhost:60080</url>
+ </service>
+ <service>
+ <role>HIVE</role>
+ <url>http://localhost:10001/cliservice</url>
+ </service>
+ <service>
+ <role>RESOURCEMANAGER</role>
+ <url>http://localhost:8088/ws</url>
+ </service>
+</topology></pre>
+ </td>
+ </tr>
+
+ <tr>
+ <td> </td>
+ <td>DELETE</td>
+ <td>Delete (and undeploy) a topology</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Request</td>
+ <td><pre>curl -iku admin:admin-password {Gateway}/topologies/mytopology -X
DELETE</pre></td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Response</td>
+ <td><pre>{ "deleted" : true }</pre></td>
+ </tr>
+
+ <tr>
+ <td>providerconfig</td>
+ <td>GET</td>
+ <td>Get an enumeration of the shared provider configurations currently
deployed to the gateway.</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Request</td>
+ <td><pre>curl -iku admin:admin-password
{GatewayAPI}/providerconfig</pre></td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Response</td>
+ <td>
+ <pre>
+{
+ "href" : "https://localhost:8443/gateway/admin/api/v1/providerconfig";,
+ "items" : [ {
+ "href" :
"https://localhost:8443/gateway/admin/api/v1/providerconfig/myproviders";,
+ "name" : "myproviders.xml"
+ },{
+ "href" :
"https://localhost:8443/gateway/admin/api/v1/providerconfig/sandbox-providers";,
+ "name" : "sandbox-providers.xml"
+ } ]
+} </pre>
+ </td>
+ </tr>
+
+ <tr>
+ <td>providerconfig/{id}</td>
+ <td>GET</td>
+ <td>Get the XML content of the specified shared provider
configuration.</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Request</td>
+ <td><pre>curl -iku admin:admin-password
{GatewayAPI}/providerconfig/sandbox-providers \
+ -H Accept:application/xml</pre></td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Response</td>
+ <td>
+ <pre>
+<gateway>
+ <provider>
+ <role>authentication</role>
+ <name>ShiroProvider</name>
+ <enabled>true</enabled>
+ <param>
+ <name>sessionTimeout</name>
+ <value>30</value>
+ </param>
+ <param>
+ <name>main.ldapRealm</name>
+
<value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value>
+ </param>
+ <param>
+ <name>main.ldapContextFactory</name>
+
<value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value>
+ </param>
+ <param>
+ <name>main.ldapRealm.contextFactory</name>
+ <value>$ldapContextFactory</value>
+ </param>
+ <param>
+ <name>main.ldapRealm.userDnTemplate</name>
+
<value>uid={0},ou=people,dc=hadoop,dc=apache,dc=org</value>
+ </param>
+ <param>
+ <name>main.ldapRealm.contextFactory.url</name>
+ <value>ldap://localhost:33389</value>
+ </param>
+ <param>
+
<name>main.ldapRealm.contextFactory.authenticationMechanism</name>
+ <value>simple</value>
+ </param>
+ <param>
+ <name>urls./**</name>
+ <value>authcBasic</value>
+ </param>
+ </provider>
+
+ <provider>
+ <role>identity-assertion</role>
+ <name>Default</name>
+ <enabled>true</enabled>
+ </provider>
+
+ <provider>
+ <role>hostmap</role>
+ <name>static</name>
+ <enabled>true</enabled>
+ <param>
+ <name>localhost</name>
+ <value>sandbox,sandbox.hortonworks.com</value>
+ </param>
+ </provider>
+</gateway></pre>
+ </td>
+ </tr>
+ </tr>
+ <td> </td>
+ <td>PUT</td>
+ <td>Add a shared provider configuration.</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Request</td>
+ <td><pre>curl -iku admin:admin-password
{GatewayAPI}/providerconfig/sandbox-providers \
+ -X PUT \
+ -H Content-Type:application/xml \
+ -d "@sandbox-providers.xml"</pre></td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Response</td>
+ <td><pre>HTTP 201 Created</pre></td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>DELETE</td>
+ <td>Delete a shared provider configuration</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Request</td>
+ <td><pre>curl -iku admin:admin-password
{GatewayAPI}/providerconfig/sandbox-providers -X DELETE</pre></td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Response</td>
+ <td>
+ <pre>{ "deleted" : "provider config sandbox-providers" }</pre>
+ </td>
+ </tr>
+
+ <tr>
+ <td>descriptors</td>
+ <td>GET</td>
+ <td>Get an enumeration of the simple descriptors currently deployed to the
gateway.</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Request</td>
+ <td><pre>curl -iku admin:admin-password {GatewayAPI}/descriptors -H
Accept:application/json</pre></td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Response</td>
+ <td>
+ <pre>
+{
+ "href" : "https://localhost:8443/gateway/admin/api/v1/descriptors";,
+ "items" : [ {
+ "href" :
"https://localhost:8443/gateway/admin/api/v1/descriptors/docker-sandbox";,
+ "name" : "docker-sandbox.json"
+ }, {
+ "href" :
"https://localhost:8443/gateway/admin/api/v1/descriptors/mytopology";,
+ "name" : "mytopology.yml"
+ } ]
+} </pre>
+ </td>
+ </tr>
+
+ <tr>
+ <td>descriptors/{id}</td>
+ <td>GET</td>
+ <td>Get the content of the specified descriptor.</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Request</td>
+ <td><pre>curl -iku admin:admin-password
{GatewayAPI}/descriptors/docker-sandbox \
+ -H Accept:application/json</pre></td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Response</td>
+ <td>
+ <pre>
+{
+ "discovery-type":"AMBARI",
+ "discovery-address":"http://sandbox.hortonworks.com:8080";,
+ "provider-config-ref":"sandbox-providers",
+ "cluster":"Sandbox",
+ "services":[
+ {"name":"NAMENODE"},
+ {"name":"JOBTRACKER"},
+ {"name":"WEBHDFS"},
+ {"name":"WEBHCAT"},
+ {"name":"OOZIE"},
+ {"name":"WEBHBASE"},
+ {"name":"HIVE"},
+ {"name":"RESOURCEMANAGER"} ]
+} </pre>
+ </td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>PUT</td>
+ <td>Add a simple descriptor (and generate and deploy a full topology
descriptor).</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Request</td>
+ <td><pre>curl -iku admin:admin-password
{GatewayAPI}/descriptors/docker-sandbox \
+ -X PUT \
+ -H Content-Type:application/json \
+ -d "@docker-sandbox.json"</pre></td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Response</td>
+ <td><pre>HTTP 201 Created</pre></td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>DELETE</td>
+ <td>Delete a simple descriptor (and undeploy the associated topology)</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>Example Request</td>
+ <td><pre>curl -iku admin:admin-password
{GatewayAPI}/descriptors/docker-sandbox -X DELETE</pre></td>
+ <tr>
+ </tr>
+ <td> </td>
+ <td>Example Response</td>
+ <td>
+ <pre>{ "deleted" : "descriptor docker-sandbox" }</pre>
+ </td>
+ </tr>
+
+</table>
+
+
+
Modified: knox/trunk/books/0.14.0/config.md
URL:
http://svn.apache.org/viewvc/knox/trunk/books/0.14.0/config.md?rev=1814225&r1=1814224&r2=1814225&view=diff
==============================================================================
--- knox/trunk/books/0.14.0/config.md (original)
+++ knox/trunk/books/0.14.0/config.md Fri Nov 3 16:54:42 2017
@@ -353,6 +353,206 @@ This can be a comma separated list of ho
When mapping from external to internal host names the first internal host name
in the list is used.
+#### Simplified Topology Descriptors ####
+
+Simplified descriptors are a means to facilitate provider configuration
sharing and service endpoint discovery. Rather than
+editing an XML topology descriptor, it's possible to create a simpler YAML (or
JSON) descriptor specifying the desired contents
+of a topology, which will yield a full topology descriptor and deployment.
+
+##### Externalized Provider Configurations #####
+
+Sometimes, the same provider configuration is applied to multiple Knox
topologies. With the provider configuration externalized
+from the simple descriptors, a single configuration can be referenced by
multiple topologies. This helps reduce the duplication
+of configuration, and the need to update multiple configuration files when a
policy change is required. Updating a provider
+configuration will trigger an update to all those topologies that reference it.
+
+The contents of externalized provider configuration is identical to the
gateway element from a full topology descriptor. The only
+difference is that it's defined in its own XML file in
`{GATEWAY_HOME}/conf/shared-providers/`
+
+*Provider Configuration Example*
+
+ <gateway>
+ <provider>
+ <role>authentication</role>
+ <name>ShiroProvider</name>
+ <enabled>true</enabled>
+ <param>
+ <name>sessionTimeout</name>
+ <value>30</value>
+ </param>
+ <param>
+ <name>main.ldapRealm</name>
+
<value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value>
+ </param>
+ <param>
+ <name>main.ldapContextFactory</name>
+
<value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value>
+ </param>
+ <param>
+ <name>main.ldapRealm.contextFactory</name>
+ <value>$ldapContextFactory</value>
+ </param>
+ <param>
+ <name>main.ldapRealm.userDnTemplate</name>
+ <value>uid={0},ou=people,dc=hadoop,dc=apache,dc=org</value>
+ </param>
+ <param>
+ <name>main.ldapRealm.contextFactory.url</name>
+ <value>ldap://localhost:33389</value>
+ </param>
+ <param>
+
<name>main.ldapRealm.contextFactory.authenticationMechanism</name>
+ <value>simple</value>
+ </param>
+ <param>
+ <name>urls./**</name>
+ <value>authcBasic</value>
+ </param>
+ </provider>
+
+ <provider>
+ <role>identity-assertion</role>
+ <name>Default</name>
+ <enabled>true</enabled>
+ </provider>
+
+ <provider>
+ <role>hostmap</role>
+ <name>static</name>
+ <enabled>true</enabled>
+
<param><name>localhost</name><value>sandbox,sandbox.hortonworks.com</value></param>
+ </provider>
+ </gateway>
+
+##### Simplified Descriptor Files #####
+
+Simplified descriptors allow service URLs to be defined explicitly, just like
full topology descriptors. However, if URLs are
+omitted for a service, Knox will attempt to discover that service's URLs from
the Hadoop cluster. Currently, this behavior is
+only supported for clusters managed by Ambari. In any case, the simplified
descriptors are much more concise than a full
+topology descriptor.
+
+*Descriptor Properties*
+
+property |
description
+------------|-----------
+discovery-type|The discovery source type. (Currently, the only supported type
is *AMBARI*).
+discovery-address|The endpoint address for the discovery source.
+discovery-user|The username with permission to access the discovery source. If
omitted, then Knox will check for an alias named *ambari.discovery.user*, and
use its value if defined.
+discovery-pwd-alias|The alias of the password for the user with permission to
access the discovery source. If omitted, then Knox will check for an alias
named *ambari.discovery.password*, and use its value if defined.
+provider-config-ref|A reference to a provider configuration in
`{GATEWAY_HOME}/conf/shared-providers/`.
+cluster|The name of the cluster from which the topology service endpoints
should be determined.
+services|The collection of services to be included in the topology.
+
+
+Two file formats are supported for two distinct purposes.
+
+- YAML is intended for the individual hand-editing a simplified descriptor
because of its readability.
+- JSON is intended to be used for [API](#Admin+API) interactions.
+
+That being said, there is nothing preventing the hand-editing of files in the
JSON format. However, the API will *not* accept YAML files as input.
+
+*YAML Example* (based on the HDP Docker Sandbox)
+
+ ---
+ # Discovery source config
+ discovery-type : AMBARI
+ discovery-address : http://sandbox.hortonworks.com:8080
+
+ # If this is not specified, the alias ambari.discovery.user is checked for
a username
+ discovery-user : maria_dev
+
+ # If this is not specified, the default alias ambari.discovery.password is
used
+ discovery-pwd-alias : sandbox.discovery.password
+
+ # Provider config reference, the contents of which will be included in the
resulting topology descriptor
+ provider-config-ref : sandbox-providers
+
+ # The cluster for which the details should be discovered
+ cluster: Sandbox
+
+ # The services to declare in the resulting topology descriptor, whose URLs
will be discovered (unless a value is specified)
+ services:
+ - name: NAMENODE
+ - name: JOBTRACKER
+ - name: WEBHDFS
+ - name: WEBHCAT
+ - name: OOZIE
+ - name: WEBHBASE
+ - name: HIVE
+ - name: RESOURCEMANAGER
+ - name: KNOXSSO
+ params:
+ knoxsso.cookie.secure.only: true
+ knoxsso.token.ttl: 100000
+ - name: AMBARI
+ urls:
+ - http://sandbox.hortonworks.com:8080
+ - name: AMBARIUI
+ urls:
+ - http://sandbox.hortonworks.com:8080
+
+
+*JSON Example* (based on the HDP Docker Sandbox)
+
+ {
+ "discovery-type":"AMBARI",
+ "discovery-address":"http://sandbox.hortonworks.com:8080";,
+ "discovery-user":"maria_dev",
+ "discovery-pwd-alias":"sandbox.discovery.password",
+ "provider-config-ref":"sandbox-providers",
+ "cluster":"Sandbox",
+ "services":[
+ {"name":"NAMENODE"},
+ {"name":"JOBTRACKER"},
+ {"name":"WEBHDFS"},
+ {"name":"WEBHCAT"},
+ {"name":"OOZIE"},
+ {"name":"WEBHBASE"},
+ {"name":"HIVE"},
+ {"name":"RESOURCEMANAGER"},
+ {"name":"KNOXSSO",
+ "params":{
+ "knoxsso.cookie.secure.only":"true",
+ "knoxsso.token.ttl":"100000"
+ }
+ },
+ {"name":"AMBARI", "urls":["http://sandbox.hortonworks.com:8080"]},
+ {"name":"AMBARIUI", "urls":["http://sandbox.hortonworks.com:8080"]}
+ ]
+ }
+
+
+Both of these examples illustrate the specification of credentials for the
interaction with Ambari. If no credentials are specified, then the default
+aliases are queried. Use of the default aliases is sufficient for scenarios
where topology discovery will only interact with a single Ambari instance.
+For multiple Ambari instances however, it's most likely that each will require
different sets of credentials. The discovery-user and discovery-pwd-alias
+properties exist for this purpose. Note that whether using the default
credential aliases or specifying a custom password alias, these
+[aliases must be defined](#Alias+creation) prior to any attempt to deploy a
topology using a simplified descriptor.
+
+
+##### Deployment Directories #####
+
+Effecting topology changes is as simple as modifying files in two specific
directories.
+
+The `{GATEWAY_HOME}/conf/shared-providers/` directory is the location where
Knox looks for provider configurations. This directory is monitored for
changes, such
+that modifying a provider configuration file therein will trigger updates to
any referencing simplified descriptors in the
`{GATEWAY_HOME}/conf/descriptors/` directory.
+*Care should be taken when deleting these files if there are referencing
descriptors; any subsequent modifications of referencing descriptors will fail
when the deleted
+provider configuration cannot be found. The references should all be modified
before deleting the provider configuration.*
+
+Likewise, the `{GATEWAY_HOME}/conf/descriptors/` directory is monitored for
changes, such that adding or modifying a simplified descriptor file in this
directory will
+trigger the generation and deployment of a topology descriptor. Deleting a
descriptor from this directory will conversely result in the removal of the
previously-generated
+topology descriptor, and the associated topology will be undeployed.
+
+If the service details for a deployed (generated) topology are changed in the
cluster, then the Knox topology can be updated by 'touch'ing the simplified
descriptor. This
+will trigger discovery and regeneration/redeployment of the topology
descriptor.
+
+Note that deleting a generated topology descriptor from
`{GATEWAY_HOME}/conf/topologies/` is not sufficient for its removal. If the
source descriptor is modified, or Knox
+is restarted, the topology descriptor will be regenerated and deployed.
Removing generated topology descriptors should be done by removing the
associated simplified descriptor.
+For the same reason, editing generated topology descriptors is strongly
discouraged since they can be inadvertently overwritten.
+
+
+Another means by which these topology changes can be effected is the [Admin
API](#Admin+API).
+
+
#### Logging ####
If necessary you can enable additional logging by editing the
`log4j.properties` file in the `conf` directory.
