Modified: knox/site/books/knox-2-0-0/user-guide.html URL: http://svn.apache.org/viewvc/knox/site/books/knox-2-0-0/user-guide.html?rev=1904414&r1=1904413&r2=1904414&view=diff ============================================================================== --- knox/site/books/knox-2-0-0/user-guide.html (original) +++ knox/site/books/knox-2-0-0/user-guide.html Wed Oct 5 10:52:56 2022 @@ -990,6 +990,11 @@ https://{gateway-host}:{gateway-port}/{g <td><code>false</code></td> </tr> <tr> + <td><code>gateway.topology.redeploy.requires.changes</code> </td> + <td>If <code>true</code>, XML topology redeployment will happen only if the topology content is different than the actually deployed one. That is, a simple <code>touch</code> command will not yield in topology redeployment in this case. </td> + <td><code>false</code></td> + </tr> + <tr> <td><code>gateway.global.rules.services</code> </td> <td>Set the list of service names that have global rules, all services that are not in this list have rules that are treated as scoped to only to that service. </td> <td><code>"NAMENODE","JOBTRACKER", "WEBHDFS", "WEBHCAT", "OOZIE", "WEBHBASE", "HIVE", "RESOURCEMANAGER"</code></td> @@ -1019,6 +1024,16 @@ https://{gateway-host}:{gateway-port}/{g <td>When this feature is enabled and server managed state is enabled and Knox is presented with a valid token which is absent in server managed state, Knox will verify it without throwing an UnknownTokenException </td> <td><code>false</code></td> </tr> + <tr> + <td><code>gateway.jetty.max.form.content.size</code> </td> + <td>This optional parameter allows end-user to configure the form content in Knox’s embedded Jetty server that a request can process is limited to protect from Denial of Service attacks. The size in bytes is limited by Jetty’s ContextHandler#getMaxFormContentSize() or if there is no context then the “org.eclipse.jetty.server.Request.maxFormContentSize” attribute. </td> + <td><code>200000</code></td> + </tr> + <tr> + <td><code>gateway.jetty.max.form.keys</code> </td> + <td>This optional parameter allows end-user to configure the number of parameters keys is limited by Knox’s embedded Jetty’s ContextHandler#getMaxFormKeys() or if there is no context then the <code>org.eclipse.jetty.server.Request.maxFormKeys</code> attribute. </td> + <td><code>1000</code></td> + </tr> </tbody> </table> <h4><a id="Topology+Descriptors">Topology Descriptors</a> <a href="#Topology+Descriptors"><img src="markbook-section-link.png"/></a></h4> @@ -1420,16 +1435,42 @@ services: <description>The interval (in seconds) for polling Ambari for cluster configuration changes.</description> </property> +<!-- Cloudera Manager specific configuration --> + <property> <name>gateway.cluster.config.monitor.cm.enabled</name> <value>false</value> - <description>Enable/disable ClouderaManager cluster configuration monitoring.</description> + <description>Enable/disable Cloudera Manager cluster configuration monitoring.</description> </property> <property> <name>gateway.cluster.config.monitor.cm.interval</name> <value>60</value> - <description>The interval (in seconds) for polling ClouderaManager for cluster configuration changes.</description> + <description>The interval (in seconds) for polling Cloudera Manager for cluster configuration changes.</description> +</property> + +<property> + <name>gateway.cloudera.manager.descriptors.monitor.interval</name> + <value>-1</value> + <description>The interval (in milliseconds) for monitoring Hadoop XML resources in "GATEWAY_HOME/data/descriptors" with `.hxr` file postfix (see details below). If this property is set to a non-positive integer, monitoring of Hadoop XML resources is disabled.</description> +</property> + +<property> + <name>gateway.cloudera.manager.advanced.service.discovery.config.monitor.interval</name> + <value>-1</value> + <description>The interval (in milliseconds) for monitoring GATEWAY_HOME/conf/auto-discovery-advanced-configuration.properties (if exists) and notifies any AdvancedServiceDiscoveryConfigChangeListener if the file is changed since the last time it was loaded. Advanced configuration processing in Knox's service discovery flow helps fine-tuned service-level enablement on the topology level (see details below). If this property is set to a non-positive integer, this feature is disabled.</description> +</property> + +<property> + <name>gateway.cloudera.manager.service.discovery.maximum.retry.attemps</name> + <value>3</value> + <description>The maximum number of attempts Knox will try to connect to the configured Cloudera Manager cluster if there was a connection error.</description> +</property> + +<property> + <name>gateway.cloudera.manager.service.discovery.repository.cache.entry.ttl</name> + <value>60</value> + <description>Upon a successful Cloudera Manager service discovery event, Knox maintains an in-memory cache (repository) of discovered data (cluster, service, and role configs). This property indicates the entry TTL of this cache. See KNOX-2680 for more details.</description> </property> </code></pre> <p>Since service discovery supports multiple Ambari or ClouderaManager instances as discovery sources, multiple instances can be monitored for cluster configuration changes.</p> @@ -1452,6 +1493,81 @@ services: </code></pre> <p>Another <em>Sandbox</em> cluster, managed by a <strong>different</strong> Ambari instance, could simultaneously be monitored by the same gateway instance.</p> <p>Now, topologies can be kept in sync with their respective target cluster configurations, without administrator intervention or service interruption.</p> +<h5><a id="Hadoop+XML+resource+monitoring">Hadoop XML resource monitoring</a> <a href="#Hadoop+XML+resource+monitoring"><img src="markbook-section-link.png"/></a></h5> +<p>As described in <a href="https://issues.apache.org/jira/browse/KNOX-2160";>KNOX-2160,</a> to support topology management in Cloudera Manager it’s beneficial that Knox is able to process a descriptor that CM can generate natively. As of now, Cloudera Manager’s CSD framework is capable of producing a file of its parameters in the following formats: - Hadoop XML - properties - gflags</p> +<p>As the <code>gateway-site.xml</code> uses the Hadoop XML format it’s quite obvious that the first option is the one that fits Knox the most. One XML type descriptor file may contain one or more Knox descriptors using the following structure:</p> +<ul> + <li>the configuration <code>name</code> would indicate the descriptor (topology) name</li> + <li>the configuration <code>value</code> would list all properties of a Knox descriptor + <ul> + <li>service discovery related information (type, address, cluster, user/password alias)</li> + <li>services + <ul> + <li>name</li> + <li>url</li> + <li>version (optional)</li> + <li>parameters (optional)</li> + </ul> + </li> + <li>applications (optional) + <ul> + <li>name</li> + <li>parameters (optional)</li> + </ul> + </li> + </ul> + </li> +</ul> +<p>A sample descriptor file would look like this:</p> +<pre><code><configuration> + <property> + <name>topology1</name> + <value> + discoveryType=ClouderaManager; + discoveryAddress=http://host:123; + discoveryUser=user; + discoveryPasswordAlias=alias; + cluster=Cluster 1; + providerConfigRef=topology1-provider; + app:knoxauth:param1.name=param1.value; + app:KNOX; + HIVE:url=http://localhost:389; + HIVE:version=1.0; + HIVE:httpclient.connectionTimeout=5m; + HIVE:httpclient.socketTimeout=100m + </value> + </property> + <property> + <name>topology2</name> + <value> + discoveryType=ClouderaManager; + discoveryAddress=http://host:123; + discoveryUser=user; + discoveryPasswordAlias=alias; + cluster=Cluster 1; + providerConfigRef=topology2-provider; + app:KNOX; + HDFS.url=https://localhost:443; + HDFS:httpclient.connectionTimeout=5m; + HDFS:httpclient.socketTimeout=100m + </value> + </property> +</configuration> +</code></pre> +<p>Workflow:</p> +<ol> + <li>this kind of descriptor should also be placed in Knox’s descriptor directory using a <code>.hxr</code> file prefix (e.g. <code>cm-descriptors.hxr</code>)</li> + <li>once it’s added or modified Knox’s Hadoop XML resource monitor should parse the XML and build one or more instance(s) of <code>org.apache.knox.gateway.topology.simple.SimpleDescriptor</code></li> + <li>after the Java object(s) got created it (they) should be saved in the Knox descriptor directory in JSON format. As a result, the same monitor should parse the new/modified JSON descriptor(s) and re-deploys it (them) using the already existing mechanism</li> +</ol> +<h5><a id="Advanced+service+discovery+configuration+monitoring">Advanced service discovery configuration monitoring</a> <a href="#Advanced+service+discovery+configuration+monitoring"><img src="markbook-section-link.png"/></a></h5> +<p>With the above-discussed Hadoop XML resource monitoring Knox is capable of processing a Hadoop XML configuration file and turn its content into Knox providers.</p> +<p>Advanced service discovery configuration monitor extends that feature in a way it supports for the following use cases that are also Cloudera Manager integration specific:</p> +<p>1.) Cloudera Manager reports if auto-discovery is enabled for each known services. That is, a list of boolean properties can be generated by CM indicating if <code>SERVICE_X</code> is enabled or not in the following form: <code>gateway.auto.discovery.enabled.SERVICE_NAME=[true|false]</code></p> +<p>The new Hadoop XML configuration parser should take this information into account, and add a certain service into the generated Knox descriptor <strong>iff</strong> that service is explicitly enabled or there is no boolean flag within the CM generated properties with that service name (indicating an unknown - custom - service).</p> +<p>Additionally, a set of known (expected) topologies can be listed by CM. These topologies (Knox descriptors) should be populated with an enabled service (only with its name) even if that particular service was not listed in the new style Hadoop XML configuration file within the descriptor that matches any of the expected topology names.</p> +<p>2.) There are some services - mainly UI services - that are not working without some more required services in place (mainly their API counterpart). For instance: <code>RANGERUI</code> won’t work properly if <code>RANGER</code> is not available.</p> +<p>Knox’s Hadoop XML configuration parser was modified to exclude any service from the generated Knox descriptor unless - all required services are available (if any) - all required services are enabled (see the previous point)</p> <h5><a id="Remote+Configuration+Monitor">Remote Configuration Monitor</a> <a href="#Remote+Configuration+Monitor"><img src="markbook-section-link.png"/></a></h5> <p>In addition to monitoring local directories for provider configurations and simplified descriptors, the gateway similarly supports monitoring ZooKeeper.</p> <p>This monitor depends on a <a href="#Remote+Configuration+Registry+Clients">remote configuration registry client</a>, and that client must be specified by setting the following property in gateway-site.xml</p> @@ -1605,7 +1721,92 @@ trustworthiness. <h5><a id="Remote+Alias+Service+-+Zookeeper">Remote Alias Service - Zookeeper</a> <a href="#Remote+Alias+Service+-+Zookeeper"><img src="markbook-section-link.png"/></a></h5> <p>The Zookeeper remote alias service is designed to store aliases into Apache Zookeeper. It supports monitoring for remote aliases that are added, deleted or updated. The Zookeeper remote alias service is configured by turning the Remote Configuration Monitor on and setting <code>gateway.remote.alias.service.config.type</code> to <code>zookeeper</code> in gateway-site.xml. Knox needs to be restarted for this change to take effect. </p> <h4><a id="Logging">Logging</a> <a href="#Logging"><img src="markbook-section-link.png"/></a></h4> -<p>If necessary you can enable additional logging by editing the <code>log4j.properties</code> file in the <code>conf</code> directory. Changing the <code>rootLogger</code> value from <code>ERROR</code> to <code>DEBUG</code> will generate a large amount of debug logging. A number of useful, more fine loggers are also provided in the file.</p> +<p>If necessary you can enable additional logging by editing the <code>log4j2.xml</code> file in the <code>conf</code> directory. Changing the <code>rootLogger</code> value from <code>ERROR</code> to <code>DEBUG</code> will generate a large amount of debug logging. A number of useful, more fine loggers are also provided in the file.</p> +<p>With the 2.0 release, Knox uses Log4j 2 instead of Log4j 1. Apache Log4j 2 is the successor of Log4j 1, but it is incompatible with its predecessor. The main differene is that from now on Knox uses XML file format for configuring logging properties instead of <code>.property</code> files.</p> +<p>If you have an existing deployment with customized Log4j 1 <code>.property</code> files you will need to convert them to the new Log4j 2 format.</p> +<h5><a id="Launcher+changes">Launcher changes</a> <a href="#Launcher+changes"><img src="markbook-section-link.png"/></a></h5> +<p>The JVM property name that specifies the location of the Log4j configuration file, was changed to <code>log4j.configurationFile</code>.</p> +<p>For example</p> +<pre><code>-Dlog4j.configurationFile=gateway-log4j2.xml +</code></pre> +<p>Instead of</p> +<pre><code>-Dlog4j.configuration=conf/gateway-log4j.properties +</code></pre> +<h5><a id="Configuration+file+changes">Configuration file changes</a> <a href="#Configuration+file+changes"><img src="markbook-section-link.png"/></a></h5> +<p>Log4j 2 uses a different configuration file format than Log4j 1.</p> +<pre><code>app.log.dir=logs +app.log.file=${launcher.name}.log +log4j.rootLogger=ERROR, drfa +log4j.appender.stdout=org.apache.log4j.ConsoleAppender +log4j.appender.stdout.layout=org.apache.log4j.PatternLayout +log4j.appender.stdout.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n +log4j.appender.drfa=org.apache.log4j.DailyRollingFileAppender +log4j.appender.drfa.File=${app.log.dir}/${app.log.file} +log4j.appender.drfa.DatePattern=.yyyy-MM-dd +log4j.appender.drfa.layout=org.apache.log4j.PatternLayout +log4j.appender.drfa.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n +log4j.logger.org.apache.http.impl.conn=INFO +log4j.logger.org.apache.http.impl.client=INFO +log4j.logger.org.apache.http.client=INFO +</code></pre> +<p>For example the equivalent of the above Log4j 1 config file looks like this:</p> +<pre><code><Configuration> + <Properties> + <Property name="app.log.dir">logs</Property> + <Property name="app.log.file">${sys:launcher.name}.log</Property> + </Properties> + <Appenders> + <Console name="stdout" target="SYSTEM_OUT"> + <PatternLayout pattern="%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n" /> + </Console> + <RollingFile name="drfa" fileName="${app.log.dir}/${app.log.file}" filePattern="${app.log.dir}/${app.log.file}.%d{yyyy-MM-dd}"> + <PatternLayout pattern="%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n" /> + <TimeBasedTriggeringPolicy /> + </RollingFile> + </Appenders> + <Loggers> + <Logger name="org.apache.http.impl.client" level="INFO" /> + <Logger name="org.apache.http.client" level="INFO" /> + <Logger name="org.apache.http.impl.conn" level="INFO" /> + <Root level="ERROR"> + <AppenderRef ref="drfa" /> + </Root> + </Loggers> +</Configuration> +</code></pre> +<p>Log levels can be set on individual Java packages similarly as before. The Loggers inherit properties like logging level and appender types from their ancestors.</p> +<h5><a id="Log4j+Migration">Log4j Migration</a> <a href="#Log4j+Migration"><img src="markbook-section-link.png"/></a></h5> +<p>If you have an existing Knox installation which uses the default Logging settings, with no customizations, then you can simply upgrade to the new version and overwrite the <code>*-log4j.property</code> files with the <code>*-log4j2.xml</code> files.</p> +<p>Old Log4j 1.x property files:</p> +<pre><code>-rw-r--r-- 1 user group 3880 Sep 10 10:49 gateway-log4j.properties +-rw-r--r-- 1 user group 1481 Sep 10 10:49 knoxcli-log4j.properties +-rw-r--r-- 1 user group 1493 Sep 10 10:49 ldap-log4j.properties +-rw-r--r-- 1 user group 1436 Sep 10 10:49 shell-log4j.properties +</code></pre> +<p>The new Log4j 2 XML configuration files:</p> +<pre><code>-rw-r--r-- 1 user group 4619 Jan 22 2020 gateway-log4j2.xml +-rw-r--r-- 1 user group 1684 Jan 22 2020 knoxcli-log4j2.xml +-rw-r--r-- 1 user group 1765 Jan 22 2020 ldap-log4j2.xml +-rw-r--r-- 1 user group 1621 Jan 22 2020 shell-log4j2.xml +</code></pre> +<p>If you have a lot of customizations in place, you will need to convert the property files to XML file format.</p> +<p>There is a third party script that helps you with the conversion:</p> +<pre><code>https://github.com/mulesoft-labs/log4j2-migrator +</code></pre> +<p>The final result is not always 100% correct, you might need to do some manual adjustments.</p> +<p>Usage:</p> +<pre><code>$ groovy log4j2migrator.groovy log4j.properties > log4j2.xml +</code></pre> +<p>The scripts uses ââAsyncLoggers by default which requires the com.lmax:disruptor library which is not distributed with Knox by default. </p> +<p>To avoid having this dependency you can replace all the <code>AsyncLogger</code> tags to <code>Logger</code>s.</p> +<pre><code> $ groovy log4j2migrator.groovy log4j.properties > log4j2.xml | sed 's/AsyncLogger/Logger/g' +</code></pre> +<p>Pay attention to the custom date time patterns in the configuration file. The script doesn’t always convert them properly.</p> +<p>You can find more information about the Log4j 2 configuration file format at here: <a href="https://logging.apache.org/log4j/2.x/manual/configuration.html#XML";>https://logging.apache.org/log4j/2.x/manual/configuration.html#XML</a></p> +<h5><a id="Custom+Appenders+and+Layouts">Custom Appenders and Layouts</a> <a href="#Custom+Appenders+and+Layouts"><img src="markbook-section-link.png"/></a></h5> +<p>Custom Appenders or Layouts based on the Log4j 1 API, are not going to work with Log4j 2. Those need to be rewritten using the new API.</p> +<p>The <code>AppenderSkeleton</code> class does not exist in Log4j 2, you should extend from <code>AbstractAppender</code> instead.</p> +<p>You can read more about extending Log4j 2 at: <a href="https://logging.apache.org/log4j/2.x/manual/extending.html";>https://logging.apache.org/log4j/2.x/manual/extending.html</a></p> <h4><a id="Java+VM+Options">Java VM Options</a> <a href="#Java+VM+Options"><img src="markbook-section-link.png"/></a></h4> <p>TODO - Java VM options doc.</p> <h4><a id="Persisting+the+Master+Secret">Persisting the Master Secret</a> <a href="#Persisting+the+Master+Secret"><img src="markbook-section-link.png"/></a></h4> @@ -5151,6 +5352,11 @@ APACHE_HOME/bin/apachectl -k stop <td>Optional parameter to use a custom cookie name in the request if <code>knox.token.use.cookie = true</code>. </td> <td>hadoop-jwt</td> </tr> + <tr> + <td>knox.token.allowed.jws.types </td> + <td>With <a href="https://issues.apache.org/jira/browse/KNOX-2149";>KNOX-2149</a>, one can define their own JWKS URL which Knox can use for verification. Previous Knox implementations only supported JWTs with <code>"typ: JWT"</code> in their headers (or not type definition at all). In previous JOSE versions, there were other supported types such as <code>at+jwt</code> which Knox can support from now on. Please note, this configuration is only applied if token verification goes through the JWKS verification path. </td> + <td><code>JWT</code></td> + </tr> </tbody> </table> <p>The optional <code>knox.token.exp.server-managed</code> parameter indicates that Knox is managing the state of tokens it issues (e.g., expiration) external from the token, and this external state should be referenced when validating tokens. This parameter can be ommitted if the global default is configured in gateway-site (see <a href="#Gateway+Server+Configuration">gateway.knox.token.exp.server-managed</a>), and matches the requirements of this provider. Otherwise, this provider parameter overrides the gateway configuration for the provider’s deployment.</p> @@ -5597,7 +5803,8 @@ APACHE_HOME/bin/apachectl -k stop "knox.token.exp.server-managed": "false", "knox.token.renewer.whitelist": "admin", "knox.token.exp.renew-interval": "86400000", - "knox.token.exp.max-lifetime": "604800000" + "knox.token.exp.max-lifetime": "604800000", + "knox.token.type": "JWT" } } ] @@ -5647,6 +5854,16 @@ APACHE_HOME/bin/apachectl -k stop <td>This is an optional configuration parameter to specify the maximum allowed lifetime (milliseconds) of a token, after which renewal will not be permitted. </td> <td>604800000 (7 days) </td> </tr> + <tr> + <td>knox.token.type </td> + <td>If this is configured the generated JWT’s header will have this value as the <code>typ</code> property </td> + <td> </td> + </tr> + <tr> + <td>knox.token.impersonation.enabled </td> + <td>This is a <code>boolean</code> flag indicates if token impersonation is enabled </td> + <td><code>true</code></td> + </tr> </tbody> </table> <p>Note that server-managed token state can be configured for all KnoxToken service deployments in gateway-site (see <a href="#Gateway+Server+Configuration">gateway.knox.token.exp.server-managed</a>). If it is configured at the gateway level, then the associated service parameter, if configured, will override the gateway configuration.</p> @@ -5768,6 +5985,7 @@ APACHE_HOME/bin/apachectl -k stop <li><code>userName</code> - the logged-in user who generated the token</li> <li><code>enabled</code> - this is a boolean flag indicating that the given token is enabled or not (a <em>disabled</em> token cannot be used for authentication purposes)</li> <li><code>comment</code> - this is optional metadata, saved only if the user enters something in the <em>Comment</em> input field on the <em>Token Generation</em> page (see below)</li> + <li><code>createdBy</code> - if the token is an impersonated token, this metadata holds the name if the user who generated the token (see <em>Token impersonation</em> below)</li> </ul> </li> </ul> @@ -5787,6 +6005,7 @@ APACHE_HOME/bin/apachectl -k stop <li>comment: this is an <em>optional</em> input field that allows end-users to add meaningful comments (mnemonics) to their generated tokens. The maximum length is 255 characters.</li> <li>the <code>Configured maximum lifetime</code> informs the clients about the <code>knox.token.ttl</code> property set in the <code>homepage</code> topology (defaults to 120 days). If that property is not set (e.g. someone removes it from he homepage topology), Knox uses a hard-coded value of 30 seconds (aka. default Knox token TTL)</li> <li>Custom token lifetime can be set by adjusting the days/hours/minutes spinners. The default configuration will yield one hour.</li> + <li>Token impersonation: an optional free text input field tha makes it possible to generate a token for someone else.</li> <li>Clicking the Generate Token button will try to create a token for you.</li> </ul> <h5><a id="About+the+generated+token+TTL">About the generated token TTL</a> <a href="#About+the+generated+token+TTL"><img src="markbook-section-link.png"/></a></h5> @@ -5819,15 +6038,98 @@ APACHE_HOME/bin/apachectl -k stop <ul> <li><code>gateway.knox.token.limit.per.user</code> - indicates the maximum number of tokens a user can manage at the same time. <code>-1</code> means that users are allowed to create/manage as many tokens as they want. This configuration only applies when the server-managed token state is enabled either in <code>gateway-site</code> or at the <code>topology</code> level. Defaults to 10.</li> </ul> +<p>This behavior can be changed by setting the following <code>KNOXTOKEN</code> service-level parameter called <code>knox.token.user.limit.exceeded.action</code>. This property may have the following values:</p> +<ul> + <li><code>REMOVE_OLDEST</code> - if thatâs configured, the oldest token of the user, who the token is being generated for, will be removed</li> + <li><code>RETURN_ERROR</code> - if thatâs configured, Knox will return an error response with 403 error code (as it did in previous versions)</li> +</ul> +<p>The default value is <code>RETURN_ERROR</code>.</p> <h5><a id="Token+Management">Token Management</a> <a href="#Token+Management"><img src="markbook-section-link.png"/></a></h5> <p>In addition to the token generation UI, Knox comes with a Token Management UI where logged-in users can see all the active tokens that they generated before. That is, if a token got expired and was removed from the underlying token store, it won’t be displayed here.</p> <p><img src="knoxtokenmanagement_token_management_ui-1.png" /></p> -<p>On this page, you will see basic information about your generated token(s) and you can execute the following actions:</p> +<p>On this page, you will see two different tables:</p> +<ol> + <li>tokens of the logged-in user</li> + <li>tokens the logged-in user generated for other users</li> +</ol> +<p>Both tables display basic information about the generated token(s). In the first one end-users can execute the following actions:</p> <ol> <li>Enable/Disable - based on the current status, you can temporarily enable/disable a token. Please note that disabled tokens are not allowed to use for authentication purposes.</li> <li>Revoke - you can remove the token from the persistent store. Please note this action cannot be undone, once you revoked a token Knox will delete it from the in-memory cache as well as the underlying persistent token storage</li> </ol> <p>In order to refresh the table, you can use the <code>Refresh icon</code> above the table (if you generated tokens on another tab for instance).</p> +<h5><a id="Token+impersonation">Token impersonation</a> <a href="#Token+impersonation"><img src="markbook-section-link.png"/></a></h5> +<p>From v2.0.0, an authenticated user can generate token(s) on behalf of other user(s). This feature was implemented in <a href="https://issues.apache.org/jira/browse/KNOX-2714";>KNOX-2714</a> and requires the following configuration to work:</p> +<ul> + <li><code>knox.token.impersonation.enabled</code> - a <code>boolean</code> flag indicates if token impersonation is enabled. Defaults to <code>true</code></li> + <li><code>knox.token.proxyuser.$username.users</code> - indicates the list of users for whom <code>$username</code> is allowed to create tokens. It is possible to set this to a 1-element list using the <code>*</code> wildcard which means <code>$username</code> can generate tokens for everyone. Defaults to an empty list that is equivalent to <code>$username</code> is not allowed to impersonate anyone.</li> + <li><code>knox.token.proxyuser.$username.groups</code> - indicates the list of group names for whose members <code>$username</code> is allowed to create tokens for. It is possible to set this to a 1-element list using the <code>*</code> wildcard which means <code>$username</code> can generate tokens for members of any group. Defaults to an empty list that is equivalent to <code>$username</code> is not allowed to impersonate members from any group.</li> + <li><code>knox.token.proxyuser.$username.hosts</code> - indicates a list of hostnames from where the requests are allowed to be accepted in case the <code>doAs</code> parameter is used when creating Knox Tokens. It is possible to set this to a 1-element list using the <code>*</code> wildcard which means <code>$username</code> can generate tokens from any host. Defaults to an empty list that is equivalent to <code>$username</code> is not allowed to create tokens from any host.</li> +</ul> +<p>Please note this configuration is applied <strong>iff</strong> the newly introduced <code>doAs</code> query parameter is present in the token generation request AND if server-managed token state is enabled in the affected topology. You may want to check out <a href="https://github.com/apache/knox/pull/545";>GitHub Pull Request #545</a> for sample configuration.</p> +<p>On the token generation page end-users can generate tokens on behalf of other users by specifying the desired user name in the token <code>impersonation</code> field. The following screenshot sows a successful token generation for user <code>tom</code> (the logged in user is <code>admin</code>).</p> +<p><img src="knoxtokenmanagement_token_generation_ui-successful-doas.png" /></p> +<h5><a id="Token+metadata">Token metadata</a> <a href="#Token+metadata"><img src="markbook-section-link.png"/></a></h5> +<p>As indicated above, the <code>KNOXTOKEN</code> service maintains some hard-coded token metadata out-of-the-box: * userName * comment * enabled * passcode * createdBy (in case of impersonated tokens)</p> +<p>In v2.0.0, the Knox team implemented a change in this service that allows end-users to add accept query parameters starting with the <code>âmd_â</code> prefix and treat them as Knox Token Metadata.</p> +<p>For instance:</p> +<p><code>curl -iku admin:admin-password -X GET 'https://localhost:8443/gateway/sandbox/knoxtoken/api/v1/token?md_notebookName=accountantKnoxToken&md_souldBeRemovedBy=31March2022&md_otherMeaningfuMetadata=KnoxIsCool'</code> When such a token is created by Knox, we should save the following metadata too:</p> +<ul> + <li>notebookName=accountantKnoxToken</li> + <li>shouldBeRemovedBy=31March2022</li> + <li>otherMeaningfulMetadata=KnoxIsCool</li> +</ul> +<p>Itâs not only Knox can save these metadata, but the Knox’s existing <code>getUserTokens API</code> endpoint is able to fetch basic token information using the supplied metadata name besides the user name information.</p> +<p>It’s important to note the following: the <code>getUserTokens</code> API returns tokens if <em>any</em> of the supplied metadata exists for the given token. Metadata values may or may not be matched: you can either use the <code>*</code> wildcard to match all metadata values with a given name <em>or</em> you can further filter the stored metadata information by specifying the desired value.</p> +<p>For instance:</p> +<p><code>curl -iku admin:admin-password -X GET 'https://localhost:8443/gateway/sandbox/knoxtoken/api/v1/token/getUserTokens?userName=admin&md_notebookName=accountantKnoxToken&md_name=*'</code></p> +<p>will return all Knox tokens where metadata with <code>ânotebookNameâ</code> exists and equals <code>âaccountantKnoxTokenâ</code> OR metadata with <code>name</code> exists.</p> +<p>Another sample:</p> +<ol> + <li>Create token1 with <code>md_Name=reina&md_Score=50</code></li> + <li>Create token2 with <code>md_Name=mary&md_Score=100</code></li> + <li>Create token3 with <code>md_Name=mary&md_Score=20&md_Grade=A</code></li> +</ol> +<p>The following table shows the returned token(s) in case metadata filtering is added in the <code>getUserTokens</code> API:</p> +<table> + <thead> + <tr> + <th>Metadata </th> + <th>Token returned </th> + </tr> + </thead> + <tbody> + <tr> + <td>md_Name=reina</td> + <td>token1</td> + </tr> + <tr> + <td>md_Name=mary</td> + <td>token2 and token3</td> + </tr> + <tr> + <td>md_Score=100</td> + <td>token2</td> + </tr> + <tr> + <td>md_Name=mary&md_Score=20</td> + <td>token2 and token3</td> + </tr> + <tr> + <td>md_Name=mary&md_Name=reina</td> + <td>token1, token2 and token3</td> + </tr> + <tr> + <td>md_Name=*</td> + <td>token1, token2 and token3</td> + </tr> + <tr> + <td>md_Uknown=*</td> + <td>Empty list</td> + </tr> + </tbody> +</table> +<p>You may want to check out <a href="https://github.com/apache/knox/pull/542";>GitHub Pull Request #542</a> for sample <code>curl</code> commands.</p> <h3><a id="Mutual+Authentication+with+SSL">Mutual Authentication with SSL</a> <a href="#Mutual+Authentication+with+SSL"><img src="markbook-section-link.png"/></a></h3> <p>To establish a stronger trust relationship between client and server, we provide mutual authentication with SSL via client certs. This is particularly useful in providing additional validation for Preauthenticated SSO with HTTP Headers. Rather than just IP address validation, connections will only be accepted by Knox from clients presenting trusted certificates.</p> <p>This behavior is configured for the entire gateway instance within the gateway-site.xml file. All topologies deployed within the configured gateway instance will require incoming connections to present trusted client certificates during the SSL handshake. Otherwise, connections will be refused.</p> @@ -6023,7 +6325,7 @@ APACHE_HOME/bin/apachectl -k stop <p>Disabling auditing can be done by decreasing the log level for the Audit appender or setting it to <code>OFF</code>.</p> <h2><a id="Knox+Auth+Service">Knox Auth Service</a> <a href="#Knox+Auth+Service"><img src="markbook-section-link.png"/></a></h2> <h3><a id="Introduction">Introduction</a> <a href="#Introduction"><img src="markbook-section-link.png"/></a></h3> -<p>With workloads moving to containers Knox needs to support new ways of authentication needs of containers. As part of this effort, the Knox team developed a new internal service, called <code>KNOX-AUTH-SERVICE</code>. This service gathers a collection of public REST API endpoints that allows other developers to integrate Knox in their microservice/DEVOPS architectures using containers (such as docker or k8s).</p> +<p>With workloads moving to containers, it was necessary that Knox supports new ways of authentication needs of containers. As part of this effort, the Knox team developed a new internal service, called <code>KNOX-AUTH-SERVICE</code>. This service gathers a collection of public REST API endpoints that allows other developers to integrate Knox in their microservice/DEVOPS architectures using containers (such as docker or k9s).</p> <h3><a id="Configuration">Configuration</a> <a href="#Configuration"><img src="markbook-section-link.png"/></a></h3> <p>This service can be added to any Knox topology as an internal service as follows:</p> <pre><code><service> @@ -6050,10 +6352,10 @@ APACHE_HOME/bin/apachectl -k stop <h4><a id="auth/api/v1/pre">auth/api/v1/pre</a> <a href="#auth/api/v1/pre"><img src="markbook-section-link.png"/></a></h4> <p>This REST API endpoint has a very simple job: if a valid principal is found in the incoming request, a header is added to the response (by default <code>X-Knox-Actor-ID</code>) with the principal name. In addition, if the authenticated subject has group(s), it (they) will be added as comma-separated entries in the header(s) of the default form of <code>X-Knox-Actor-Groups-#num</code>. Each group header has a character limit of 1000 to keep them reasonably sized. The header names can be customized via the <code>preauth.auth.header.actor.id.name</code> and <code>preauth.auth.header.actor.groups.prefix</code> service parameters.</p> <p>End users may filter user groups by setting the <code>preauth.group.filter.pattern</code> service parameter to a valid regular expression. By default, all the user gropus are added into the <code>X-Knox-Actor-Groups-#num</code> header.</p> -<p>Sample <code>curl</code> command are available in this <a href="https://github.com/apache/knox/pull/625";>GitHub Pull Request</a>.</p> +<p>Sample <code>curl</code> commands are available in this <a href="https://github.com/apache/knox/pull/625";>GitHub Pull Request</a>.</p> <h4><a id="auth/api/v1/bearer">auth/api/v1/bearer</a> <a href="#auth/api/v1/bearer"><img src="markbook-section-link.png"/></a></h4> -<p>This REST API enpoint populates the HTTP “Authorization” header with the <code>Bearer Token</code> in the HTTP response obejctobtained from an environment variable. The current implementation assumes that the token is not rotated as it never gets exposed to the end-user. By default, the <code>BEARER_AUTH_TOKEN</code> environment variable is expected to hold the Bearer token. This can be customized by configuring the <code>auth.bearer.token.env</code> service parameter to the desired value.</p> -<p>Sample <code>curl</code> command are available in this <a href="https://github.com/apache/knox/pull/627";>GitHub Pull Request</a>.</p> +<p>This REST API enpoint populates the HTTP “Authorization” header with the <code>Bearer Token</code> in the HTTP response object obtained from an environment variable. The current implementation assumes that the token is not rotated as it never gets exposed to the end-user. By default, the <code>BEARER_AUTH_TOKEN</code> environment variable is expected to hold the Bearer token. This can be customized by configuring the <code>auth.bearer.token.env</code> service parameter to the desired value.</p> +<p>Sample <code>curl</code> commands are available in this <a href="https://github.com/apache/knox/pull/627";>GitHub Pull Request</a>.</p> <h2><a id="Client+Details">Client Details</a> <a href="#Client+Details"><img src="markbook-section-link.png"/></a></h2> <p>The KnoxShell release artifact provides a small footprint client environment that removes all unnecessary server dependencies, configuration, binary scripts, etc. It is comprised a couple different things that empower different sorts of users.</p> <ul>
Modified: knox/site/index.html URL: http://svn.apache.org/viewvc/knox/site/index.html?rev=1904414&r1=1904413&r2=1904414&view=diff ============================================================================== --- knox/site/index.html (original) +++ knox/site/index.html Wed Oct 5 10:52:56 2022 @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/index.md at 2022-09-08 + | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/index.md at 2022-10-05 | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20220908" /> + <meta name="Date-Revision-yyyymmdd" content="20221005" /> <meta http-equiv="Content-Language" content="en" /> <title>Knox Gateway – Announcing Apache Knox 1.6.1!</title> <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" /> @@ -40,7 +40,7 @@ <div id="breadcrumbs"> <ul class="breadcrumb"> - <li id="publishDate">Last Published: 2022-09-08</li> + <li id="publishDate">Last Published: 2022-10-05</li> </ul> </div> <div class="row-fluid"> Modified: knox/site/issue-management.html URL: http://svn.apache.org/viewvc/knox/site/issue-management.html?rev=1904414&r1=1904413&r2=1904414&view=diff ============================================================================== --- knox/site/issue-management.html (original) +++ knox/site/issue-management.html Wed Oct 5 10:52:56 2022 @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia Site Renderer 1.8.1 from org.apache.maven.plugins:maven-project-info-reports-plugin:3.0.0:issue-management at 2022-09-08 + | Generated by Apache Maven Doxia Site Renderer 1.8.1 from org.apache.maven.plugins:maven-project-info-reports-plugin:3.0.0:issue-management at 2022-10-05 | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20220908" /> + <meta name="Date-Revision-yyyymmdd" content="20221005" /> <meta http-equiv="Content-Language" content="en" /> <title>Knox Gateway – Issue Management</title> <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" /> @@ -40,7 +40,7 @@ <div id="breadcrumbs"> <ul class="breadcrumb"> - <li id="publishDate">Last Published: 2022-09-08</li> + <li id="publishDate">Last Published: 2022-10-05</li> </ul> </div> <div class="row-fluid"> Modified: knox/site/licenses.html URL: http://svn.apache.org/viewvc/knox/site/licenses.html?rev=1904414&r1=1904413&r2=1904414&view=diff ============================================================================== --- knox/site/licenses.html (original) +++ knox/site/licenses.html Wed Oct 5 10:52:56 2022 @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia Site Renderer 1.8.1 from org.apache.maven.plugins:maven-project-info-reports-plugin:3.0.0:licenses at 2022-09-08 + | Generated by Apache Maven Doxia Site Renderer 1.8.1 from org.apache.maven.plugins:maven-project-info-reports-plugin:3.0.0:licenses at 2022-10-05 | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20220908" /> + <meta name="Date-Revision-yyyymmdd" content="20221005" /> <meta http-equiv="Content-Language" content="en" /> <title>Knox Gateway – Project Licenses</title> <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" /> @@ -40,7 +40,7 @@ <div id="breadcrumbs"> <ul class="breadcrumb"> - <li id="publishDate">Last Published: 2022-09-08</li> + <li id="publishDate">Last Published: 2022-10-05</li> </ul> </div> <div class="row-fluid"> Modified: knox/site/mailing-lists.html URL: http://svn.apache.org/viewvc/knox/site/mailing-lists.html?rev=1904414&r1=1904413&r2=1904414&view=diff ============================================================================== --- knox/site/mailing-lists.html (original) +++ knox/site/mailing-lists.html Wed Oct 5 10:52:56 2022 @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia Site Renderer 1.8.1 from org.apache.maven.plugins:maven-project-info-reports-plugin:3.0.0:mailing-lists at 2022-09-08 + | Generated by Apache Maven Doxia Site Renderer 1.8.1 from org.apache.maven.plugins:maven-project-info-reports-plugin:3.0.0:mailing-lists at 2022-10-05 | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20220908" /> + <meta name="Date-Revision-yyyymmdd" content="20221005" /> <meta http-equiv="Content-Language" content="en" /> <title>Knox Gateway – Project Mailing Lists</title> <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" /> @@ -40,7 +40,7 @@ <div id="breadcrumbs"> <ul class="breadcrumb"> - <li id="publishDate">Last Published: 2022-09-08</li> + <li id="publishDate">Last Published: 2022-10-05</li> </ul> </div> <div class="row-fluid"> Modified: knox/site/project-info.html URL: http://svn.apache.org/viewvc/knox/site/project-info.html?rev=1904414&r1=1904413&r2=1904414&view=diff ============================================================================== --- knox/site/project-info.html (original) +++ knox/site/project-info.html Wed Oct 5 10:52:56 2022 @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia Site Renderer 1.8.1 from org.apache.maven.plugins:maven-site-plugin:3.7.1:CategorySummaryDocumentRenderer at 2022-09-08 + | Generated by Apache Maven Doxia Site Renderer 1.8.1 from org.apache.maven.plugins:maven-site-plugin:3.7.1:CategorySummaryDocumentRenderer at 2022-10-05 | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20220908" /> + <meta name="Date-Revision-yyyymmdd" content="20221005" /> <meta http-equiv="Content-Language" content="en" /> <title>Knox Gateway – Project Information</title> <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" /> @@ -40,7 +40,7 @@ <div id="breadcrumbs"> <ul class="breadcrumb"> - <li id="publishDate">Last Published: 2022-09-08</li> + <li id="publishDate">Last Published: 2022-10-05</li> </ul> </div> <div class="row-fluid"> Modified: knox/site/team.html URL: http://svn.apache.org/viewvc/knox/site/team.html?rev=1904414&r1=1904413&r2=1904414&view=diff ============================================================================== --- knox/site/team.html (original) +++ knox/site/team.html Wed Oct 5 10:52:56 2022 @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia Site Renderer 1.8.1 from org.apache.maven.plugins:maven-project-info-reports-plugin:3.0.0:team at 2022-09-08 + | Generated by Apache Maven Doxia Site Renderer 1.8.1 from org.apache.maven.plugins:maven-project-info-reports-plugin:3.0.0:team at 2022-10-05 | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20220908" /> + <meta name="Date-Revision-yyyymmdd" content="20221005" /> <meta http-equiv="Content-Language" content="en" /> <title>Knox Gateway – Project Team</title> <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" /> @@ -40,7 +40,7 @@ <div id="breadcrumbs"> <ul class="breadcrumb"> - <li id="publishDate">Last Published: 2022-09-08</li> + <li id="publishDate">Last Published: 2022-10-05</li> </ul> </div> <div class="row-fluid"> @@ -257,7 +257,22 @@ <td>Kevin Risden</td> <td><a href="krisden(at)apache(dot)org">krisden(at)apache(dot)org</a></td> <td>-</td> -<td>PMC</td></tr></table></div> +<td>PMC</td></tr> +<tr class="a"> +<td><img src="https://www.gravatar.com/avatar/3024dda44d5e80eb010f0e7e3942920a?d=mm&s=60"; alt="" /></td> +<td><a name="smolnar"></a>smolnar</td> +<td>Sandor Molnar</td> +<td><a href="smolnar(at)apache(dot)org">smolnar(at)apache(dot)org</a></td> +<td>Cloudera</td> +<td>PMC</td></tr> +<tr class="a"> +<td><img src="https://www.gravatar.com/avatar/3024dda44d5e80eb010f0e7e3942920a?d=mm&s=60"; alt="" /></td> +<td><a name="amagyar"></a>amagyar</td> +<td>Attila Magyar</td> +<td><a href="amagyar(at)apache(dot)org">amagyar(at)apache(dot)org</a></td> +<td>Cloudera</td> +<td>PMC</td></tr> +</table></div> <div class="section"> <h3><a name="Contributors"></a>Contributors</h3><a name="Contributors"></a> <p>The following additional people have contributed to this project through the way of suggestions, patches or documentation.</p> Modified: knox/trunk/books/2.0.0/config.md URL: http://svn.apache.org/viewvc/knox/trunk/books/2.0.0/config.md?rev=1904414&r1=1904413&r2=1904414&view=diff ============================================================================== --- knox/trunk/books/2.0.0/config.md (original) +++ knox/trunk/books/2.0.0/config.md Wed Oct 5 10:52:56 2022 @@ -167,12 +167,16 @@ Property | Description | Default `gateway.dispatch.whitelist` | A semicolon-delimited list of regular expressions for controlling to which endpoints Knox dispatches and redirects will be permitted. If `DEFAULT` is specified, or the property is omitted entirely, then a default domain-based whitelist will be derived from the Knox host. If `HTTPS_ONLY` is specified a default domain-based whitelist will be derived from the Knox host for only HTTPS urls. An empty value means no dispatches will be permitted. | null `gateway.dispatch.whitelist.services` | A comma-delimited list of service roles to which the `gateway.dispatch.whitelist` will be applied. | none `gateway.strict.topology.validation` | If true, topology XML files will be validated against the topology schema during redeploy | `false` +`gateway.topology.redeploy.requires.changes` | If `true`, XML topology redeployment will happen only if the topology content is different than the actually deployed one. That is, a simple `touch` command will not yield in topology redeployment in this case. | `false` `gateway.global.rules.services` | Set the list of service names that have global rules, all services that are not in this list have rules that are treated as scoped to only to that service. | `"NAMENODE","JOBTRACKER", "WEBHDFS", "WEBHCAT", "OOZIE", "WEBHBASE", "HIVE", "RESOURCEMANAGER"` `gateway.xforwarded.header.context.append.servicename` | Add service name to x-forward-context header for the defined list of services. | `LIVYSERVER` `gateway.knox.token.exp.server-managed` | Default server-managed token state configuration for all KnoxToken service and JWT provider deployments | `false` `gateway.knox.token.eviction.interval` | The period (seconds) about which the token state reaper will evict state for expired tokens. This configuration only applies when server-managed token state is enabled either in gateway-site or at the topology level. | `300` (5 minutes) `gateway.knox.token.eviction.grace.period` | A duration (seconds) beyond a token's expiration to wait before evicting its state. This configuration only applies when server-managed token state is enabled either in gateway-site or at the topology level. | `86400` (24 hours) `gateway.knox.token.permissive.validation` | When this feature is enabled and server managed state is enabled and Knox is presented with a valid token which is absent in server managed state, Knox will verify it without throwing an UnknownTokenException | `false` +`gateway.jetty.max.form.content.size` | This optional parameter allows end-user to configure the form content in Knox's embedded Jetty server that a request can process is limited to protect from Denial of Service attacks. The size in bytes is limited by Jetty's ContextHandler#getMaxFormContentSize() or if there is no context then the "org.eclipse.jetty.server.Request.maxFormContentSize" attribute. | `200000` +`gateway.jetty.max.form.keys` | This optional parameter allows end-user to configure the number of parameters keys is limited by Knox's embedded Jetty's ContextHandler#getMaxFormKeys() or if there is no context then the `org.eclipse.jetty.server.Request.maxFormKeys` attribute. | `1000` + #### Topology Descriptors #### @@ -669,26 +673,50 @@ The following properties in gateway-site <value>false</value> <description>Enable/disable Ambari cluster configuration monitoring.</description> </property> - + <property> <name>gateway.cluster.config.monitor.ambari.interval</name> <value>60</value> <description>The interval (in seconds) for polling Ambari for cluster configuration changes.</description> </property> - + + <!-- Cloudera Manager specific configuration --> + <property> <name>gateway.cluster.config.monitor.cm.enabled</name> <value>false</value> - <description>Enable/disable ClouderaManager cluster configuration monitoring.</description> + <description>Enable/disable Cloudera Manager cluster configuration monitoring.</description> </property> - + <property> <name>gateway.cluster.config.monitor.cm.interval</name> <value>60</value> - <description>The interval (in seconds) for polling ClouderaManager for cluster configuration changes.</description> + <description>The interval (in seconds) for polling Cloudera Manager for cluster configuration changes.</description> + </property> + + <property> + <name>gateway.cloudera.manager.descriptors.monitor.interval</name> + <value>-1</value> + <description>The interval (in milliseconds) for monitoring Hadoop XML resources in "GATEWAY_HOME/data/descriptors" with `.hxr` file postfix (see details below). If this property is set to a non-positive integer, monitoring of Hadoop XML resources is disabled.</description> </property> + <property> + <name>gateway.cloudera.manager.advanced.service.discovery.config.monitor.interval</name> + <value>-1</value> + <description>The interval (in milliseconds) for monitoring GATEWAY_HOME/conf/auto-discovery-advanced-configuration.properties (if exists) and notifies any AdvancedServiceDiscoveryConfigChangeListener if the file is changed since the last time it was loaded. Advanced configuration processing in Knox's service discovery flow helps fine-tuned service-level enablement on the topology level (see details below). If this property is set to a non-positive integer, this feature is disabled.</description> + </property> + <property> + <name>gateway.cloudera.manager.service.discovery.maximum.retry.attemps</name> + <value>3</value> + <description>The maximum number of attempts Knox will try to connect to the configured Cloudera Manager cluster if there was a connection error.</description> + </property> + + <property> + <name>gateway.cloudera.manager.service.discovery.repository.cache.entry.ttl</name> + <value>60</value> + <description>Upon a successful Cloudera Manager service discovery event, Knox maintains an in-memory cache (repository) of discovered data (cluster, service, and role configs). This property indicates the entry TTL of this cache. See KNOX-2680 for more details.</description> + </property> Since service discovery supports multiple Ambari or ClouderaManager instances as discovery sources, multiple instances can be monitored for cluster configuration changes. @@ -714,6 +742,91 @@ Another *Sandbox* cluster, managed by a Now, topologies can be kept in sync with their respective target cluster configurations, without administrator intervention or service interruption. +##### Hadoop XML resource monitoring +As described in [KNOX-2160,](https://issues.apache.org/jira/browse/KNOX-2160) to support topology management in Cloudera Manager it's beneficial that Knox is able to process a descriptor that CM can generate natively. As of now, Cloudera Manager's CSD framework is capable of producing a file of its parameters in the following formats: + - Hadoop XML + - properties + - gflags + +As the `gateway-site.xml` uses the Hadoop XML format it's quite obvious that the first option is the one that fits Knox the most. +One XML type descriptor file may contain one or more Knox descriptors using the following structure: + + - the configuration `name` would indicate the descriptor (topology) name + - the configuration `value` would list all properties of a Knox + descriptor + - service discovery related information (type, address, + cluster, user/password alias) + - services + - name + - url + - version (optional) + - parameters (optional) + - applications (optional) + - name + - parameters (optional) + +A sample descriptor file would look like this: + + + <configuration> + <property> + <name>topology1</name> + <value> + discoveryType=ClouderaManager; + discoveryAddress=http://host:123; + discoveryUser=user; + discoveryPasswordAlias=alias; + cluster=Cluster 1; + providerConfigRef=topology1-provider; + app:knoxauth:param1.name=param1.value; + app:KNOX; + HIVE:url=http://localhost:389; + HIVE:version=1.0; + HIVE:httpclient.connectionTimeout=5m; + HIVE:httpclient.socketTimeout=100m + </value> + </property> + <property> + <name>topology2</name> + <value> + discoveryType=ClouderaManager; + discoveryAddress=http://host:123; + discoveryUser=user; + discoveryPasswordAlias=alias; + cluster=Cluster 1; + providerConfigRef=topology2-provider; + app:KNOX; + HDFS.url=https://localhost:443; + HDFS:httpclient.connectionTimeout=5m; + HDFS:httpclient.socketTimeout=100m + </value> + </property> + </configuration> + +Workflow: + +1. this kind of descriptor should also be placed in Knox's descriptor directory using a `.hxr` file prefix (e.g. `cm-descriptors.hxr`) +2. once it's added or modified Knox's Hadoop XML resource monitor should parse the XML and build one or more instance(s) of `org.apache.knox.gateway.topology.simple.SimpleDescriptor` +3. after the Java object(s) got created it (they) should be saved in the Knox descriptor directory in JSON format. As a result, the same monitor should parse the new/modified JSON descriptor(s) and re-deploys it (them) using the already existing mechanism + +##### Advanced service discovery configuration monitoring + +With the above-discussed Hadoop XML resource monitoring Knox is capable of processing a Hadoop XML configuration file and turn its content into Knox providers. + +Advanced service discovery configuration monitor extends that feature in a way it supports for the following use cases that are also Cloudera Manager integration specific: + +1.) Cloudera Manager reports if auto-discovery is enabled for each known services. That is, a list of boolean properties can be generated by CM indicating if `SERVICE_X` is enabled or not in the following form: `gateway.auto.discovery.enabled.SERVICE_NAME=[true|false]` + +The new Hadoop XML configuration parser should take this information into account, and add a certain service into the generated Knox descriptor **iff** that service is explicitly enabled or there is no boolean flag within the CM generated properties with that service name (indicating an unknown - custom - service). + +Additionally, a set of known (expected) topologies can be listed by CM. These topologies (Knox descriptors) should be populated with an enabled service (only with its name) even if that particular service was not listed in the new style Hadoop XML configuration file within the descriptor that matches any of the expected topology names. + +2.) There are some services - mainly UI services - that are not working without some more required services in place (mainly their API counterpart). For instance: `RANGERUI` won't work properly if `RANGER` is not available. + +Knox's Hadoop XML configuration parser was modified to exclude any service from the generated Knox descriptor unless + - all required services are available (if any) + - all required services are enabled (see the previous point) + ##### Remote Configuration Monitor ##### Modified: knox/trunk/books/2.0.0/config_knox_token.md URL: http://svn.apache.org/viewvc/knox/trunk/books/2.0.0/config_knox_token.md?rev=1904414&r1=1904413&r2=1904414&view=diff ============================================================================== --- knox/trunk/books/2.0.0/config_knox_token.md (original) +++ knox/trunk/books/2.0.0/config_knox_token.md Wed Oct 5 10:52:56 2022 @@ -20,7 +20,8 @@ The Knox Token Service configuration can "knox.token.exp.server-managed": "false", "knox.token.renewer.whitelist": "admin", "knox.token.exp.renew-interval": "86400000", - "knox.token.exp.max-lifetime": "604800000" + "knox.token.exp.max-lifetime": "604800000", + "knox.token.type": "JWT" } } ] @@ -36,6 +37,8 @@ knox.token.exp.server-managed | This is knox.token.renewer.whitelist | This is an optional configuration parameter to authorize the comma-separated list of users to invoke the associated token renewal and revocation APIs. | | knox.token.exp.renew-interval | This is an optional configuration parameter to specify the amount of time (milliseconds) to be added to a token's TTL when a renewal request is approved. | 86400000 (24 hours) | knox.token.exp.max-lifetime | This is an optional configuration parameter to specify the maximum allowed lifetime (milliseconds) of a token, after which renewal will not be permitted. | 604800000 (7 days) | +knox.token.type | If this is configured the generated JWT's header will have this value as the `typ` property | | +knox.token.impersonation.enabled | This is a `boolean` flag indicates if token impersonation is enabled | `true` Note that server-managed token state can be configured for all KnoxToken service deployments in gateway-site (see [gateway.knox.token.exp.server-managed](#Gateway+Server+Configuration)). If it is configured at the gateway level, then the associated service parameter, if configured, will override the gateway configuration. @@ -199,6 +202,7 @@ As you can see, there are only 2 tables: authentication purposes) * `comment` - this is optional metadata, saved only if the user enters something in the _Comment_ input field on the _Token Generation_ page (see below) + * `createdBy` - if the token is an impersonated token, this metadata holds the name if the user who generated the token (see _Token impersonation_ below) ##### Generating a token @@ -216,6 +220,7 @@ The following sections are displayed on * comment: this is an _optional_ input field that allows end-users to add meaningful comments (mnemonics) to their generated tokens. The maximum length is 255 characters. * the `Configured maximum lifetime` informs the clients about the `knox.token.ttl` property set in the `homepage` topology (defaults to 120 days). If that property is not set (e.g. someone removes it from he homepage topology), Knox uses a hard-coded value of 30 seconds (aka. default Knox token TTL) * Custom token lifetime can be set by adjusting the days/hours/minutes spinners. The default configuration will yield one hour. +* Token impersonation: an optional free text input field tha makes it possible to generate a token for someone else. * Clicking the Generate Token button will try to create a token for you. ##### About the generated token TTL @@ -257,15 +262,96 @@ The above error message indicates a fail - `gateway.knox.token.limit.per.user` - indicates the maximum number of tokens a user can manage at the same time. `-1` means that users are allowed to create/manage as many tokens as they want. This configuration only applies when the server-managed token state is enabled either in `gateway-site` or at the `topology` level. Defaults to 10. +This behavior can be changed by setting the following `KNOXTOKEN` service-level parameter called `knox.token.user.limit.exceeded.action`. This property may have the following values: + +- `REMOVE_OLDEST` - if thatâs configured, the oldest token of the user, who the token is being generated for, will be removed +- `RETURN_ERROR` - if thatâs configured, Knox will return an error response with 403 error code (as it did in previous versions) + +The default value is `RETURN_ERROR`. + ##### Token Management In addition to the token generation UI, Knox comes with a Token Management UI where logged-in users can see all the active tokens that they generated before. That is, if a token got expired and was removed from the underlying token store, it won't be displayed here.  -On this page, you will see basic information about your generated token(s) and you can execute the following actions: +On this page, you will see two different tables: + +1. tokens of the logged-in user +2. tokens the logged-in user generated for other users + +Both tables display basic information about the generated token(s). In the first one end-users can execute the following actions: 1. Enable/Disable - based on the current status, you can temporarily enable/disable a token. Please note that disabled tokens are not allowed to use for authentication purposes. 2. Revoke - you can remove the token from the persistent store. Please note this action cannot be undone, once you revoked a token Knox will delete it from the in-memory cache as well as the underlying persistent token storage In order to refresh the table, you can use the `Refresh icon` above the table (if you generated tokens on another tab for instance). + +##### Token impersonation + +From v2.0.0, an authenticated user can generate token(s) on behalf of other user(s). This feature was implemented in [KNOX-2714](https://issues.apache.org/jira/browse/KNOX-2714) and requires the following configuration to work: + +* `knox.token.impersonation.enabled` - a `boolean` flag indicates if token impersonation is enabled. Defaults to `true` +* `knox.token.proxyuser.$username.users` - indicates the list of users for whom `$username` is allowed to create tokens. It is possible to set this to a 1-element list using the `*` wildcard which means `$username` can generate tokens for everyone. Defaults to an empty list that is equivalent to `$username` is not allowed to impersonate anyone. +* `knox.token.proxyuser.$username.groups` - indicates the list of group names for whose members `$username` is allowed to create tokens for. It is possible to set this to a 1-element list using the `*` wildcard which means `$username` can generate tokens for members of any group. Defaults to an empty list that is equivalent to `$username` is not allowed to impersonate members from any group. +* `knox.token.proxyuser.$username.hosts` - indicates a list of hostnames from where the requests are allowed to be accepted in case the `doAs` parameter is used when creating Knox Tokens. It is possible to set this to a 1-element list using the `*` wildcard which means `$username` can generate tokens from any host. Defaults to an empty list that is equivalent to `$username` is not allowed to create tokens from any host. + +Please note this configuration is applied **iff** the newly introduced `doAs` query parameter is present in the token generation request AND if server-managed token state is enabled in the affected topology. +You may want to check out [GitHub Pull Request #545](https://github.com/apache/knox/pull/545) for sample configuration. + +On the token generation page end-users can generate tokens on behalf of other users by specifying the desired user name in the token `impersonation` field. The following screenshot sows a successful token generation for user `tom` (the logged in user is `admin`). + +  + +##### Token metadata + +As indicated above, the `KNOXTOKEN` service maintains some hard-coded token metadata out-of-the-box: +* userName +* comment +* enabled +* passcode +* createdBy (in case of impersonated tokens) + +In v2.0.0, the Knox team implemented a change in this service that allows end-users to add accept query parameters starting with the `âmd_â` prefix and treat them as Knox Token Metadata. + +For instance: + +`curl -iku admin:admin-password -X GET 'https://localhost:8443/gateway/sandbox/knoxtoken/api/v1/token?md_notebookName=accountantKnoxToken&md_souldBeRemovedBy=31March2022&md_otherMeaningfuMetadata=KnoxIsCool'` +When such a token is created by Knox, we should save the following metadata too: + +* notebookName=accountantKnoxToken +* shouldBeRemovedBy=31March2022 +* otherMeaningfulMetadata=KnoxIsCool + +Itâs not only Knox can save these metadata, but the Knox's existing `getUserTokens API` endpoint is able to fetch basic token information using the supplied metadata name besides the user name information. + +It's important to note the following: the `getUserTokens` API returns tokens if *any* of the supplied metadata exists for the given token. Metadata values may or may not be matched: you can either use the `*` wildcard to match all metadata values with a given name *or* you can further filter the stored metadata information by specifying the desired value. + +For instance: + +`curl -iku admin:admin-password -X GET 'https://localhost:8443/gateway/sandbox/knoxtoken/api/v1/token/getUserTokens?userName=admin&md_notebookName=accountantKnoxToken&md_name=*'` + +will return all Knox tokens where metadata with `ânotebookNameâ` exists and equals `âaccountantKnoxTokenâ` OR metadata with `name` exists. + + +Another sample: + +1. Create token1 with `md_Name=reina&md_Score=50` +2. Create token2 with `md_Name=mary&md_Score=100` +3. Create token3 with `md_Name=mary&md_Score=20&md_Grade=A` + +The following table shows the returned token(s) in case metadata filtering is added in the `getUserTokens` API: + +| Metadata | Token returned | +|--|--| +|md_Name=reina|token1| +|md_Name=mary|token2 and token3| +|md_Score=100|token2| +|md_Name=mary&md_Score=20|token2 and token3| +|md_Name=mary&md_Name=reina|token1, token2 and token3| +|md_Name=*|token1, token2 and token3| +|md_Uknown=*|Empty list| + +You may want to check out [GitHub Pull Request #542](https://github.com/apache/knox/pull/542) for sample `curl` commands. + + Modified: knox/trunk/books/2.0.0/config_knoxauth_service.md URL: http://svn.apache.org/viewvc/knox/trunk/books/2.0.0/config_knoxauth_service.md?rev=1904414&r1=1904413&r2=1904414&view=diff ============================================================================== --- knox/trunk/books/2.0.0/config_knoxauth_service.md (original) +++ knox/trunk/books/2.0.0/config_knoxauth_service.md Wed Oct 5 10:52:56 2022 @@ -1,7 +1,7 @@ ## Knox Auth Service ### Introduction -With workloads moving to containers Knox needs to support new ways of authentication needs of containers. As part of this effort, the Knox team developed a new internal service, called `KNOX-AUTH-SERVICE`. This service gathers a collection of public REST API endpoints that allows other developers to integrate Knox in their microservice/DEVOPS architectures using containers (such as docker or k8s). +With workloads moving to containers, it was necessary that Knox supports new ways of authentication needs of containers. As part of this effort, the Knox team developed a new internal service, called `KNOX-AUTH-SERVICE`. This service gathers a collection of public REST API endpoints that allows other developers to integrate Knox in their microservice/DEVOPS architectures using containers (such as docker or k9s). ### Configuration @@ -36,10 +36,11 @@ This REST API endpoint has a very simple End users may filter user groups by setting the `preauth.group.filter.pattern` service parameter to a valid regular expression. By default, all the user gropus are added into the `X-Knox-Actor-Groups-#num` header. -Sample `curl` command are available in this [GitHub Pull Request](https://github.com/apache/knox/pull/625). +Sample `curl` commands are available in this [GitHub Pull Request](https://github.com/apache/knox/pull/625). #### auth/api/v1/bearer -This REST API enpoint populates the HTTP "Authorization" header with the `Bearer Token` in the HTTP response obejctobtained from an environment variable. The current implementation assumes that the token is not rotated as it never gets exposed to the end-user. By default, the `BEARER_AUTH_TOKEN` environment variable is expected to hold the Bearer token. This can be customized by configuring the `auth.bearer.token.env` service parameter to the desired value. +This REST API enpoint populates the HTTP "Authorization" header with the `Bearer Token` in the HTTP response object obtained from an environment variable. The current implementation assumes that the token is not rotated as it never gets exposed to the end-user. By default, the `BEARER_AUTH_TOKEN` environment variable is expected to hold the Bearer token. This can be customized by configuring the `auth.bearer.token.env` service parameter to the desired value. + +Sample `curl` commands are available in this [GitHub Pull Request](https://github.com/apache/knox/pull/627). -Sample `curl` command are available in this [GitHub Pull Request](https://github.com/apache/knox/pull/627). \ No newline at end of file Modified: knox/trunk/books/2.0.0/config_sso_cookie_provider.md URL: http://svn.apache.org/viewvc/knox/trunk/books/2.0.0/config_sso_cookie_provider.md?rev=1904414&r1=1904413&r2=1904414&view=diff ============================================================================== --- knox/trunk/books/2.0.0/config_sso_cookie_provider.md (original) +++ knox/trunk/books/2.0.0/config_sso_cookie_provider.md Wed Oct 5 10:52:56 2022 @@ -114,6 +114,7 @@ knox.token.exp.server-managed | Optional knox.token.verification.pem | Optional parameter that specifies public key used to validate the token. The key must be in PEM encoded format excluding the header and footer lines.| N/A knox.token.use.cookie | Optional parameter that indicates if the JWT token can be retrieved from an HTTP cookie instead of the Authorization header. If this is set to `true`, then Knox will first check if the `hadoop-jwt` cookie (the cookie name is configurable) is available in the request and, if that's the case, Knox will try to fetch a JWT from that cookie. If the cookie is not present in the request, Knox will continue its authentication flow using the Authorization header. If the cookie is there, but it holds an invalid JWT, then authentication will fail. Sample use cases and `curl` commands are available in this [GitHub Pull Request](https://github.com/apache/knox/pull/623). | false knox.token.cookie.name | Optional parameter to use a custom cookie name in the request if `knox.token.use.cookie = true`. | hadoop-jwt +knox.token.allowed.jws.types | With [KNOX-2149](https://issues.apache.org/jira/browse/KNOX-2149), one can define their own JWKS URL which Knox can use for verification. Previous Knox implementations only supported JWTs with `"typ: JWT"` in their headers (or not type definition at all). In previous JOSE versions, there were other supported types such as `at+jwt` which Knox can support from now on. Please note, this configuration is only applied if token verification goes through the JWKS verification path. | `JWT` The optional `knox.token.exp.server-managed` parameter indicates that Knox is managing the state of tokens it issues (e.g., expiration) external from the token, and this external state should be referenced when validating tokens. This parameter can be ommitted if the global default is configured in gateway-site (see [gateway.knox.token.exp.server-managed](#Gateway+Server+Configuration)), and matches the requirements of this provider. Otherwise, this provider parameter overrides the gateway configuration for the provider's deployment. Modified: knox/trunk/books/static/knoxtokenmanagement_token_generation_ui-1.png URL: http://svn.apache.org/viewvc/knox/trunk/books/static/knoxtokenmanagement_token_generation_ui-1.png?rev=1904414&r1=1904413&r2=1904414&view=diff ============================================================================== Binary files - no diff available. Modified: knox/trunk/books/static/knoxtokenmanagement_token_generation_ui-fail.png URL: http://svn.apache.org/viewvc/knox/trunk/books/static/knoxtokenmanagement_token_generation_ui-fail.png?rev=1904414&r1=1904413&r2=1904414&view=diff ============================================================================== Binary files - no diff available. Added: knox/trunk/books/static/knoxtokenmanagement_token_generation_ui-successful-doas.png URL: http://svn.apache.org/viewvc/knox/trunk/books/static/knoxtokenmanagement_token_generation_ui-successful-doas.png?rev=1904414&view=auto ============================================================================== Binary file - no diff available. Propchange: knox/trunk/books/static/knoxtokenmanagement_token_generation_ui-successful-doas.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Modified: knox/trunk/books/static/knoxtokenmanagement_token_generation_ui-successful.png URL: http://svn.apache.org/viewvc/knox/trunk/books/static/knoxtokenmanagement_token_generation_ui-successful.png?rev=1904414&r1=1904413&r2=1904414&view=diff ============================================================================== Binary files - no diff available. Modified: knox/trunk/books/static/knoxtokenmanagement_token_management_ui-1.png URL: http://svn.apache.org/viewvc/knox/trunk/books/static/knoxtokenmanagement_token_management_ui-1.png?rev=1904414&r1=1904413&r2=1904414&view=diff ============================================================================== Binary files - no diff available.
