Added: knox/trunk/books/2.0.0/config_knox_token.md URL: http://svn.apache.org/viewvc/knox/trunk/books/2.0.0/config_knox_token.md?rev=1899392&view=auto ============================================================================== --- knox/trunk/books/2.0.0/config_knox_token.md (added) +++ knox/trunk/books/2.0.0/config_knox_token.md Wed Mar 30 15:22:57 2022 @@ -0,0 +1,271 @@ +## KnoxToken Configuration + +### Introduction +--- + +The Knox Token Service enables the ability for clients to acquire the same JWT token that is used for KnoxSSO with WebSSO flows for UIs to be used for accessing REST APIs. By acquiring the token and setting it as a Bearer token on a request, a client is able to access REST APIs that are protected with the JWTProvider federation provider. + +This section describes the overall setup requirements and options for KnoxToken service. + +### KnoxToken service +The Knox Token Service configuration can be configured in any descriptor/topology, tailored to issue tokens to authenticated users, and constrain the usage of the tokens in a number of ways. + + "services": [ + { + "name": "KNOXTOKEN", + "params": { + "knox.token.ttl": "36000000", + "knox.token.audiences": "tokenbased", + "knox.token.target.url": "https://localhost:8443/gateway/tokenbased";, + "knox.token.exp.server-managed": "false", + "knox.token.renewer.whitelist": "admin", + "knox.token.exp.renew-interval": "86400000", + "knox.token.exp.max-lifetime": "604800000" + } + } + ] + +#### KnoxToken Configuration Parameters + +Parameter | Description | Default | +-------------------------------- |------------ |----------- | +knox.token.ttl | This indicates the lifespan (milliseconds) of the token. Once it expires a new token must be acquired from KnoxToken service. The 36000000 in the topology above gives you 10 hrs. | 30000 (30 seconds) | +knox.token.audiences | This is a comma-separated list of audiences to add to the JWT token. This is used to ensure that a token received by a participating application knows that the token was intended for use with that application. It is optional. In the event that an endpoint has expected audiences and they are not present the token must be rejected. In the event where the token has audiences and the endpoint has none expected then the token is accepted.| empty | +knox.token.target.url | This is an optional configuration parameter to indicate the intended endpoint for which the token may be used. The KnoxShell token credential collector can pull this URL from a knoxtokencache file to be used in scripts. This eliminates the need to prompt for or hardcode endpoints in your scripts. | n/a | +knox.token.exp.server-managed | This is an optional configuration parameter to enable/disable server-managed token state, to support the associated token renewal and revocation APIs. | false | +knox.token.renewer.whitelist | This is an optional configuration parameter to authorize the comma-separated list of users to invoke the associated token renewal and revocation APIs. | | +knox.token.exp.renew-interval | This is an optional configuration parameter to specify the amount of time (milliseconds) to be added to a token's TTL when a renewal request is approved. | 86400000 (24 hours) | +knox.token.exp.max-lifetime | This is an optional configuration parameter to specify the maximum allowed lifetime (milliseconds) of a token, after which renewal will not be permitted. | 604800000 (7 days) | + +Note that server-managed token state can be configured for all KnoxToken service deployments in gateway-site (see [gateway.knox.token.exp.server-managed](#Gateway+Server+Configuration)). If it is configured at the gateway level, then the associated service parameter, if configured, will override the gateway configuration. + +Adding the KnoxToken configuration shown above to a topology that is protected with the ShrioProvider is a very simple and effective way to expose an endpoint from which a Knox token can be requested. Once it is acquired it may be used to access resources at intended endpoints until it expires. + +The following curl command can be used to acquire a token from the Knox Token service as configured in the sandbox topology: + + curl -ivku guest:guest-password https://localhost:8443/gateway/sandbox/knoxtoken/api/v1/token + +Resulting in a JSON response that contains the token, the expiration and the optional target endpoint: + + `{"access_token":"eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJndWVzdCIsImF1ZCI6InRva2VuYmFzZWQiLCJpc3MiOiJLTk9YU1NPIiwiZXhwIjoxNDg5OTQyMTg4fQ.bcqSK7zMnABEM_HVsm3oWNDrQ_ei7PcMI4AtZEERY9LaPo9dzugOg3PA5JH2BRF-lXM3tuEYuZPaZVf8PenzjtBbuQsCg9VVImuu2r1YNVJlcTQ7OV-eW50L6OTI0uZfyrFwX6C7jVhf7d7YR1NNxs4eVbXpS1TZ5fDIRSfU3MU","target_url":"https://localhost:8443/gateway/tokenbased","token_type":"Bearer ","expires_in":1489942188233}` + +The following curl example shows how to add a bearer token to an Authorization header: + + curl -ivk -H "Authorization: Bearer eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJndWVzdCIsImF1ZCI6InRva2VuYmFzZWQiLCJpc3MiOiJLTk9YU1NPIiwiZXhwIjoxNDg5OTQyMTg4fQ.bcqSK7zMnABEM_HVsm3oWNDrQ_ei7PcMI4AtZEERY9LaPo9dzugOg3PA5JH2BRF-lXM3tuEYuZPaZVf8PenzjtBbuQsCg9VVImuu2r1YNVJlcTQ7OV-eW50L6OTI0uZfyrFwX6C7jVhf7d7YR1NNxs4eVbXpS1TZ5fDIRSfU3MU" https://localhost:8443/gateway/tokenbased/webhdfs/v1/tmp?op=LISTSTATUS + +#### KnoxToken Renewal and Revocation + +The KnoxToken service supports the renewal and explicit revocation of tokens it has issued. +Support for both requires server-managed token state to be enabled with at least one renewer white-listed. + +##### Renewal + + curl -ivku admin:admin-password -X POST -d $TOKEN 'https://localhost:8443/gateway/sandbox/knoxtoken/api/v1/token/renew' + +The JSON responses include a flag indicating success or failure. + +A successful result includes the updated expiration time. + + { + "renewed": "true", + "expires": "1584278311658" + } + +Error results include a message describing the reason for failure. + +Invalid token + + { + "renewed": "false", + "error": "Unknown token: 9caf743e-1e0d-4708-a9ac-a684a576067c" + } + +Unauthorized caller + + { + "renewed": "false", + "error": "Caller (guest) not authorized to renew tokens." + } + +##### Revocation + + curl -ivku admin:admin-password -X POST -d $TOKEN 'https://localhost:8443/gateway/sandbox/knoxtoken/api/v1/token/revoke' + +The JSON responses include a flag indicating success or failure. + + { + "revoked": "true" + } + +Error results include a message describing the reason for the failure. + +Invalid token + + { + "revoked": "false", + "error": "Unknown token: 9caf743e-1e0d-4708-a9ac-a684a576067c" + } + +Unauthorized caller + + { + "revoked": "false", + "error": "Caller (guest) not authorized to revoke tokens." + } + + +See documentation in Client Details for KnoxShell init, list and destroy for commands that leverage this token service for CLI sessions. + +#### Token Generation/Management UIs + +##### Overview + +In Apache Knox v2.0.0 the team added two new UIs that are directly accessible from the Knox Home page: + +* Token Generation +* Token Management + +By default, the `homepage` topology comes with the `KNOXTOKEN` service enabled with the following attributes: + +* token TTL is set to 120 days +* token service is enabled (default to keystore-based token state service) +* the admin user is allowed to renew/revoke tokens + +In this topology, homepage, two new applications were added in order to display the above-listed UIs: + +* `tokengen`: this is an old-style JSP UI, with a relatively simple JS code included. The source is located in the [gateway-applications](https://github.com/apache/knox/tree/v2.0.0/gateway-applications/src/main/resources/applications/tokengen) Maven sub-module. +* `token-management`: this is an Angular UI. The source is located in its own [knox-token-management-ui](https://github.com/apache/knox/tree/v2.0.0/knox-token-management-ui) Maven sub-module. + +On the Knox Home page, you will see a new town in the General Proxy Information table like this: + +  + +However, the _Integration Token_ links are disabled by default, because token integration requires a gateway-level alias - called `knox.token.hash.key` - being created and without that alias, it [does not make sense to show those links](https://github.com/apache/knox/pull/512). + +##### Creating the token hash key + +As explained, if you would like to use Knox's token generation features, you will have to create a gateway-level alias with a 256, 384, or 512-bit length JWK. You can do it in - at least - two different ways: + +1. You generate your own MAC (using [this online tool](https://8gwifi.org/jwkfunctions.jsp) for instance) and save it as an alias using Knox CLI. +2. You do it running the following Knox CLI command: + `generate-jwk --saveAlias knox.token.hash.key` + +The second option involves a newly created Knox CLI command called `generate-jwk`: + +##### Token state service implementations + +There was an important step the Knox team made to provide more flexibility for our end-users: there are some internal service implementations in Knox that were hard-coded in the Java source code. One of those services is the `Token State` service implementation which you can change in gateway-site.xml going forward by setting the `gateway.service.tokenstate.impl` property to any of: + +1. `org.apache.knox.gateway.services.token.impl.DefaultTokenStateService` - keeps all token information in memory, therefore all of this information is lost when Knox is shut down +2. `org.apache.knox.gateway.services.token.impl.AliasBasedTokenStateService` - token information is stored in the gateway credential store. This is a durable option, but not suitable for HA deployments +3. `org.apache.knox.gateway.services.token.impl.JournalBasedTokenStateService` - token information is stored in plain files within `$KNOX_DATA_DIR/security/token-state` folder. This option also provides a durable persistence layer for tokens and it might be good for HA scenarios too (in case of KNOX_DATA_DIR is on a shared drive), but the token data is written out in plain text (i.e. not encrypted) so it's less secure. +4. `org.apache.knox.gateway.services.token.impl.ZookeeperTokenStateService` - this is an extension of the keystore-based approach. In this case, token information is stored in Zookeeper using Knox aliases. The token's alias name equals to its generated token ID. +5. `org.apache.knox.gateway.services.token.impl.JDBCTokenStateService` - stores token information in relational databases. It's not only durable, but it's perfectly fine with HA deployments. Currently, PostgreSQL and MySQL databases are supported. + +By default, the `AliasBasedTokenStateService` implementation is used. + +##### Configuring the JDBC token state service + +If you want to use the newly implemented database token management, youâve to set `gateway.service.tokenstate.impl` in _gateway-site.xml_ to `org.apache.knox.gateway.services.token.impl.JDBCTokenStateService`. + +Now, that you have configured your token state backend, you need to configure a valid database in _gateway-site.xml_. There are two ways to do that: + +1. You either declare database connection properties one-by-one: + `gateway.database.type` - should be set to `postgresql` or `mysql` + `gateway.database.host` - the host where your DB server is running + `gateway.database.port` - the port that your DB server is listening on + `gateway.database.name` - the name of the database you are connecting to +2. Or you declare an all-in-one JDBC connection string called `gateway.database.connection.url`. The following value will show you how to connect to an SSL enabled PostgreSQL server: + <code>jdbc:postgresql://$myPostgresServerHost:5432/postgres?user=postgres&ssl=true&sslmode=verify-full&sslrootcert=/usr/local/var/postgresql@10/data/root.crt</code> + +If your database requires user/password authentication, the following aliases must be saved into the Knox Gatewayâs credential store (__gateway-credentials.jceks): + +* `gateway_database_user` - the username +* `gateway_database_password` - the password + +###### Database design + +  + +As you can see, there are only 2 tables: + + * `KNOXTOKENS` contains basic information about the generated token + * `KNOX_TOKEN_METADATA` contains an arbitrary number of metadata information for the generated token. At the time of this document being written the following metadata exist: + * `passcode` - this is the BASE-64 encoded value of the generated + passcode token MAC. That is, the BASE-64 decoded value is a generated + MAC. + * `userName` - the logged-in user who generated the token + * `enabled` - this is a boolean flag indicating that the given token is enabled or not (a _disabled_ token cannot be used for + authentication purposes) + * `comment` - this is optional metadata, saved only if the user enters something in the _Comment_ input field on the _Token + Generation_ page (see below) + +##### Generating a token + +Once you configured the `knox.token.hash.key` alias and optionally customized your token state service, you are all set to generate Knox tokens using the new Token Generation UI: + +  + +The following sections are displayed on the page: + +* status bar: here you can see an informative message on the configured Token State backend. There are 3 different statuses: + * ERROR: shown in red. This indicates a problem with the service backend which makes the feature not work. Usually, this is visible when end-users configure JDBC token state service, but they make a mistake in their DB settings + * WARN: displayed in yellow (see above picture). This indicates that the feature is enabled and working, but there are some limitations + * INFO: displayed in green. This indicates when the token management backend is properly configured for HA and production deployments +* there is an information label explaining the purpose of the token generation page +* comment: this is an _optional_ input field that allows end-users to add meaningful comments (mnemonics) to their generated tokens. The maximum length is 255 characters. +* the `Configured maximum lifetime` informs the clients about the `knox.token.ttl` property set in the `homepage` topology (defaults to 120 days). If that property is not set (e.g. someone removes it from he homepage topology), Knox uses a hard-coded value of 30 seconds (aka. default Knox token TTL) +* Custom token lifetime can be set by adjusting the days/hours/minutes spinners. The default configuration will yield one hour. +* Clicking the Generate Token button will try to create a token for you. + +##### About the generated token TTL + +Out of the box, Knox will display the custom lifetime spinners on the Token Generation page. However, they can be hidden by setting the `knox.token.lifespan.input.enabled` property to `false` in the `homepage` topology. Given that possibility and the configured maximum lifetime the generated token can have the following TTL value: + +* there is no configured token TTL and lifespan inputs are disabled -> the default TTL is used (30 seconds) +* there is configured TTL and lifespan inputs are disabled -> the configured TTL is used +* there is configured TTL and lifespan inputs are enabled and lifespan inputs result in a value that is less than or equal to the configured TTL -> the lifespan query param is used +* there is configured TTL and lifespan inputs are enabled and lifespan inputs result in a value that is greater than the configured TTL -> the configured TTL is used + +##### Successful token generation + +  + +On the resulting page there is two sensitive information that you can use in Knox to authenticate your request: + +1. **JWT token** - this is the serialized JWT and is fully compatible with the old-style Bearer authorization method. Clicking the `JWT Token` label on the page will copy the value into the clipboard. You might want to use it as the âTokenâ user: + + `$ curl -ku Token:eyJqa3UiOiJodHRwczpcL1wvbG9jYWxob3N0Ojg0NDNcL2dhdGV3YXlcL2hvbWVwYWdlXC9rbm94dG9rZW5cL2FwaVwvdjFcL2p3a3MuanNvbiIsImtpZCI6IkdsOTZfYTM2MTJCZWFsS2tURFRaOTZfVkVsLVhNRVRFRmZuNTRMQ1A2UDQiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJhZG1pbiIsImprdSI6Imh0dHBzOlwvXC9sb2NhbGhvc3Q6ODQ0M1wvZ2F0ZXdheVwvaG9tZXBhZ2VcL2tub3h0b2tlblwvYXBpXC92MVwvandrcy5qc29uIiwia2lkIjoiR2w5Nl9hMzYxMkJlYWxLa1REVFo5Nl9WRWwtWE1FVEVGZm41NExDUDZQNCIsImlzcyI6IktOT1hTU08iLCJleHAiOjE2MzY2MjU3MTAsIm1hbmFnZWQudG9rZW4iOiJ0cnVlIiwia25veC5pZCI6ImQxNjFjYWMxLWY5M2UtNDIyOS1hMGRkLTNhNzdhYjkxNDg3MSJ9.e_BNPf_G1iBrU0m3hul5VmmSbpw0w1pUAXl3czOcuxFOQ0Tki-Gq76fCBFUNdKt4QwLpNXxM321cH1TeMG4IhL-92QORSIZgRxY4OUtUgERzcU7-27VNYOzJbaRCjrx-Vb4bSriRJJDwbbXyAoEw_bjiP8EzFFJTPmGcctEzrOLWFk57cLO-2QLd2nbrNd4qmrRR6sEfP81Jg8UL-Ptp66vH_xalJJWuoyoNgGRmH8IMdLVwBgeLeVHiI7NmokuhO-vbctoEwV3Rt4pMpA0VSWGFN0MI4WtU0crjXXHg8U9xSZyOeyT3fMZBXctvBomhGlWaAvuT5AxQGyMMP3VLGw https:/localhost:8443/gateway/sandbox/webhdfs/v1?op=LISTSTATUS` +`{"FileStatuses":{"FileStatus":[{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16386,"group":"supergroup","length":0,"modificationTime":1621238405734,"owner":"hdfs","pathSuffix":"tmp","permission":"1777","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16387,"group":"supergroup","length":0,"modificationTime":1621238326078,"owner":"hdfs","pathSuffix":"user","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}]}}` + +2. **Passcode token** - this is the serialized passcode token, which you can use as the âPasscodeâ user (Clicking the `Passcode Token` label on the page will copy the value into the clipboard): + + `$ curl -ku Passcode:WkRFMk1XTmhZekV0WmprelpTMDBNakk1TFdFd1pHUXRNMkUzTjJGaU9URTBPRGN4OjpPVEV5Tm1KbFltUXROVEUyWkMwME9HSTBMVGd4TTJZdE1HRmxaalJrWlRVNFpXRTA= https://localhost:8443/gateway/sandbox/webhdfs/v1?op=LISTSTATUS` +`{"FileStatuses":{"FileStatus":[{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16386,"group":"supergroup","length":0,"modificationTime":1621238405734,"owner":"hdfs","pathSuffix":"tmp","permission":"1777","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16387,"group":"supergroup","length":0,"modificationTime":1621238326078,"owner":"hdfs","pathSuffix":"user","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}]}}` + +The reason, we needed to support the shorter `Passcode token`, is that there are 3rd party tools where the long JWT exceeds input fields limitations so we need to address this issue with shorter token values. + +The rest of the fields are complementary information such as the expiration date/time of the generated token or the user who created it. + +##### Token generation failed + +If there was an error during token generation, you will see a failure right under the input field boxes (above the Generate Token button): + +  + +The above error message indicates a failure that the admin user already generated more tokens than they are allowed to. This limitation is configurable in the `gateway-site.xml`: + +- `gateway.knox.token.limit.per.user` - indicates the maximum number of tokens a user can manage at the same time. `-1` means that users are allowed to create/manage as many tokens as they want. This configuration only applies when the server-managed token state is enabled either in `gateway-site` or at the `topology` level. Defaults to 10. + +##### Token Management + +In addition to the token generation UI, Knox comes with a Token Management UI where logged-in users can see all the active tokens that they generated before. That is, if a token got expired and was removed from the underlying token store, it won't be displayed here. + +  + +On this page, you will see basic information about your generated token(s) and you can execute the following actions: + +1. Enable/Disable - based on the current status, you can temporarily enable/disable a token. Please note that disabled tokens are not allowed to use for authentication purposes. +2. Revoke - you can remove the token from the persistent store. Please note this action cannot be undone, once you revoked a token Knox will delete it from the in-memory cache as well as the underlying persistent token storage + +In order to refresh the table, you can use the `Refresh icon` above the table (if you generated tokens on another tab for instance).
Added: knox/trunk/books/2.0.0/config_ldap_authc_cache.md URL: http://svn.apache.org/viewvc/knox/trunk/books/2.0.0/config_ldap_authc_cache.md?rev=1899392&view=auto ============================================================================== --- knox/trunk/books/2.0.0/config_ldap_authc_cache.md (added) +++ knox/trunk/books/2.0.0/config_ldap_authc_cache.md Wed Mar 30 15:22:57 2022 @@ -0,0 +1,211 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + https://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +---> + +### LDAP Authentication Caching ### + +Knox can be configured to cache LDAP authentication information. Knox leverages Shiro's built in +caching mechanisms and has been tested with Shiro's EhCache cache manager implementation. + +The following provider snippet demonstrates how to configure turning on the cache using the ShiroProvider. In addition to +using `org.apache.knox.gateway.shirorealm.KnoxLdapRealm` in the Shiro configuration, and setting up the cache you *must* set +the flag for enabling caching authentication to true. Please see the property, `main.ldapRealm.authenticationCachingEnabled` below. + + + <provider> + <role>authentication</role> + <name>ShiroProvider</name> + <enabled>true</enabled> + <param> + <name>main.ldapRealm</name> + <value>org.apache.knox.gateway.shirorealm.KnoxLdapRealm</value> + </param> + <param> + <name>main.ldapGroupContextFactory</name> + <value>org.apache.knox.gateway.shirorealm.KnoxLdapContextFactory</value> + </param> + <param> + <name>main.ldapRealm.contextFactory</name> + <value>$ldapGroupContextFactory</value> + </param> + <param> + <name>main.ldapRealm.contextFactory.url</name> + <value>ldap://localhost:33389</value> + </param> + <param> + <name>main.ldapRealm.userDnTemplate</name> + <value>uid={0},ou=people,dc=hadoop,dc=apache,dc=org</value> + </param> + <param> + <name>main.ldapRealm.authorizationEnabled</name> + <!-- defaults to: false --> + <value>true</value> + </param> + <param> + <name>main.ldapRealm.searchBase</name> + <value>ou=groups,dc=hadoop,dc=apache,dc=org</value> + </param> + <param> + <name>main.cacheManager</name> + <value>org.apache.knox.gateway.shirorealm.KnoxCacheManager</value> + </param> + <param> + <name>main.securityManager.cacheManager</name> + <value>$cacheManager</value> + </param> + <param> + <name>main.ldapRealm.authenticationCachingEnabled</name> + <value>true</value> + </param> + <param> + <name>main.ldapRealm.memberAttributeValueTemplate</name> + <value>uid={0},ou=people,dc=hadoop,dc=apache,dc=org</value> + </param> + <param> + <name>main.ldapRealm.contextFactory.systemUsername</name> + <value>uid=guest,ou=people,dc=hadoop,dc=apache,dc=org</value> + </param> + <param> + <name>main.ldapRealm.contextFactory.systemPassword</name> + <value>guest-password</value> + </param> + <param> + <name>urls./**</name> + <value>authcBasic</value> + </param> + </provider> + + +### Trying out caching ### + +Knox bundles a template topology files that can be used to try out the caching functionality. +The template file located under `{GATEWAY_HOME}/templates` is `sandbox.knoxrealm.ehcache.xml`. + +To try this out + + cd {GATEWAY_HOME} + cp templates/sandbox.knoxrealm.ehcache.xml conf/topologies/sandbox.xml + bin/ldap.sh start + bin/gateway.sh start + +The following call to WebHDFS should report: `{"Path":"/user/tom"}` + + curl -i -v -k -u tom:tom-password -X GET https://localhost:8443/gateway/sandbox/webhdfs/v1?op=GETHOMEDIRECTORY + +In order to see the cache working, LDAP can now be shutdown and the user will still authenticate successfully. + + bin/ldap.sh stop + +and then the following should still return successfully like it did earlier. + + curl -i -v -k -u tom:tom-password -X GET https://localhost:8443/gateway/sandbox/webhdfs/v1?op=GETHOMEDIRECTORY + + +#### Advanced Caching Config #### + +By default the EhCache support in Shiro contains a ehcache.xml in its classpath which is the following + + <ehcache name="knox-YOUR_TOPOLOGY_NAME"> + + <!-- Sets the path to the directory where cache .data files are created. + + If the path is a Java System Property it is replaced by + its value in the running VM. The following properties are translated: + + user.home - User's home directory + user.dir - User's current working directory + java.io.tmpdir - Default temp file path + --> + <diskStore path="java.io.tmpdir/shiro-ehcache"/> + + + <!--Default Cache configuration. These will applied to caches programmatically created through + the CacheManager. + + The following attributes are required: + + maxElementsInMemory - Sets the maximum number of objects that will be created in memory + eternal - Sets whether elements are eternal. If eternal, timeouts are ignored and the + element is never expired. + overflowToDisk - Sets whether elements can overflow to disk when the in-memory cache + has reached the maxInMemory limit. + + The following attributes are optional: + timeToIdleSeconds - Sets the time to idle for an element before it expires. + i.e. The maximum amount of time between accesses before an element expires + Is only used if the element is not eternal. + Optional attribute. A value of 0 means that an Element can idle for infinity. + The default value is 0. + timeToLiveSeconds - Sets the time to live for an element before it expires. + i.e. The maximum time between creation time and when an element expires. + Is only used if the element is not eternal. + Optional attribute. A value of 0 means that and Element can live for infinity. + The default value is 0. + diskPersistent - Whether the disk store persists between restarts of the Virtual Machine. + The default value is false. + diskExpiryThreadIntervalSeconds- The number of seconds between runs of the disk expiry thread. The default value + is 120 seconds. + memoryStoreEvictionPolicy - Policy would be enforced upon reaching the maxElementsInMemory limit. Default + policy is Least Recently Used (specified as LRU). Other policies available - + First In First Out (specified as FIFO) and Less Frequently Used + (specified as LFU) + --> + + <defaultCache + maxElementsInMemory="10000" + eternal="false" + timeToIdleSeconds="120" + timeToLiveSeconds="120" + overflowToDisk="false" + diskPersistent="false" + diskExpiryThreadIntervalSeconds="120" + /> + + <!-- We want eternal="true" and no timeToIdle or timeToLive settings because Shiro manages session + expirations explicitly. If we set it to false and then set corresponding timeToIdle and timeToLive properties, + ehcache would evict sessions without Shiro's knowledge, which would cause many problems + (e.g. "My Shiro session timeout is 30 minutes - why isn't a session available after 2 minutes?" + Answer - ehcache expired it due to the timeToIdle property set to 120 seconds.) + + diskPersistent=true since we want an enterprise session management feature - ability to use sessions after + even after a JVM restart. --> + <cache name="shiro-activeSessionCache" + maxElementsInMemory="10000" + overflowToDisk="true" + eternal="true" + timeToLiveSeconds="0" + timeToIdleSeconds="0" + diskPersistent="true" + diskExpiryThreadIntervalSeconds="600"/> + + <cache name="org.apache.shiro.realm.text.PropertiesRealm-0-accounts" + maxElementsInMemory="1000" + eternal="true" + overflowToDisk="true"/> + + </ehcache> + +A custom configuration file (ehcache.xml) can be used in place of this in order to set specific caching configuration. + +In order to set the ehcache.xml file to use for a particular topology, set the following parameter in the configuration +for the ShiroProvider: + + <param> + <name>main.cacheManager.cacheManagerConfigFile</name> + <value>classpath:ehcache.xml</value> + </param> + +In the above example, place the ehcache.xml file under `{GATEWAY_HOME}/conf` and restart the gateway server. Added: knox/trunk/books/2.0.0/config_ldap_group_lookup.md URL: http://svn.apache.org/viewvc/knox/trunk/books/2.0.0/config_ldap_group_lookup.md?rev=1899392&view=auto ============================================================================== --- knox/trunk/books/2.0.0/config_ldap_group_lookup.md (added) +++ knox/trunk/books/2.0.0/config_ldap_group_lookup.md Wed Mar 30 15:22:57 2022 @@ -0,0 +1,228 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + https://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +---> + +### LDAP Group Lookup ### + +Knox can be configured to look up LDAP groups that the authenticated user belong to. +Knox can look up both Static LDAP Groups and Dynamic LDAP Groups. +The looked up groups are populated as Principal(s) in the Java Subject of the authenticated user. +Therefore service authorization rules can be defined in terms of LDAP groups looked up from a LDAP directory. + +To look up LDAP groups of authenticated user from LDAP, you have to use `org.apache.knox.gateway.shirorealm.KnoxLdapRealm` in Shiro configuration. + +Please see below a sample Shiro configuration snippet from a topology file that was tested looking LDAP groups. + + <provider> + <role>authentication</role> + <name>ShiroProvider</name> + <enabled>true</enabled> + <!-- + session timeout in minutes, this is really idle timeout, + defaults to 30mins, if the property value is not defined,, + current client authentication would expire if client idles continuously for more than this value + --> + <!-- defaults to: 30 minutes + <param> + <name>sessionTimeout</name> + <value>30</value> + </param> + --> + + <!-- + Use single KnoxLdapRealm to do authentication and ldap group look up + --> + <param> + <name>main.ldapRealm</name> + <value>org.apache.knox.gateway.shirorealm.KnoxLdapRealm</value> + </param> + <param> + <name>main.ldapGroupContextFactory</name> + <value>org.apache.knox.gateway.shirorealm.KnoxLdapContextFactory</value> + </param> + <param> + <name>main.ldapRealm.contextFactory</name> + <value>$ldapGroupContextFactory</value> + </param> + <!-- defaults to: simple + <param> + <name>main.ldapRealm.contextFactory.authenticationMechanism</name> + <value>simple</value> + </param> + --> + <param> + <name>main.ldapRealm.contextFactory.url</name> + <value>ldap://localhost:33389</value> + </param> + <param> + <name>main.ldapRealm.userDnTemplate</name> + <value>uid={0},ou=people,dc=hadoop,dc=apache,dc=org</value> + </param> + + <param> + <name>main.ldapRealm.authorizationEnabled</name> + <!-- defaults to: false --> + <value>true</value> + </param> + <!-- defaults to: simple + <param> + <name>main.ldapRealm.contextFactory.systemAuthenticationMechanism</name> + <value>simple</value> + </param> + --> + <param> + <name>main.ldapRealm.searchBase</name> + <value>ou=groups,dc=hadoop,dc=apache,dc=org</value> + </param> + <!-- defaults to: groupOfNames + <param> + <name>main.ldapRealm.groupObjectClass</name> + <value>groupOfNames</value> + </param> + --> + <!-- defaults to: member + <param> + <name>main.ldapRealm.memberAttribute</name> + <value>member</value> + </param> + --> + <param> + <name>main.cacheManager</name> + <value>org.apache.shiro.cache.MemoryConstrainedCacheManager</value> + </param> + <param> + <name>main.securityManager.cacheManager</name> + <value>$cacheManager</value> + </param> + <param> + <name>main.ldapRealm.memberAttributeValueTemplate</name> + <value>uid={0},ou=people,dc=hadoop,dc=apache,dc=org</value> + </param> + <!-- the above element is the template for most ldap servers + for active directory use the following instead and + remove the above configuration. + <param> + <name>main.ldapRealm.memberAttributeValueTemplate</name> + <value>cn={0},ou=people,dc=hadoop,dc=apache,dc=org</value> + </param> + --> + <param> + <name>main.ldapRealm.contextFactory.systemUsername</name> + <value>uid=guest,ou=people,dc=hadoop,dc=apache,dc=org</value> + </param> + <param> + <name>main.ldapRealm.contextFactory.systemPassword</name> + <value>${ALIAS=ldcSystemPassword}</value> + </param> + + <param> + <name>urls./**</name> + <value>authcBasic</value> + </param> + + </provider> + +The configuration shown above would look up Static LDAP groups of the authenticated user and populate the group principals in the Java Subject corresponding to the authenticated user. + +If you want to look up Dynamic LDAP Groups instead of Static LDAP Groups, you would have to specify groupObjectClass and memberAttribute params as shown below: + + <param> + <name>main.ldapRealm.groupObjectClass</name> + <value>groupOfUrls</value> + </param> + <param> + <name>main.ldapRealm.memberAttribute</name> + <value>memberUrl</value> + </param> + +### Template topology files and LDIF files to try out LDAP Group Look up ### + +Knox bundles some template topology files and ldif files that you can use to try and test LDAP Group Lookup and associated authorization ACLs. +All these template files are located under `{GATEWAY_HOME}/templates`. + + +#### LDAP Static Group Lookup Templates, authentication and group lookup from the same directory #### + +* topology file: sandbox.knoxrealm1.xml +* ldif file: users.ldapgroups.ldif + +To try this out + + cd {GATEWAY_HOME} + cp templates/sandbox.knoxrealm1.xml conf/topologies/sandbox.xml + cp templates/users.ldapgroups.ldif conf/users.ldif + java -jar bin/ldap.jar conf + java -Dsandbox.ldcSystemPassword=guest-password -jar bin/gateway.jar -persist-master + +Following call to WebHDFS should report HTTP/1.1 401 Unauthorized +As guest is not a member of group "analyst", authorization provider states user should be member of group "analyst" + + curl -i -v -k -u guest:guest-password -X GET https://localhost:8443/gateway/sandbox/webhdfs/v1?op=GETHOMEDIRECTORY + +Following call to WebHDFS should report: {"Path":"/user/sam"} +As sam is a member of group "analyst", authorization provider states user should be member of group "analyst" + + curl -i -v -k -u sam:sam-password -X GET https://localhost:8443/gateway/sandbox/webhdfs/v1?op=GETHOMEDIRECTORY + + +#### LDAP Static Group Lookup Templates, authentication and group lookup from different directories #### + +* topology file: sandbox.knoxrealm2.xml +* ldif file: users.ldapgroups.ldif + +To try this out + + cd {GATEWAY_HOME} + cp templates/sandbox.knoxrealm2.xml conf/topologies/sandbox.xml + cp templates/users.ldapgroups.ldif conf/users.ldif + java -jar bin/ldap.jar conf + java -Dsandbox.ldcSystemPassword=guest-password -jar bin/gateway.jar -persist-master + +Following call to WebHDFS should report HTTP/1.1 401 Unauthorized +As guest is not a member of group "analyst", authorization provider states user should be member of group "analyst" + + curl -i -v -k -u guest:guest-password -X GET https://localhost:8443/gateway/sandbox/webhdfs/v1?op=GETHOMEDIRECTORY + +Following call to WebHDFS should report: {"Path":"/user/sam"} +As sam is a member of group "analyst", authorization provider states user should be member of group "analyst" + + curl -i -v -k -u sam:sam-password -X GET https://localhost:8443/gateway/sandbox/webhdfs/v1?op=GETHOMEDIRECTORY + +#### LDAP Dynamic Group Lookup Templates, authentication and dynamic group lookup from same directory #### + +* topology file: sandbox.knoxrealmdg.xml +* ldif file: users.ldapdynamicgroups.ldif + +To try this out + + cd {GATEWAY_HOME} + cp templates/sandbox.knoxrealmdg.xml conf/topologies/sandbox.xml + cp templates/users.ldapdynamicgroups.ldif conf/users.ldif + java -jar bin/ldap.jar conf + java -Dsandbox.ldcSystemPassword=guest-password -jar bin/gateway.jar -persist-master + +Please note that user.ldapdynamicgroups.ldif also loads necessary schema to create dynamic groups in Apache DS. + +Following call to WebHDFS should report HTTP/1.1 401 Unauthorized +As guest is not a member of dynamic group "directors", authorization provider states user should be member of group "directors" + + curl -i -v -k -u guest:guest-password -X GET https://localhost:8443/gateway/sandbox/webhdfs/v1?op=GETHOMEDIRECTORY + +Following call to WebHDFS should report: {"Path":"/user/bob"} +As bob is a member of dynamic group "directors", authorization provider states user should be member of group "directors" + + curl -i -v -k -u sam:sam-password -X GET https://localhost:8443/gateway/sandbox/webhdfs/v1?op=GETHOMEDIRECTORY + Added: knox/trunk/books/2.0.0/config_metrics.md URL: http://svn.apache.org/viewvc/knox/trunk/books/2.0.0/config_metrics.md?rev=1899392&view=auto ============================================================================== --- knox/trunk/books/2.0.0/config_metrics.md (added) +++ knox/trunk/books/2.0.0/config_metrics.md Wed Mar 30 15:22:57 2022 @@ -0,0 +1,49 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + https://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +---> + +### Metrics ### + +See the KIP for details on the implementation of metrics available in the gateway. + +[Metrics KIP](https://cwiki.apache.org/confluence/display/KNOX/KIP-2+Metrics) + +#### Metrics Configuration #### + +Metrics configuration can be done in `gateway-site.xml`. + +The initial configuration is mainly for turning on or off the metrics collection and then enabling reporters with their required config. + +The two initial reporters implemented are JMX and Graphite. + + gateway.metrics.enabled + +Turns on or off the metrics, default is 'true' + + gateway.jmx.metrics.reporting.enabled + +Turns on or off the jmx reporter, default is 'true' + + gateway.graphite.metrics.reporting.enabled + +Turns on or off the graphite reporter, default is 'false' + + gateway.graphite.metrics.reporting.host + gateway.graphite.metrics.reporting.port + gateway.graphite.metrics.reporting.frequency + +The above are the host, port and frequency of reporting (in seconds) parameters for the graphite reporter. + Added: knox/trunk/books/2.0.0/config_mutual_authentication_ssl.md URL: http://svn.apache.org/viewvc/knox/trunk/books/2.0.0/config_mutual_authentication_ssl.md?rev=1899392&view=auto ============================================================================== --- knox/trunk/books/2.0.0/config_mutual_authentication_ssl.md (added) +++ knox/trunk/books/2.0.0/config_mutual_authentication_ssl.md Wed Mar 30 15:22:57 2022 @@ -0,0 +1,43 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + https://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +---> + +### Mutual Authentication with SSL ### + +To establish a stronger trust relationship between client and server, we provide mutual authentication with SSL via client certs. This is particularly useful in providing additional validation for Preauthenticated SSO with HTTP Headers. Rather than just IP address validation, connections will only be accepted by Knox from clients presenting trusted certificates. + +This behavior is configured for the entire gateway instance within the gateway-site.xml file. All topologies deployed within the configured gateway instance will require incoming connections to present trusted client certificates during the SSL handshake. Otherwise, connections will be refused. + +The following table describes the configuration elements related to mutual authentication and their defaults: + +| Configuration Element | Description | +| -----------------------------------------------|-----------------------------------------------------------| +| gateway.client.auth.needed | True\|False - indicating the need for client authentication. Default is False.| +| gateway.truststore.path | Fully qualified path to the trust store to use. Default is the keystore used to hold the Gateway's identity. See `gateway.tls.keystore.path`.| +| gateway.truststore.type | Keystore type of the trust store. Default is JKS. | +| gateway.truststore.password.alias | Alias for the password to the trust store.| +| gateway.trust.all.certs | Allows for all certificates to be trusted. Default is false.| + +By only indicating that it is needed with `gateway.client.auth.needed`, the keystore identified by `gateway.tls.keystore.path` is used. By default this is `{GATEWAY_HOME}/data/security/keystores/gateway.jks`. +This is the identity keystore for the server, which can also be used as the truststore. +To use a dedicated truststore, `gateway.truststore.path` may be set to the absolute path of the truststore file. +The type of truststore file should be set using `gateway.truststore.type`; else, JKS will be assumed. +If the truststore password is different from the Gateway's master secret then it can be set using + + knoxcli.sh create-alias {password-alias} --value {pwd} + +The password alias name (`{password-alias}`) is set using `gateway.truststore.password.alias`; else, the alias name of "gateway-truststore-password" should be used. +If a password is not found using the provided (or default) alias name, then the Gateway's master secret will be used. Added: knox/trunk/books/2.0.0/config_pac4j_provider.md URL: http://svn.apache.org/viewvc/knox/trunk/books/2.0.0/config_pac4j_provider.md?rev=1899392&view=auto ============================================================================== --- knox/trunk/books/2.0.0/config_pac4j_provider.md (added) +++ knox/trunk/books/2.0.0/config_pac4j_provider.md Wed Mar 30 15:22:57 2022 @@ -0,0 +1,188 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + https://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +---> + +### Pac4j Provider - CAS / OAuth / SAML / OpenID Connect ### + +<p align="center"> + <img src="https://www.pac4j.org/img/logo-knox.png"; width="300" /> +</p> + +[pac4j](https://github.com/pac4j/pac4j) is a Java security engine to authenticate users, get their profiles and manage their authorizations in order to secure Java web applications. + +It supports many authentication mechanisms for UI and web services and is implemented by many frameworks and tools. + +For Knox, it is used as a federation provider to support the OAuth, CAS, SAML and OpenID Connect protocols. It must be used for SSO, in association with the KnoxSSO service and optionally with the SSOCookieProvider for access to REST APIs. + + +#### Configuration #### +##### SSO topology ##### + +To enable SSO for REST API access through the Knox gateway, you need to protect your Hadoop services with the SSOCookieProvider configured to use the KnoxSSO service (sandbox.xml topology): + + <gateway> + <provider> + <role>federation</role> + <name>SSOCookieProvider</name> + <enabled>true</enabled> + <param> + <name>sso.authentication.provider.url</name> + <value>https://127.0.0.1:8443/gateway/knoxsso/api/v1/websso</value> + </param> + </provider> + <provider> + <role>identity-assertion</role> + <name>Default</name> + <enabled>true</enabled> + </provider> + </gateway> + + <service> + <role>NAMENODE</role> + <url>hdfs://localhost:8020</url> + </service> + + ... + +and protect the KnoxSSO service by the pac4j provider (knoxsso.xml topology): + + <gateway> + <provider> + <role>federation</role> + <name>pac4j</name> + <enabled>true</enabled> + <param> + <name>pac4j.callbackUrl</name> + <value>https://127.0.0.1:8443/gateway/knoxsso/api/v1/websso</value> + </param> + <param> + <name>cas.loginUrl</name> + <value>https://casserverpac4j.herokuapp.com/login</value> + </param> + </provider> + <provider> + <role>identity-assertion</role> + <name>Default</name> + <enabled>true</enabled> + </provider> + </gateway> + + <service> + <role>KNOXSSO</role> + <param> + <name>knoxsso.cookie.secure.only</name> + <value>true</value> + </param> + <param> + <name>knoxsso.token.ttl</name> + <value>100000</value> + </param> + <param> + <name>knoxsso.redirect.whitelist.regex</name> + <value>^https?:\/\/(localhost|127\.0\.0\.1|0:0:0:0:0:0:0:1|::1):[0-9].*$</value> + </param> + </service> + +Notice that the pac4j callback URL is the KnoxSSO URL (`pac4j.callbackUrl` parameter). An additional `pac4j.cookie.domain.suffix` parameter allows you to define the domain suffix for the pac4j cookies. + +In this example, the pac4j provider is configured to authenticate users via a CAS server hosted at: https://casserverpac4j.herokuapp.com/login. + +##### Parameters ##### + +You can define the identity provider client/s to be used for authentication with the appropriate parameters - as defined below. +When configuring any pac4j identity provider client there is a mandatory parameter that must be defined to indicate the order in which the providers should be engaged with the first in the comma separated list being the default. Consuming applications may indicate their desire to use one of the configured clients with a query parameter called client_name. When there is no client_name specified, the default (first) provider is selected. + + <param> + <name>clientName</name> + <value>CLIENTNAME[,CLIENTNAME]</value> + </param> + +Valid client names are: `FacebookClient`, `TwitterClient`, `CasClient`, `SAML2Client` or `OidcClient` + +For tests only, you can use a basic authentication where login equals password by defining the following configuration: + + <param> + <name>clientName</name> + <value>testBasicAuth</value> + </param> + +NOTE: This is NOT a secure mechanism and must NOT be used in production deployments. + +Otherwise, you can use Facebook, Twitter, a CAS server, a SAML IdP or an OpenID Connect provider by using the following parameters: + +##### For OAuth support: + +Name | Value +-----|------ +facebook.id | Identifier of the OAuth Facebook application +facebook.secret | Secret of the OAuth Facebook application +facebook.scope | Requested scope at Facebook login +facebook.fields | Fields returned by Facebook +twitter.id | Identifier of the OAuth Twitter application +twitter.secret | Secret of the OAuth Twitter application + +##### For CAS support: + +Name | Value +-----|------ +cas.loginUrl | Login URL of the CAS server +cas.protocol | CAS protocol (`CAS10`, `CAS20`, `CAS20_PROXY`, `CAS30`, `CAS30_PROXY`, `SAML`) + +##### For SAML support: + +Name | Value +-----|------ +saml.keystorePassword | Password of the keystore (storepass) +saml.privateKeyPassword | Password for the private key (keypass) +saml.keystorePath | Path of the keystore +saml.identityProviderMetadataPath | Path of the identity provider metadata +saml.maximumAuthenticationLifetime | Maximum lifetime for authentication +saml.serviceProviderEntityId | Identifier of the service provider +saml.serviceProviderMetadataPath | Path of the service provider metadata + +> Get more details on the [pac4j wiki](https://github.com/pac4j/pac4j/wiki/Clients#saml-support). + +The SSO URL in your SAML 2 provider config will need to include a special query parameter that lets the pac4j provider know that the request is coming back from the provider rather than from a redirect from a KnoxSSO participating application. This query parameter is "pac4jCallback=true". + +This results in a URL that looks something like: + + https://hostname:8443/gateway/knoxsso/api/v1/websso?pac4jCallback=true&client_name=SAML2Client + +This also means that the SP Entity ID should also include this query parameter as appropriate for your provider. +Often something like the above URL is used for both the SSO URL and SP Entity ID. + +##### For OpenID Connect support: + +Name | Value +-----|------ +oidc.id | Identifier of the OpenID Connect provider +oidc.secret | Secret of the OpenID Connect provider +oidc.discoveryUri | Direcovery URI of the OpenID Connect provider +oidc.useNonce | Whether to use nonce during login process +oidc.preferredJwsAlgorithm | Preferred JWS algorithm +oidc.maxClockSkew | Max clock skew during login process +oidc.customParamKey1 | Key of the first custom parameter +oidc.customParamValue1 | Value of the first custom parameter +oidc.customParamKey2 | Key of the second custom parameter +oidc.customParamValue2 | Value of the second custom parameter + +> Get more details on the [pac4j wiki](https://github.com/pac4j/pac4j/wiki/Clients#openid-connect-support). + +In fact, you can even define several identity providers at the same time, the first being chosen by default unless you define a `client_name` parameter to specify it (`FacebookClient`, `TwitterClient`, `CasClient`, `SAML2Client` or `OidcClient`). + +##### UI invocation + +In a browser, when calling your Hadoop service (for example: `https://127.0.0.1:8443/gateway/sandbox/webhdfs/v1/tmp?op=LISTSTATUS`), you are redirected to the identity provider for login. Then, after a successful authentication, your are redirected back to your originally requested URL and your KnoxSSO session is initialized. Added: knox/trunk/books/2.0.0/config_pam_authn.md URL: http://svn.apache.org/viewvc/knox/trunk/books/2.0.0/config_pam_authn.md?rev=1899392&view=auto ============================================================================== --- knox/trunk/books/2.0.0/config_pam_authn.md (added) +++ knox/trunk/books/2.0.0/config_pam_authn.md Wed Mar 30 15:22:57 2022 @@ -0,0 +1,98 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + https://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +---> + +### PAM based Authentication ### + +There is a large number of pluggable authentication modules available on many Linux installations and from vendors of authentication solutions that are great to leverage for authenticating access to Hadoop through the Knox Gateway. In addition to LDAP support described in this guide, the ShiroProvider also includes support for PAM based authentication for unix based systems. + +This opens up the integration possibilities to many other readily available authentication mechanisms as well as other implementations for LDAP based authentication. More flexibility may be available through various PAM modules for group lookup, more complicated LDAP schemas or other areas where the KnoxLdapRealm is not sufficient. + +#### Configuration #### +##### Overview ##### +The primary motivation for leveraging PAM based authentication is to provide the ability to use the configuration provided by existing PAM modules that are available in a system's `/etc/pam.d/` directory. Therefore, the solution provided here is as simple as possible in order to allow the PAM module config itself to be the source of truth. What we do need to configure is the fact that we are using PAM through the `main.pamRealm` parameter and the KnoxPamRealm classname and the particular PAM module to use with the `main.pamRealm.service` parameter in the below example we have 'login'. + + <provider> + <role>authentication</role> + <name>ShiroProvider</name> + <enabled>true</enabled> + <param> + <name>sessionTimeout</name> + <value>30</value> + </param> + <param> + <name>main.pamRealm</name> + <value>org.apache.knox.gateway.shirorealm.KnoxPamRealm</value> + </param> + <param> + <name>main.pamRealm.service</name> + <value>login</value> + </param> + <param> + <name>urls./**</name> + <value>authcBasic</value> + </param> + </provider> + + +As a non-normative example of a PAM config file see the below from my MacBook `/etc/pam.d/login`: + + # login: auth account password session + auth optional pam_krb5.so use_kcminit + auth optional pam_ntlm.so try_first_pass + auth optional pam_mount.so try_first_pass + auth required pam_opendirectory.so try_first_pass + account required pam_nologin.so + account required pam_opendirectory.so + password required pam_opendirectory.so + session required pam_launchd.so + session required pam_uwtmp.so + session optional pam_mount.so + +The first four fields are: service-name, module-type, control-flag and module-filename. The fifth and greater fields are for optional arguments that are specific to the individual authentication modules. + +The second field in the configuration file is the module-type, it indicates which of the four PAM management services the corresponding module will provide to the application. Our sample configuration file refers to all four groups: + +* auth: identifies the PAMs that are invoked when the application calls pam_authenticate() and pam_setcred(). +* account: maps to the pam_acct_mgmt() function. +* session: indicates the mapping for the pam_open_session() and pam_close_session() calls. +* password: group refers to the pam_chauthtok() function. + +Generally, you only need to supply mappings for the functions that are needed by a specific application. For example, the standard password changing application, passwd, only requires a password group entry; any other entries are ignored. + +The third field indicates what action is to be taken based on the success or failure of the corresponding module. Choices for tokens to fill this field are: + +* requisite: Failure instantly returns control to the application indicating the nature of the first module failure. +* required: All these modules are required to succeed for libpam to return success to the application. +* sufficient: Given that all preceding modules have succeeded, the success of this module leads to an immediate and successful return to the application (failure of this module is ignored). +* optional: The success or failure of this module is generally not recorded. + +The fourth field contains the name of the loadable module, pam_*.so. For the sake of readability, the full pathname of each module is not given. Before Linux-PAM-0.56 was released, there was no support for a default authentication-module directory. If you have an earlier version of Linux-PAM installed, you will have to specify the full path for each of the modules. Your distribution most likely placed these modules exclusively in one of the following directories: /lib/security/ or /usr/lib/security/. + +Also, find below a non-normative example of a PAM config file(/etc/pam.d/login) for Ubuntu: + + #%PAM-1.0 + + auth required pam_sepermit.so + # pam_selinux.so close should be the first session rule + session required pam_selinux.so close + session required pam_loginuid.so + # pam_selinux.so open should only be followed by sessions to be executed in the user context + session required pam_selinux.so open env_params + session optional pam_keyinit.so force revoke + + session required pam_env.so user_readenv=1 envfile=/etc/default/locale + @include password-auth Added: knox/trunk/books/2.0.0/config_preauth_sso_provider.md URL: http://svn.apache.org/viewvc/knox/trunk/books/2.0.0/config_preauth_sso_provider.md?rev=1899392&view=auto ============================================================================== --- knox/trunk/books/2.0.0/config_preauth_sso_provider.md (added) +++ knox/trunk/books/2.0.0/config_preauth_sso_provider.md Wed Mar 30 15:22:57 2022 @@ -0,0 +1,87 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + https://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +---> + +### Preauthenticated SSO Provider ### + +A number of SSO solutions provide mechanisms for federating an authenticated identity across applications. These mechanisms are at times simple HTTP Header type tokens that can be used to propagate the identity across process boundaries. + +Knox Gateway needs a pluggable mechanism for consuming these tokens and federating the asserted identity through an interaction with the Hadoop cluster. + +**CAUTION: The use of this provider requires that proper network security and identity provider configuration and deployment does not allow requests directly to the Knox gateway. Otherwise, this provider will leave the gateway exposed to identity spoofing.** + +#### Configuration #### +##### Overview ##### +This provider was designed for use with identity solutions such as those provided by CA's SiteMinder and IBM's Tivoli Access Manager. While direct testing with these products has not been done, there has been extensive unit and functional testing that ensure that it should work with such providers. + +The HeaderPreAuth provider is configured within the topology file and has a minimal configuration that assumes SM_USER for CA SiteMinder. The following example is the bare minimum configuration for SiteMinder (with no IP address validation). + + <provider> + <role>federation</role> + <name>HeaderPreAuth</name> + <enabled>true</enabled> + </provider> + +The following table describes the configuration options for the web app security provider: + +##### Descriptions ##### + +Name | Description | Default +---------|-----------|-------- +preauth.validation.method | Optional parameter that indicates the types of trust validation to perform on incoming requests. There could be one or more comma-separated validators defined in this property. If there are multiple validators, Apache Knox validates each validator in the same sequence as it is configured. This works similar to short-circuit AND operation i.e. if any validator fails, Knox does not perform further validation and returns overall failure immediately. Possible values are: null, preauth.default.validation, preauth.ip.validation, custom validator (details described in [Custom Validator](dev-guide.html#Validator)). Failure results in a 403 forbidden HTTP status response.| null - which means 'preauth.default.validation' that is no validation will be performed and that we are assuming that the network security and external authentication system is sufficient. +preauth.ip.addresses | Optional parameter that indicates the list of trusted ip addresses. When preauth.ip.validation is indicated as the validation method this parameter must be provided to indicate the trusted ip address set. Wildcarded IPs may be used to indicate subnet level trust. ie. 127.0.* | null - which means that no validation will be performed. +preauth.custom.header | Required parameter for indicating a custom header to use for extracting the preauthenticated principal. The value extracted from this header is utilized as the PrimaryPrincipal within the established Subject. An incoming request that is missing the configured header will be refused with a 401 unauthorized HTTP status. | SM_USER for SiteMinder usecase +preauth.custom.group.header | Optional parameter for indicating a HTTP header name that contains a comma separated list of groups. These are added to the authenticated Subject as group principals. A missing group header will result in no groups being extracted from the incoming request and a log entry but processing will continue. | null - which means that there will be no group principals extracted from the request and added to the established Subject. + +NOTE: Mutual authentication can be used to establish a strong trust relationship between clients and servers while using the Preauthenticated SSO provider. See the configuration for Mutual Authentication with SSL in this document. + +##### Configuration for SiteMinder +The following is an example of a configuration of the preauthenticated SSO provider that leverages the default SM_USER header name - assuming use with CA SiteMinder. It further configures the validation based on the IP address from the incoming request. + + <provider> + <role>federation</role> + <name>HeaderPreAuth</name> + <enabled>true</enabled> + <param><name>preauth.validation.method</name><value>preauth.ip.validation</value></param> + <param><name>preauth.ip.addresses</name><value>127.0.0.2,127.0.0.1</value></param> + </provider> + +##### REST Invocation for SiteMinder +The following curl command can be used to request a directory listing from HDFS while passing in the expected header SM_USER. + + curl -k -i --header "SM_USER: guest" -v https://localhost:8443/gateway/sandbox/webhdfs/v1/tmp?op=LISTSTATUS + +Omitting the `--header "SM_USER: guest"` above will result in a rejected request. + +##### Configuration for IBM Tivoli AM +As an example for configuring the preauthenticated SSO provider for another SSO provider, the following illustrates the values used for IBM's Tivoli Access Manager: + + <provider> + <role>federation</role> + <name>HeaderPreAuth</name> + <enabled>true</enabled> + <param><name>preauth.custom.header</name><value>iv_user</value></param> + <param><name>preauth.custom.group.header</name><value>iv_group</value></param> + <param><name>preauth.validation.method</name><value>preauth.ip.validation</value></param> + <param><name>preauth.ip.addresses</name><value>127.0.0.2,127.0.0.1</value></param> + </provider> + +##### REST Invocation for Tivoli AM +The following curl command can be used to request a directory listing from HDFS while passing in the expected headers of iv_user and iv_group. Note that the iv_group value in this command matches the expected ACL for webhdfs in the above topology file. Changing this from "admin" to "admin2" should result in a 401 unauthorized response. + + curl -k -i --header "iv_user: guest" --header "iv_group: admin" -v https://localhost:8443/gateway/sandbox/webhdfs/v1/tmp?op=LISTSTATUS + +Omitting the `--header "iv_user: guest"` above will result in a rejected request. Added: knox/trunk/books/2.0.0/config_sandbox.md URL: http://svn.apache.org/viewvc/knox/trunk/books/2.0.0/config_sandbox.md?rev=1899392&view=auto ============================================================================== --- knox/trunk/books/2.0.0/config_sandbox.md (added) +++ knox/trunk/books/2.0.0/config_sandbox.md Wed Mar 30 15:22:57 2022 @@ -0,0 +1,51 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + https://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +---> + +## Sandbox Configuration ## + +### Sandbox 2.x Configuration ### + +TODO + +### Sandbox 1.x Configuration ### + +TODO - Update this section to use hostmap if that simplifies things. + +This version of the Apache Knox Gateway is tested against [Hortonworks Sandbox 1.x][sandbox] + +Currently there is an issue with Sandbox that prevents it from being easily used with the gateway. +In order to correct the issue, you can use the commands below to login to the Sandbox VM and modify the configuration. +This assumes that the name sandbox is setup to resolve to the Sandbox VM. +It may be necessary to use the IP address of the Sandbox VM instead. +*This is frequently but not always `192.168.56.101`.* + + ssh root@sandbox + cp /usr/lib/hadoop/conf/hdfs-site.xml /usr/lib/hadoop/conf/hdfs-site.xml.orig + sed -e s/localhost/sandbox/ /usr/lib/hadoop/conf/hdfs-site.xml.orig > /usr/lib/hadoop/conf/hdfs-site.xml + shutdown -r now + +In addition to make it very easy to follow along with the samples for the gateway you can configure your local system to resolve the address of the Sandbox by the names `vm` and `sandbox`. +The IP address that is shown below should be that of the Sandbox VM as it is known on your system. +*This will likely, but not always, be `192.168.56.101`.* + +On Linux or Macintosh systems add a line like this to the end of the file `/etc/hosts` on your local machine, *not the Sandbox VM*. +_Note: The character between the 192.168.56.101 and vm below is a *tab* character._ + + 192.168.56.101 vm sandbox + +On Windows systems a similar but different mechanism can be used. On recent +versions of windows the file that should be modified is `%systemroot%\system32\drivers\etc\hosts` Added: knox/trunk/books/2.0.0/config_sso_cookie_provider.md URL: http://svn.apache.org/viewvc/knox/trunk/books/2.0.0/config_sso_cookie_provider.md?rev=1899392&view=auto ============================================================================== --- knox/trunk/books/2.0.0/config_sso_cookie_provider.md (added) +++ knox/trunk/books/2.0.0/config_sso_cookie_provider.md Wed Mar 30 15:22:57 2022 @@ -0,0 +1,118 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + https://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +---> + +### SSO Cookie Provider ### + +#### Overview #### +The SSOCookieProvider enables the federation of the authentication event that occurred through KnoxSSO. KnoxSSO is a typical SP initiated websso mechanism that sets a cookie to be presented by browsers to participating applications and cryptographically verified. + +Knox Gateway needs a pluggable mechanism for consuming these cookies and federating the KnoxSSO authentication event as an asserted identity in its interaction with the Hadoop cluster for REST API invocations. This provider is useful when an application that is integrated with KnoxSSO for authentication also consumes REST APIs through the Knox Gateway. + +Based on our understanding of the WebSSO flow it should behave like: + +* SSOCookieProvider checks for hadoop-jwt cookie and in its absence redirects to the configured SSO provider URL (knoxsso endpoint) +* The configured Provider on the KnoxSSO endpoint challenges the user in a provider specific way (presents form, redirects to SAML IdP, etc.) +* The authentication provider on KnoxSSO validates the identity of the user through credentials/tokens +* The WebSSO service exchanges the normalized Java Subject into a JWT token and sets it on the response as a cookie named `hadoop-jwt` +* The WebSSO service then redirects the user agent back to the originally requested URL - the requested Knox service subsequent invocations will find the cookie in the incoming request and not need to engage the WebSSO service again until it expires. + +#### Configuration #### +##### sandbox.json Topology Example +Configuring one of the cluster topologies to use the SSOCookieProvider instead of the out of the box ShiroProvider would look something like the following: + +sso-provider.json + + { + "providers": [ + { + "role": "federation", + "name": "SSOCookieProvider", + "enabled": "true", + "params": { + "sso.authentication.provider.url": "https://localhost:9443/gateway/idp/api/v1/websso"; + } + } + ] + } + +sandbox.json + + { + "provider-config-ref": "sso-provider", + "services": [ + { + "name": "WEBHDFS", + "urls": [ + "http://localhost:50070/webhdfs"; + ] + }, + { + "name": "WEBHCAT", + "urls": [ + "http://localhost:50111/templeton"; + ] + } + ] + } + +The following table describes the configuration options for the sso cookie provider: + +##### Descriptions ##### + +Name | Description | Default +---------|-----------|--------- +sso.authentication.provider.url | Required parameter that indicates the location of the KnoxSSO endpoint and where to redirect the useragent when no SSO cookie is found in the incoming request. | N/A +sso.token.verification.pem | Optional parameter that specifies public key used to validate hadoop-jwt token. The key must be in PEM encoded format excluding the header and footer lines.| N/A +sso.expected.audiences | Optional parameter used to constrain the use of tokens on this endpoint to those that have tokens with at least one of the configured audience claims. | N/A +sso.unauthenticated.path.list | Optional - List of paths that should bypass the SSO flow. | favicon.ico + +### JWT Provider ### + +#### Overview #### +The JWT federation provider accepts JWT tokens as Bearer tokens within the Authorization header of the incoming request. Upon successfully extracting and verifying the token, the request is then processed on behalf of the user represented by the JWT token. + +This provider is closely related to the [Knox Token Service](#KnoxToken+Configuration) and is essentially the provider that is used to consume the tokens issued by the [Knox Token Service](#KnoxToken+Configuration). + +Typical deployments have the KnoxToken service defined in a topology that authenticates users based on username and password with the ShiroProvider. They also have another topology dedicated to clients that wish to use KnoxTokens to access Hadoop resources through Knox. +The following provider configuration can be used with such a topology. + + "providers": [ + { + "role": "federation", + "name": "JWTProvider", + "enabled": "true", + "params": { + "knox.token.audiences": "tokenbased" + } + } + ] + +The `knox.token.audiences` parameter above indicates that any token in an incoming request must contain an audience claim called "tokenbased". In this case, the idea is that the issuing KnoxToken service will be configured to include such an audience claim and that the resulting token is valid to use in the topology that contains configuration like above. This would generally be the name of the topology but you can standardize on anything. + +The following table describes the configuration options for the JWT federation provider: + +##### Descriptions ##### + +Name | Description | Default +---------|-----------|-------- +knox.token.audiences | Optional parameter. This parameter allows the administrator to constrain the use of tokens on this endpoint to those that have tokens with at least one of the configured audience claims. These claims have associated configuration within the KnoxToken service as well. This provides an interesting way to make sure that the token issued based on authentication to a particular LDAP server or other IdP is accepted but not others.|N/A +knox.token.exp.server-managed | Optional parameter for specifying that server-managed token state should be referenced for evaluating token validity. | false +knox.token.verification.pem | Optional parameter that specifies public key used to validate the token. The key must be in PEM encoded format excluding the header and footer lines.| N/A + +The optional `knox.token.exp.server-managed` parameter indicates that Knox is managing the state of tokens it issues (e.g., expiration) external from the token, and this external state should be referenced when validating tokens. This parameter can be ommitted if the global default is configured in gateway-site (see [gateway.knox.token.exp.server-managed](#Gateway+Server+Configuration)), and matches the requirements of this provider. Otherwise, this provider parameter overrides the gateway configuration for the provider's deployment. + +See the [documentation for the Knox Token service](#KnoxToken+Configuration) for related details. Added: knox/trunk/books/2.0.0/config_tls_client_certificate_authentication_provider.md URL: http://svn.apache.org/viewvc/knox/trunk/books/2.0.0/config_tls_client_certificate_authentication_provider.md?rev=1899392&view=auto ============================================================================== --- knox/trunk/books/2.0.0/config_tls_client_certificate_authentication_provider.md (added) +++ knox/trunk/books/2.0.0/config_tls_client_certificate_authentication_provider.md Wed Mar 30 15:22:57 2022 @@ -0,0 +1,31 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + https://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +---> + +## TLS Client Certificate Provider ## + +The TLS client certificate authentication provider enables establishing the user based on the client provided TLS certificate. The user will be the DN from the certificate. This provider requires that the gateway is configured to require client authentication with either `gateway.client.auth.wanted` or `gateway.client.auth.needed` ( #[Mutual Authentication with SSL] ). + +### Configuration ### + +```xml +<provider> + <role>authentication</role> + <name>ClientCert</name> + <enabled>true</enabled> +</provider> +``` +
![https://www.pac4j.org/img/logo-knox.png"](https://www.pac4j.org/img/logo-knox.png"){kind=link}