Added: knox/trunk/books/0.14.0/book_troubleshooting.md URL: http://svn.apache.org/viewvc/knox/trunk/books/0.14.0/book_troubleshooting.md?rev=1809165&view=auto ============================================================================== --- knox/trunk/books/0.14.0/book_troubleshooting.md (added) +++ knox/trunk/books/0.14.0/book_troubleshooting.md Thu Sep 21 13:57:57 2017 @@ -0,0 +1,320 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +---> + +## Troubleshooting ## + +### Finding Logs ### + +When things aren't working the first thing you need to do is examine the diagnostic logs. +Depending upon how you are running the gateway these diagnostic logs will be output to different locations. + +#### java -jar bin/gateway.jar #### + +When the gateway is run this way the diagnostic output is written directly to the console. +If you want to capture that output you will need to redirect the console output to a file using OS specific techniques. + + java -jar bin/gateway.jar > gateway.log + +#### bin/gateway.sh start #### + +When the gateway is run this way the diagnostic output is written to `{GATEWAY_HOME}/log/knox.out` and `{GATEWAY_HOME}/log/knox.err`. +Typically only knox.out will have content. + + +### Increasing Logging ### + +The `log4j.properties` files `{GATEWAY_HOME}/conf` can be used to change the granularity of the logging done by Knox. +The Knox server must be restarted in order for these changes to take effect. +There are various useful loggers pre-populated but commented out. + + log4j.logger.org.apache.hadoop.gateway=DEBUG # Use this logger to increase the debugging of Apache Knox itself. + log4j.logger.org.apache.shiro=DEBUG # Use this logger to increase the debugging of Apache Shiro. + log4j.logger.org.apache.http=DEBUG # Use this logger to increase the debugging of Apache HTTP components. + log4j.logger.org.apache.http.client=DEBUG # Use this logger to increase the debugging of Apache HTTP client component. + log4j.logger.org.apache.http.headers=DEBUG # Use this logger to increase the debugging of Apache HTTP header. + log4j.logger.org.apache.http.wire=DEBUG # Use this logger to increase the debugging of Apache HTTP wire traffic. + + +### LDAP Server Connectivity Issues ### + +If the gateway cannot contact the configured LDAP server you will see errors in the gateway diagnostic output. + + 13/11/15 16:30:17 DEBUG authc.BasicHttpAuthenticationFilter: Attempting to execute login with headers [Basic Z3Vlc3Q6Z3Vlc3QtcGFzc3dvcmQ=] + 13/11/15 16:30:17 DEBUG ldap.JndiLdapRealm: Authenticating user 'guest' through LDAP + 13/11/15 16:30:17 DEBUG ldap.JndiLdapContextFactory: Initializing LDAP context using URL [ldap://localhost:33389] and principal [uid=guest,ou=people,dc=hadoop,dc=apache,dc=org] with pooling disabled + 13/11/15 16:30:17 DEBUG servlet.SimpleCookie: Added HttpServletResponse Cookie [rememberMe=deleteMe; Path=/gateway/vaultservice; Max-Age=0; Expires=Thu, 14-Nov-2013 21:30:17 GMT] + 13/11/15 16:30:17 DEBUG authc.BasicHttpAuthenticationFilter: Authentication required: sending 401 Authentication challenge response. + +The client should see something along the lines of: + + HTTP/1.1 401 Unauthorized + WWW-Authenticate: BASIC realm="application" + Content-Length: 0 + Server: Jetty(8.1.12.v20130726) + +Resolving this will require ensuring that the LDAP server is running and that connection information is correct. +The LDAP server connection information is configured in the cluster's topology file (e.g. {GATEWAY_HOME}/deployments/sandbox.xml). + + +### Hadoop Cluster Connectivity Issues ### + +If the gateway cannot contact one of the services in the configured Hadoop cluster you will see errors in the gateway diagnostic output. + + 13/11/18 18:49:45 WARN hadoop.gateway: Connection exception dispatching request: http://localhost:50070/webhdfs/v1/?user.name=guest&op=LISTSTATUS org.apache.http.conn.HttpHostConnectException: Connection to http://localhost:50070 refused + org.apache.http.conn.HttpHostConnectException: Connection to http://localhost:50070 refused + at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190) + at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294) + at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645) + at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480) + at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) + at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) + at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784) + at org.apache.hadoop.gateway.dispatch.HttpClientDispatch.executeRequest(HttpClientDispatch.java:99) + +The resulting behavior on the client will differ by client. +For the client DSL executing the `{GATEWAY_HOME}/samples/ExampleWebHdfsLs.groovy` the output will look like this. + + Caught: org.apache.hadoop.gateway.shell.HadoopException: org.apache.hadoop.gateway.shell.ErrorResponse: HTTP/1.1 500 Server Error + org.apache.hadoop.gateway.shell.HadoopException: org.apache.hadoop.gateway.shell.ErrorResponse: HTTP/1.1 500 Server Error + at org.apache.hadoop.gateway.shell.AbstractRequest.now(AbstractRequest.java:72) + at org.apache.hadoop.gateway.shell.AbstractRequest$now.call(Unknown Source) + at ExampleWebHdfsLs.run(ExampleWebHdfsLs.groovy:28) + +When executing commands requests via cURL the output might look similar to the following example. + + Set-Cookie: JSESSIONID=16xwhpuxjr8251ufg22f8pqo85;Path=/gateway/sandbox;Secure + Content-Type: text/html;charset=ISO-8859-1 + Cache-Control: must-revalidate,no-cache,no-store + Content-Length: 21856 + Server: Jetty(8.1.12.v20130726) + + <html> + <head> + <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/> + <title>Error 500 Server Error</title> + </head> + <body><h2>HTTP ERROR 500</h2> + +Resolving this will require ensuring that the Hadoop services are running and that connection information is correct. +Basic Hadoop connectivity can be evaluated using cURL as described elsewhere. +Otherwise the Hadoop cluster connection information is configured in the cluster's topology file (e.g. `{GATEWAY_HOME}/deployments/sandbox.xml`). + +### HTTP vs HTTPS protocol issues ### +When Knox is configured to accept requests over SSL and is presented with a request over plain HTTP, the client is presented with an error such as seen in the following: + + curl -i -k -u guest:guest-password -X GET 'http://localhost:8443/gateway/sandbox/webhdfs/v1/?op=LISTSTATUS' + the following error is returned + curl: (52) Empty reply from server + +This is the default behavior for Jetty SSL listener. While the credentials to the default authentication provider continue to be username and password, we do not want to encourage sending these in clear text. Since preemptively sending BASIC credentials is a common pattern with REST APIs it would be unwise to redirect to a HTTPS listener thus allowing clear text passwords. + +To resolve this issue, we have two options: + +1. change the scheme in the URL to https and deal with any trust relationship issues with the presented server certificate +2. Disabling SSL in gateway-site.xml - this is not encouraged due to the reasoning described above. + +### Check Hadoop Cluster Access via cURL ### + +When you are experiencing connectivity issue it can be helpful to "bypass" the gateway and invoke the Hadoop REST APIs directly. +This can easily be done using the cURL command line utility or many other REST/HTTP clients. +Exactly how to use cURL depends on the configuration of your Hadoop cluster. +In general however you will use a command line the one that follows. + + curl -ikv -X GET 'http://namenode-host:50070/webhdfs/v1/?op=LISTSTATUS' + +If you are using Sandbox the WebHDFS or NameNode port will be mapped to localhost so this command can be used. + + curl -ikv -X GET 'http://localhost:50070/webhdfs/v1/?op=LISTSTATUS' + +If you are using a cluster secured with Kerberos you will need to have used `kinit` to authenticate to the KDC. +Then the command below should verify that WebHDFS in the Hadoop cluster is accessible. + + curl -ikv --negotiate -u : -X 'http://localhost:50070/webhdfs/v1/?op=LISTSTATUS' + + +### Authentication Issues ### +The following log information is available when you enable debug level logging for shiro. This can be done within the conf/log4j.properties file. Not the "Password not correct for user" message. + + 13/11/15 16:37:15 DEBUG authc.BasicHttpAuthenticationFilter: Attempting to execute login with headers [Basic Z3Vlc3Q6Z3Vlc3QtcGFzc3dvcmQw] + 13/11/15 16:37:15 DEBUG ldap.JndiLdapRealm: Authenticating user 'guest' through LDAP + 13/11/15 16:37:15 DEBUG ldap.JndiLdapContextFactory: Initializing LDAP context using URL [ldap://localhost:33389] and principal [uid=guest,ou=people,dc=hadoop,dc=apache,dc=org] with pooling disabled + 2013-11-15 16:37:15,899 INFO Password not correct for user 'uid=guest,ou=people,dc=hadoop,dc=apache,dc=org' + 2013-11-15 16:37:15,899 INFO Authenticator org.apache.directory.server.core.authn.SimpleAuthenticator@354c78e3 failed to authenticate: BindContext for DN 'uid=guest,ou=people,dc=hadoop,dc=apache,dc=org', credentials <0x67 0x75 0x65 0x73 0x74 0x2D 0x70 0x61 0x73 0x73 0x77 0x6F 0x72 0x64 0x30 > + 2013-11-15 16:37:15,899 INFO Cannot bind to the server + 13/11/15 16:37:15 DEBUG servlet.SimpleCookie: Added HttpServletResponse Cookie [rememberMe=deleteMe; Path=/gateway/vaultservice; Max-Age=0; Expires=Thu, 14-Nov-2013 21:37:15 GMT] + 13/11/15 16:37:15 DEBUG authc.BasicHttpAuthenticationFilter: Authentication required: sending 401 Authentication challenge response. + +The client will likely see something along the lines of: + + HTTP/1.1 401 Unauthorized + WWW-Authenticate: BASIC realm="application" + Content-Length: 0 + Server: Jetty(8.1.12.v20130726) + +#### Using ldapsearch to verify LDAP connectivity and credentials + +If your authentication to Knox fails and you believe your are using correct credentials, you could try to verify the connectivity and credentials using ldapsearch, assuming you are using LDAP directory for authentication. + +Assuming you are using the default values that came out of box with Knox, your ldapsearch command would be like the following + + ldapsearch -h localhost -p 33389 -D "uid=guest,ou=people,dc=hadoop,dc=apache,dc=org" -w guest-password -b "uid=guest,ou=people,dc=hadoop,dc=apache,dc=org" "objectclass=*" + +This should produce output like the following + + # extended LDIF + + LDAPv3 + base <uid=guest,ou=people,dc=hadoop,dc=apache,dc=org> with scope subtree + filter: objectclass=* + requesting: ALL + + + # guest, people, hadoop.apache.org + dn: uid=guest,ou=people,dc=hadoop,dc=apache,dc=org + objectClass: organizationalPerson + objectClass: person + objectClass: inetOrgPerson + objectClass: top + uid: guest + cn: Guest + sn: User + userpassword:: Z3Vlc3QtcGFzc3dvcmQ= + + # search result + search: 2 + result: 0 Success + + # numResponses: 2 + # numEntries: 1 + +In a more general form the ldapsearch command would be + + ldapsearch -h {HOST} -p {PORT} -D {DN of binding user} -w {bind password} -b {DN of binding user} "objectclass=*} + +### Hostname Resolution Issues ### + +The deployments/sandbox.xml topology file has the host mapping feature enabled. +This is required due to the way networking is setup in the Sandbox VM. +Specifically the VM's internal hostname is sandbox.hortonworks.com. +Since this hostname cannot be resolved to the actual VM Knox needs to map that hostname to something resolvable. + +If for example host mapping is disabled but the Sandbox VM is still used you will see an error in the diagnostic output similar to the below. + + 13/11/18 19:11:35 WARN hadoop.gateway: Connection exception dispatching request: http://sandbox.hortonworks.com:50075/webhdfs/v1/user/guest/example/README?op=CREATE&namenoderpcaddress=sandbox.hortonworks.com:8020&user.name=guest&overwrite=false java.net.UnknownHostException: sandbox.hortonworks.com + java.net.UnknownHostException: sandbox.hortonworks.com + at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) + +On the other hand if you are migrating from the Sandbox based configuration to a cluster you have deployment you may see a similar error. +However in this case you may need to disable host mapping. +This can be done by modifying the topology file (e.g. deployments/sandbox.xml) for the cluster. + + ... + <provider> + <role>hostmap</role> + <name>static</name> + <enabled>false</enabled> + <param><name>localhost</name><value>sandbox,sandbox.hortonworks.com</value></param> + </provider> + .... + + +### Job Submission Issues - HDFS Home Directories ### + +If you see error like the following in your console while submitting a Job using groovy shell, it is likely that the authenticated user does not have a home directory on HDFS. + + Caught: org.apache.hadoop.gateway.shell.HadoopException: org.apache.hadoop.gateway.shell.ErrorResponse: HTTP/1.1 403 Forbidden + org.apache.hadoop.gateway.shell.HadoopException: org.apache.hadoop.gateway.shell.ErrorResponse: HTTP/1.1 403 Forbidden + +You would also see this error if you try file operation on the home directory of the authenticating user. + +The error would look a little different as shown below if you are attempting to the operation with cURL. + + {"RemoteException":{"exception":"AccessControlException","javaClassName":"org.apache.hadoop.security.AccessControlException","message":"Permission denied: user=tom, access=WRITE, inode=\"/user\":hdfs:hdfs:drwxr-xr-x"}}* + +#### Resolution + +Create the home directory for the user on HDFS. +The home directory is typically of the form `/user/{userid}` and should be owned by the user. +user 'hdfs' can create such a directory and make the user owner of the directory. + + +### Job Submission Issues - OS Accounts ### + +If the Hadoop cluster is not secured with Kerberos, the user submitting a job need not have an OS account on the Hadoop Nodemanagers. + +If the Hadoop cluster is secured with Kerberos, the user submitting the job should have an OS account on Hadoop Nodemanagers. + +In either case if the user does not have such OS account, his file permissions are based on user ownership of files or "other" permission in "ugo" posix permission. +The user does not get any file permission as a member of any group if you are using default hadoop.security.group.mapping. + +TODO: add sample error message from running test on secure cluster with missing OS account + +### HBase Issues ### + +If you experience problems running the HBase samples with the Sandbox VM it may be necessary to restart HBase and the HBASE REST API. +This can sometimes occur with the Sandbox VM is restarted from a saved state. +If the client hangs after emitting the last line in the sample output below you are most likely affected. + + System version : {...} + Cluster version : 0.96.0.2.0.6.0-76-hadoop2 + Status : {...} + Creating table 'test_table'... + +HBase and the HBASE REST API can be restarted using the following commands on the Hadoop Sandbox VM. +You will need to ssh into the VM in order to run these commands. + + sudo -u hbase /usr/lib/hbase/bin/hbase-daemon.sh stop master + sudo -u hbase /usr/lib/hbase/bin/hbase-daemon.sh start master + sudo -u hbase /usr/lib/hbase/bin/hbase-daemon.sh restart rest + + +### SSL Certificate Issues ### + +Clients that do not trust the certificate presented by the server will behave in different ways. +A browser will typically warn you of the inability to trust the received certificate and give you an opportunity to add an exception for the particular certificate. +Curl will present you with the follow message and instructions for turning of certificate verification: + + curl performs SSL certificate verification by default, using a "bundle" + of Certificate Authority (CA) public keys (CA certs). If the default + bundle file isn't adequate, you can specify an alternate file + using the --cacert option. + If this HTTPS server uses a certificate signed by a CA represented + the bundle, the certificate verification probably failed due to a + problem with the certificate (it might be expired, or the name might + not match the domain name in the URL). + If you'd like to turn off curl's verification of the certificate, use + the -k (or --insecure) option. + + +### SPNego Authentication Issues ### + +Calls from Knox to Secure Hadoop Cluster fails, with SPNego authentication problems, +if there was a TGT for Knox in disk cache when Knox was started. + +You are likely to run into this situation on developer machines where the developer could have kinited for some testing. + +Work Around: clear TGT of Knox from disk cache (calling `kdestroy` would do it), before starting Knox + +### Filing Bugs ### + +Bugs can be filed using [Jira][jira]. +Please include the results of this command below in the Environment section. +Also include the version of Hadoop being used in the same section. + + cd {GATEWAY_HOME} + java -jar bin/gateway.jar -version +
Added: knox/trunk/books/0.14.0/book_ui_service_details.md URL: http://svn.apache.org/viewvc/knox/trunk/books/0.14.0/book_ui_service_details.md?rev=1809165&view=auto ============================================================================== --- knox/trunk/books/0.14.0/book_ui_service_details.md (added) +++ knox/trunk/books/0.14.0/book_ui_service_details.md Thu Sep 21 13:57:57 2017 @@ -0,0 +1,444 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +## UI Service Details ## + +In the sections that follow, the integrations for proxying various UIs currently available out of the box with the +gateway will be described. These sections will include examples that demonstrate how to access each of these services +via the gateway. + +These are the current Hadoop services with built-in support for their UIs. + +* #[Name Node UI] +* #[Job History UI] +* #[Oozie UI] +* #[HBase UI] +* #[Yarn UI] +* #[Spark UI] +* #[Ambari UI] +* #[Ranger Admin Console] +* #[Atlas UI] +* #[Zeppelin UI] + +### Assumptions + +This section assumes an environment setup similar to the one in the REST services section #[Service Details] + +### Name Node UI ### + +The Name Node UI is available on the same host and port combination that WebHDFS is available on. As mentioned in the +WebHDFS REST service configuration section, the values for the host and port can be obtained from the following +properties in hdfs-site.xml + + <property> + <name>dfs.namenode.http-address</name> + <value>sandbox.hortonworks.com:50070</value> + </property> + <property> + <name>dfs.https.namenode.https-address</name> + <value>sandbox.hortonworks.com:50470</value> + </property> + +The values above need to be reflected in each topology descriptor file deployed to the gateway. +The gateway by default includes a sample topology descriptor file `{GATEWAY_HOME}/deployments/sandbox.xml`. +The values in this sample are configured to work with an installed Sandbox VM. + + <service> + <role>HDFSUI</role> + <url>http://sandbox.hortonworks.com:50070/webhdfs</url> + </service> + +In addition to the service configuration for HDFSUI, the REST service configuration for WEBHDFS is also required. + + <service> + <role>NAMENODE</role> + <url>hdfs://sandbox.hortonworks.com:8020</url> + </service> + <service> + <role>WEBHDFS</role> + <url>http://sandbox.hortonworks.com:50070/webhdfs</url> + </service> + +By default the gateway is configured to use the HTTP endpoint for WebHDFS in the Sandbox. +This could alternatively be configured to use the HTTPS endpoint by providing the correct address. + +#### Name Node UI URL Mapping #### + +For Name Node UI URLs, the mapping of Knox Gateway accessible HDFS UI URLs to direct HDFS UI URLs is: + +| ------- | ----------------------------------------------------------------------------- | +| Gateway | `https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/hdfs` | +| Cluster | `http://{webhdfs-host}:50070/` | + +For example to browse the file system using the NameNode UI the URL in a web browser would be: + + http://sandbox.hortonworks.com:50070/explorer.html# + +And using the gateway to access the same page the URL would be (where the gateway host:port is 'localhost:8443') + + https://localhost:8443/gateway/sandbox/hdfs/explorer.html# + + +### Job History UI ### + +The Job History UI service can be configured in a topology by adding the following snippet. The values in this sample +are configured to work with an installed Sandbox VM. + + <service> + <role>JOBHISTORYUI</role> + <url>http://sandbox.hortonworks.com:19888</url> + </service> + +The values for the host and port can be obtained from the following property in mapred-site.xml + + <property> + <name>mapreduce.jobhistory.webapp.address</name> + <value>sandbox.hortonworks.com:19888</value> + </property> + + + +#### Job History UI URL Mapping #### + +For Job History UI URLs, the mapping of Knox Gateway accessible Job History UI URLs to direct Job History UI URLs is: + +| ------- | ----------------------------------------------------------------------------- | +| Gateway | `https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/jobhistory` | +| Cluster | `http://{jobhistory-host}:19888/jobhistory` | + + +### Oozie UI ### + +The Oozie UI service can be configured in a topology by adding the following snippet. The values in this sample +are configured to work with an installed Sandbox VM. + + <service> + <role>OOZIEUI</role> + <url>http://sandbox.hortonworks.com:11000/oozie</url> + </service> + +The value for the URL can be obtained from the following property in oozie-site.xml + + <property> + <name>oozie.base.url</name> + <value>http://sandbox.hortonworks.com:11000/oozie</value> + </property> + + + +#### Oozie UI URL Mapping #### + +For Oozie UI URLs, the mapping of Knox Gateway accessible Oozie UI URLs to direct Oozie UI URLs is: + +| ------- | ----------------------------------------------------------------------------- | +| Gateway | `https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/oozie/` | +| Cluster | `http://{oozie-host}:11000/oozie/` | + + +### HBase UI ### + +The HBase UI service can be configured in a topology by adding the following snippet. The values in this sample +are configured to work with an installed Sandbox VM. + + <service> + <role>HBASEUI</role> + <url>http://sandbox.hortonworks.com:16010</url> + </service> + +The values for the host and port can be obtained from the following property in hbase-site.xml. +Below the hostname of the hbase master is used since the bindAddress is 0.0.0.0 + + <property> + <name>hbase.master.info.bindAddress</name> + <value>0.0.0.0</value> + </property> + <property> + <name>hbase.master.info.port</name> + <value>16010</value> + </property> + +#### HBase UI URL Mapping #### + +For HBase UI URLs, the mapping of Knox Gateway accessible HBase UI URLs to direct HBase Master +UI URLs is: + +| ------- | -------------------------------------------------------------------------------------| +| Gateway | `https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/hbase/webui/` | +| Cluster | `http://{hbase-master-host}:16010/` | + +### YARN UI ### + +The YARN UI service can be configured in a topology by adding the following snippet. The values in this sample +are configured to work with an installed Sandbox VM. + + <service> + <role>YARNUI</role> + <url>http://sandbox.hortonworks.com:8088</url> + </service> + +The values for the host and port can be obtained from the following property in mapred-site.xml + + <property> + <name>yarn.resourcemanager.webapp.address</name> + <value>sandbox.hortonworks.com:8088</value> + </property> + +#### YARN UI URL Mapping #### + +For Resource Manager UI URLs, the mapping of Knox Gateway accessible Resource Manager UI URLs to direct Resource Manager +UI URLs is: + +| ------- | ----------------------------------------------------------------------------- | +| Gateway | `https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/yarn` | +| Cluster | `http://{resource-manager-host}:8088/cluster` | + +### Spark UI ### + +The Spark History UI service can be configured in a topology by adding the following snippet. The values in this sample +are configured to work with an installed Sandbox VM. + + + <service> + <role>SPARKHISTORYUI</role> + <url>http://sandbox.hortonworks.com:18080/</url> + </service> + +#### Spark History UI URL Mapping #### + +For Spark History UI URLs, the mapping of Knox Gateway accessible Spark History UI URLs to direct Spark History +UI URLs is: + +| ------- | ----------------------------------------------------------------------------- | +| Gateway | `https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/sparkhistory` | +| Cluster | `http://{spark-history-host}:18080` | + + +### Ambari UI ### + +Ambari UI has functionality around provisioning and managing services in a hadoop cluster. This UI can now be used +behind the Knox gateway. + + +To enable this functionality, a topology file needs to have the following configuration: + + <service> + <role>AMBARIUI</role> + <url>http://<hostname>:<port></url> + </service> + +The default Ambari http port is 8080. Also please note that the UI service also requires the Ambari REST API service + to be enabled to function properly. An example of a more complete topology is given below. + + +#### Ambari UI URL Mapping #### + +For Ambari UI URLs, the mapping of Knox Gateway accessible URLs to direct Ambari UI URLs is the following. + +| ------- | ------------------------------------------------------------------------------------- | +| Gateway | `https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/ambari/` | +| Cluster | `http://{ambari-host}:{ambari-port}/}` | + +#### Example Topology #### + +The Ambari UI service may require a separate topology file due to its requirements around authentication. Knox passes +through authentication challenge and credentials to the service in this case. + + + <topology> + <gateway> + <provider> + <role>authentication</role> + <name>Anonymous</name> + <enabled>true</enabled> + </provider> + <provider> + <role>identity-assertion</role> + <name>Default</name> + <enabled>false</enabled> + </provider> + </gateway> + <service> + <role>AMBARI</role> + <url>http://localhost:8080</url> + </service> + + <service> + <role>AMBARIUI</role> + <url>http://localhost:8080</url> + </service> + </topology> + +Please look at JIRA issue [KNOX-705] for a known issue with this release. + +### Ranger Admin Console ### + +The Ranger Admin console can now be used behind the Knox gateway. + +To enable this functionality, a topology file needs to have the following configuration: + + <service> + <role>RANGERUI</role> + <url>http://<hostname>:<port></url> + </service> + +The default Ranger http port is 8060. Also please note that the UI service also requires the Ranger REST API service + to be enabled to function properly. An example of a more complete topology is given below. + + +#### Ranger Admin Console URL Mapping #### + +For Ranger Admin console URLs, the mapping of Knox Gateway accessible URLs to direct Ranger Admin console URLs is the following. + +| ------- | ------------------------------------------------------------------------------------- | +| Gateway | `https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/ranger/` | +| Cluster | `http://{ranger-host}:{ranger-port}/}` | + +#### Example Topology #### + +The Ranger UI service may require a separate topology file due to its requirements around authentication. Knox passes +through authentication challenge and credentials to the service in this case. + + + <topology> + <gateway> + <provider> + <role>authentication</role> + <name>Anonymous</name> + <enabled>true</enabled> + </provider> + <provider> + <role>identity-assertion</role> + <name>Default</name> + <enabled>false</enabled> + </provider> + </gateway> + <service> + <role>RANGER</role> + <url>http://localhost:8060</url> + </service> + + <service> + <role>RANGERUI</role> + <url>http://localhost:8060</url> + </service> + </topology> + + </service> + </topology> + + + +### Atlas UI ### + +### Atlas Rest API ### + +The Atlas Rest API can now be used behind the Knox gateway. +To enable this functionality, a topology file needs to have the following configuration. + + <service> + <role>ATLAS-API</role> + <url>http://<ATLAS_HOST>:<ATLAS_PORT></url> + </service> + +The default Atlas http port is 21000. Also please note that the UI service also requires the Atlas REST API +service to be enabled to function properly. An example of a more complete topology is given below. + +Atlas Rest API URL Mapping +For Atlas Rest URLs, the mapping of Knox Gateway accessible URLs to direct Atlas Rest URLs is the following. + + +| ------- | ------------------------------------------------------------------------------------- | +| Gateway | `https://{gateway-host}:{gateway-port}/{gateway-path}/{topology}/atlas/` | +| Cluster | `http://{atlas-host}:{atlas-port}/}` | + + + +Access Atlas Api using Curl call + + curl -i -k -L -u admin:admin -X GET \ + 'https://knox-gateway:8443/gateway/{topology}/atlas/api/atlas/v2/types/typedefs?type=classification&_=1495442879421' + + +### Atlas UI ### +In addition to the Atlas REST API, from this release there is the ability to access some of the functionality via a web. The initial functionality is very limited and serves more as a starting point/placeholder. The details are below. +Atlas UI URL + +The URL mapping for the Atlas UI is: + +| ------- | ------------------------------------------------------------------------------------- | +|Gateway | `https://{gateway-host}:{gateway-port}/{gateway-path}/{topology}/atlas/index.html` + +#### Example Topology for Atlas #### + + <topology> + <gateway> + <provider> + <role>authentication</role> + <name>Anonymous</name> + <enabled>true</enabled> + </provider> + <provider> + <role>identity-assertion</role> + <name>Default</name> + <enabled>false</enabled> + </provider> + </gateway> + + <service> + <role>ATLAS-API</role> + <url>http://<ATLAS_HOST>:<ATLAS_PORT></url> + </service> + + <service> + <role>ATLAS</role> + <url>http://<ATLAS_HOST>:<ATLAS_PORT></url> + </service> + </topology> + Atlas + +Note : - This feature will allow for 'anonymous' authentication. Essentially bypassing any LDAP or other authentication done by Knox and allow the proxied service to do the actual authentication. + + +### Zeppelin UI ### +Apache Knox can be used to proxy Zeppelin UI and also supports websocket protocol used by Zeppelin. + +The URL mapping for the Zeppelin UI is: + +| ------- | ------------------------------------------------------------------------------------- | +|Gateway | `https://{gateway-host}:{gateway-port}/{gateway-path}/{topology}/zeppelin/` + +By default websocket functionality is disabled, it needs to be enabed for Zeppelin UI to work properly, it can enable it by changing the `gateway.websocket.feature.enabled` property to 'true' in `<KNOX-HOME>/conf/gateway-site.xml` file, for e.g. + + <property> + <name>gateway.websocket.feature.enabled</name> + <value>true</value> + <description>Enable/Disable websocket feature.</description> + </property> + +Example service definition for Zeppelin in topology file is as follows, note that both ZEPPELINWS and ZEPPELINUI service declarations are required. + + <service> + <role>ZEPPELINWS</role> + <url>ws://<ZEPPELIN_HOST>:<ZEPPELIN_PORT>/ws</url> + </service> + + <service> + <role>ZEPPELINUI</role> + <url>http://<ZEPPELIN_HOST>:<ZEPPELIN_PORT></url> + </service> + +Knox also supports secure Zeppelin UIs, for secure UIs one needs to provision Zeppelin certificate into Knox truststore. \ No newline at end of file Added: knox/trunk/books/0.14.0/config.md URL: http://svn.apache.org/viewvc/knox/trunk/books/0.14.0/config.md?rev=1809165&view=auto ============================================================================== --- knox/trunk/books/0.14.0/config.md (added) +++ knox/trunk/books/0.14.0/config.md Thu Sep 21 13:57:57 2017 @@ -0,0 +1,540 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +---> + +### Configuration ### + +Configuration for Apache Knox includes: + +1. #[Related Cluster Configuration] that must be done within the Hadoop cluster to allow Knox to communicate with various services +2. #[Gateway Server Configuration] - which is the configurable elements of the server itself which applies to behavior that spans all topologies or managed Hadoop clusters +3. #[Topology Descriptors] which are the descriptors for controlling access to Hadoop clusters in various ways + +### Related Cluster Configuration ### + +The following configuration changes must be made to your cluster to allow Apache Knox to +dispatch requests to the various service components on behalf of end users. + +#### Grant Proxy privileges for Knox user in `core-site.xml` on Hadoop master nodes #### + +Update `core-site.xml` and add the following lines towards the end of the file. + +Replace `FQDN_OF_KNOX_HOST` with the fully qualified domain name of the host running the Knox gateway. +You can usually find this by running `hostname -f` on that host. + +You can use `*` for local developer testing if the Knox host does not have a static IP. + + <property> + <name>hadoop.proxyuser.knox.groups</name> + <value>users</value> + </property> + <property> + <name>hadoop.proxyuser.knox.hosts</name> + <value>FQDN_OF_KNOX_HOST</value> + </property> + +#### Grant proxy privilege for Knox in `webhcat-site.xml` on Hadoop master nodes #### + +Update `webhcat-site.xml` and add the following lines towards the end of the file. + +Replace `FQDN_OF_KNOX_HOST` with the fully qualified domain name of the host running the Knox gateway. +You can use `*` for local developer testing if the Knox host does not have a static IP. + + <property> + <name>webhcat.proxyuser.knox.groups</name> + <value>users</value> + </property> + <property> + <name>webhcat.proxyuser.knox.hosts</name> + <value>FQDN_OF_KNOX_HOST</value> + </property> + +#### Grant proxy privilege for Knox in `oozie-site.xml` on Oozie host #### + +Update `oozie-site.xml` and add the following lines towards the end of the file. + +Replace `FQDN_OF_KNOX_HOST` with the fully qualified domain name of the host running the Knox gateway. +You can use `*` for local developer testing if the Knox host does not have a static IP. + + <property> + <name>oozie.service.ProxyUserService.proxyuser.knox.groups</name> + <value>users</value> + </property> + <property> + <name>oozie.service.ProxyUserService.proxyuser.knox.hosts</name> + <value>FQDN_OF_KNOX_HOST</value> + </property> + +#### Enable http transport mode and use substitution in HiveServer2 #### + +Update `hive-site.xml` and set the following properties on HiveServer2 hosts. +Some of the properties may already be in the hive-site.xml. +Ensure that the values match the ones below. + + <property> + <name>hive.server2.allow.user.substitution</name> + <value>true</value> + </property> + + <property> + <name>hive.server2.transport.mode</name> + <value>http</value> + <description>Server transport mode. "binary" or "http".</description> + </property> + + <property> + <name>hive.server2.thrift.http.port</name> + <value>10001</value> + <description>Port number when in HTTP mode.</description> + </property> + + <property> + <name>hive.server2.thrift.http.path</name> + <value>cliservice</value> + <description>Path component of URL endpoint when in HTTP mode.</description> + </property> + +#### Gateway Server Configuration #### + +The following table illustrates the configurable elements of the Apache Knox Gateway at the server level via gateway-site.xml. + +property | description | default +------------|-----------|----------- +gateway.deployment.dir|The directory within GATEWAY_HOME that contains gateway topology deployments.|{GATEWAY_HOME}/data/deployments +gateway.security.dir|The directory within GATEWAY_HOME that contains the required security artifacts|{GATEWAY_HOME}/data/security +gateway.data.dir|The directory within GATEWAY_HOME that contains the gateway instance data|{GATEWAY_HOME}/data +gateway.services.dir|The directory within GATEWAY_HOME that contains the gateway services definitions.|{GATEWAY_HOME}/services +gateway.hadoop.conf.dir|The directory within GATEWAY_HOME that contains the gateway configuration|{GATEWAY_HOME}/conf +gateway.frontend.url|The URL that should be used during rewriting so that it can rewrite the URLs with the correct "frontend" URL|none +gateway.xforwarded.enabled|Indicates whether support for some X-Forwarded-* headers is enabled|true +gateway.trust.all.certs|Indicates whether all presented client certs should establish trust|false +gateway.client.auth.needed|Indicates whether clients are required to establish a trust relationship with client certificates|false +gateway.truststore.path|Location of the truststore for client certificates to be trusted|gateway.jks +gateway.truststore.type|Indicates the type of truststore|JKS +gateway.keystore.type|Indicates the type of keystore for the identity store|JKS +gateway.jdk.tls.ephemeralDHKeySize|jdk.tls.ephemeralDHKeySize, is defined to customize the ephemeral DH key sizes. The minimum acceptable DH key size is 1024 bits, except for exportable cipher suites or legacy mode (jdk.tls.ephemeralDHKeySize=legacy)|2048 +gateway.threadpool.max|The maximum concurrent requests the server will process. The default is 254. Connections beyond this will be queued.|254 +gateway.httpclient.maxConnections|The maximum number of connections that a single httpclient will maintain to a single host:port. The default is 32.|32 +gateway.httpclient.connectionTimeout|The amount of time to wait when attempting a connection. The natural unit is milliseconds but a 's' or 'm' suffix may be used for seconds or minutes respectively. The default timeout is system dependent. | System Dependent +gateway.httpclient.socketTimeout|The amount of time to wait for data on a socket before aborting the connection. The natural unit is milliseconds but a 's' or 'm' suffix may be used for seconds or minutes respectively. The default timeout is system dependent but is likely to be indefinite. | System Dependent +gateway.httpserver.requestBuffer|The size of the HTTP server request buffer. The default is 16K.|16384 +gateway.httpserver.requestHeaderBuffer|The size of the HTTP server request header buffer. The default is 8K.|8192 +gateway.httpserver.responseBuffer|The size of the HTTP server response buffer. The default is 32K.|32768 +gateway.httpserver.responseHeaderBuffer|The size of the HTTP server response header buffer. The default is 8K.|8192 +gateway.websocket.feature.enabled|Enable/Disable websocket feature.|false +gateway.gzip.compress.mime.types|Content types to be gzip compressed by Knox on the way out to browser.|text/html, text/plain, text/xml, text/css, application/javascript, text/javascript, application/x-javascript +gateway.signing.keystore.name|OPTIONAL Filename of keystore file that contains the signing keypair. NOTE: An alias needs to be created using "knoxcli.sh create-alias" for the alias name signing.key.passphrase in order to provide the passphrase to access the keystore.|null +gateway.signing.key.alias|OPTIONAL alias for the signing keypair within the keystore specified via gateway.signing.keystore.name.|null +ssl.enabled|Indicates whether SSL is enabled for the Gateway|true +ssl.include.ciphers|A comma separated list of ciphers to accept for SSL. See the [JSSE Provider docs](http://docs.oracle.com/javase/8/docs/technotes/guides/security/SunProviders.html#SunJSSEProvider) for possible ciphers. These can also contain regular expressions as shown in the [Jetty documentation](http://www.eclipse.org/jetty/documentation/current/configuring-ssl.html).|all| +ssl.exclude.ciphers|A comma separated list of ciphers to reject for SSL. See the [JSSE Provider docs](http://docs.oracle.com/javase/8/docs/technotes/guides/security/SunProviders.html#SunJSSEProvider) for possible ciphers. These can also contain regular expressions as shown in the [Jetty documentation](http://www.eclipse.org/jetty/documentation/current/configuring-ssl.html).|none| +ssl.exclude.protocols|Excludes a comma separated list of protocols to not accept for SSL or "none"|SSLv3 + + +#### Topology Descriptors #### + +The topology descriptor files provide the gateway with per-cluster configuration information. +This includes configuration for both the providers within the gateway and the services within the Hadoop cluster. +These files are located in `{GATEWAY_HOME}/conf/topologies`. +The general outline of this document looks like this. + + <topology> + <gateway> + <provider> + </provider> + </gateway> + <service> + </service> + </topology> + +There are typically multiple `<provider>` and `<service>` elements. + +/topology +: Defines the provider and configuration and service topology for a single Hadoop cluster. + +/topology/gateway +: Groups all of the provider elements + +/topology/gateway/provider +: Defines the configuration of a specific provider for the cluster. + +/topology/service +: Defines the location of a specific Hadoop service within the Hadoop cluster. + +##### Provider Configuration ##### + +Provider configuration is used to customize the behavior of a particular gateway feature. +The general outline of a provider element looks like this. + + <provider> + <role>authentication</role> + <name>ShiroProvider</name> + <enabled>true</enabled> + <param> + <name></name> + <value></value> + </param> + </provider> + +/topology/gateway/provider +: Groups information for a specific provider. + +/topology/gateway/provider/role +: Defines the role of a particular provider. +There are a number of pre-defined roles used by out-of-the-box provider plugins for the gateway. +These roles are: authentication, identity-assertion, rewrite and hostmap + +/topology/gateway/provider/name +: Defines the name of the provider for which this configuration applies. +There can be multiple provider implementations for a given role. +Specifying the name is used to identify which particular provider is being configured. +Typically each topology descriptor should contain only one provider for each role but there are exceptions. + +/topology/gateway/provider/enabled +: Allows a particular provider to be enabled or disabled via `true` or `false` respectively. +When a provider is disabled any filters associated with that provider are excluded from the processing chain. + +/topology/gateway/provider/param +: These elements are used to supply provider configuration. +There can be zero or more of these per provider. + +/topology/gateway/provider/param/name +: The name of a parameter to pass to the provider. + +/topology/gateway/provider/param/value +: The value of a parameter to pass to the provider. + +##### Service Configuration ##### + +Service configuration is used to specify the location of services within the Hadoop cluster. +The general outline of a service element looks like this. + + <service> + <role>WEBHDFS</role> + <url>http://localhost:50070/webhdfs</url> + </service> + +/topology/service +: Provider information about a particular service within the Hadoop cluster. +Not all services are necessarily exposed as gateway endpoints. + +/topology/service/role +: Identifies the role of this service. +Currently supported roles are: WEBHDFS, WEBHCAT, WEBHBASE, OOZIE, HIVE, NAMENODE, JOBTRACKER, RESOURCEMANAGER +Additional service roles can be supported via plugins. + +topology/service/url +: The URL identifying the location of a particular service within the Hadoop cluster. + +#### Hostmap Provider #### + +The purpose of the Hostmap provider is to handle situations where hosts are known by one name within the cluster and another name externally. +This frequently occurs when virtual machines are used and in particular when using cloud hosting services. +Currently, the Hostmap provider is configured as part of the topology file. +The basic structure is shown below. + + <topology> + <gateway> + ... + <provider> + <role>hostmap</role> + <name>static</name> + <enabled>true</enabled> + <param><name>external-host-name</name><value>internal-host-name</value></param> + </provider> + ... + </gateway> + ... + </topology> + +This mapping is required because the Hadoop services running within the cluster are unaware that they are being accessed from outside the cluster. +Therefore URLs returned as part of REST API responses will typically contain internal host names. +Since clients outside the cluster will be unable to resolve those host name they must be mapped to external host names. + +##### Hostmap Provider Example - EC2 ##### + +Consider an EC2 example where two VMs have been allocated. +Each VM has an external host name by which it can be accessed via the internet. +However the EC2 VM is unaware of this external host name and instead is configured with the internal host name. + + External HOSTNAMES: + ec2-23-22-31-165.compute-1.amazonaws.com + ec2-23-23-25-10.compute-1.amazonaws.com + + Internal HOSTNAMES: + ip-10-118-99-172.ec2.internal + ip-10-39-107-209.ec2.internal + +The Hostmap configuration required to allow access external to the Hadoop cluster via the Apache Knox Gateway would be this. + + <topology> + <gateway> + ... + <provider> + <role>hostmap</role> + <name>static</name> + <enabled>true</enabled> + <param> + <name>ec2-23-22-31-165.compute-1.amazonaws.com</name> + <value>ip-10-118-99-172.ec2.internal</value> + </param> + <param> + <name>ec2-23-23-25-10.compute-1.amazonaws.com</name> + <value>ip-10-39-107-209.ec2.internal</value> + </param> + </provider> + ... + </gateway> + ... + </topology> + +##### Hostmap Provider Example - Sandbox ##### + +The Hortonworks Sandbox 2.x poses a different challenge for host name mapping. +This version of the Sandbox uses port mapping to make the Sandbox VM appear as though it is accessible via localhost. +However the Sandbox VM is internally configured to consider sandbox.hortonworks.com as the host name. +So from the perspective of a client accessing Sandbox the external host name is localhost. +The Hostmap configuration required to allow access to Sandbox from the host operating system is this. + + <topology> + <gateway> + ... + <provider> + <role>hostmap</role> + <name>static</name> + <enabled>true</enabled> + <param> + <name>localhost</name> + <value>sandbox,sandbox.hortonworks.com</value> + </param> + </provider> + ... + </gateway> + ... + </topology> + +##### Hostmap Provider Configuration ##### + +Details about each provider configuration element is enumerated below. + +topology/gateway/provider/role +: The role for a Hostmap provider must always be `hostmap`. + +topology/gateway/provider/name +: The Hostmap provider supplied out-of-the-box is selected via the name `static`. + +topology/gateway/provider/enabled +: Host mapping can be enabled or disabled by providing `true` or `false`. + +topology/gateway/provider/param +: Host mapping is configured by providing parameters for each external to internal mapping. + +topology/gateway/provider/param/name +: The parameter names represent the external host names associated with the internal host names provided by the value element. +This can be a comma separated list of host names that all represent the same physical host. +When mapping from internal to external host name the first external host name in the list is used. + +topology/gateway/provider/param/value +: The parameter values represent the internal host names associated with the external host names provider by the name element. +This can be a comma separated list of host names that all represent the same physical host. +When mapping from external to internal host names the first internal host name in the list is used. + + +#### Logging #### + +If necessary you can enable additional logging by editing the `log4j.properties` file in the `conf` directory. +Changing the `rootLogger` value from `ERROR` to `DEBUG` will generate a large amount of debug logging. +A number of useful, more fine loggers are also provided in the file. + + +#### Java VM Options #### + +TODO - Java VM options doc. + + +#### Persisting the Master Secret #### + +The master secret is required to start the server. +This secret is used to access secured artifacts by the gateway instance. +Keystore, trust stores and credential stores are all protected with the master secret. + +You may persist the master secret by supplying the *\-persist-master* switch at startup. +This will result in a warning indicating that persisting the secret is less secure than providing it at startup. +We do make some provisions in order to protect the persisted password. + +It is encrypted with AES 128 bit encryption and where possible the file permissions are set to only be accessible by the user that the gateway is running as. + +After persisting the secret, ensure that the file at data/security/master has the appropriate permissions set for your environment. +This is probably the most important layer of defense for master secret. +Do not assume that the encryption is sufficient protection. + +A specific user should be created to run the gateway. This user will be the only user with permissions for the persisted master file. + +See the Knox CLI section for descriptions of the command line utilities related to the master secret. + +#### Management of Security Artifacts #### + +There are a number of artifacts that are used by the gateway in ensuring the security of wire level communications, access to protected resources and the encryption of sensitive data. +These artifacts can be managed from outside of the gateway instances or generated and populated by the gateway instance itself. + +The following is a description of how this is coordinated with both standalone (development, demo, etc) gateway instances and instances as part of a cluster of gateways in mind. + +Upon start of the gateway server we: + +1. Look for an identity store at `data/security/keystores/gateway.jks`. + The identity store contains the certificate and private key used to represent the identity of the server for SSL connections and signature creation. + * If there is no identity store we create one and generate a self-signed certificate for use in standalone/demo mode. + The certificate is stored with an alias of gateway-identity. + * If there is an identity store found than we ensure that it can be loaded using the provided master secret and that there is an alias called gateway-identity. +2. Look for a credential store at `data/security/keystores/__gateway-credentials.jceks`. + This credential store is used to store secrets/passwords that are used by the gateway. + For instance, this is where the passphrase for accessing the gateway-identity certificate is kept. + * If there is no credential store found then we create one and populate it with a generated passphrase for the alias `gateway-identity-passphrase`. + This is coordinated with the population of the self-signed cert into the identity-store. + * If a credential store is found then we ensure that it can be loaded using the provided master secret and that the expected aliases have been populated with secrets. + +Upon deployment of a Hadoop cluster topology within the gateway we: + +1. Look for a credential store for the topology. For instance, we have a sample topology that gets deployed out of the box. We look for `data/security/keystores/sandbox-credentials.jceks`. This topology specific credential store is used for storing secrets/passwords that are used for encrypting sensitive data with topology specific keys. + * If no credential store is found for the topology being deployed then one is created for it. + Population of the aliases is delegated to the configured providers within the system that will require the use of a secret for a particular task. + They may programmatically set the value of the secret or choose to have the value for the specified alias generated through the AliasService. + * If a credential store is found then we ensure that it can be loaded with the provided master secret and the configured providers have the opportunity to ensure that the aliases are populated and if not to populate them. + +By leveraging the algorithm described above we can provide a window of opportunity for management of these artifacts in a number of ways. + +1. Using a single gateway instance as a master instance the artifacts can be generated or placed into the expected location and then replicated across all of the slave instances before startup. +2. Using an NFS mount as a central location for the artifacts would provide a single source of truth without the need to replicate them over the network. Of course, NFS mounts have their own challenges. +3. Using the KnoxCLI to create and manage the security artifacts. + +See the Knox CLI section for descriptions of the command line utilities related to the security artifact management. + +#### Keystores #### +In order to provide your own certificate for use by the gateway, you will need to either import an existing key pair into a Java keystore or generate a self-signed cert using the Java keytool. + +##### Importing a key pair into a Java keystore ##### +One way to accomplish this is to start with a PKCS12 store for your key pair and then convert it to a Java keystore or JKS. + +The following example uses openssl to create a PKCS12 encoded store from your provided certificate and private key that are in PEM format. + + openssl pkcs12 -export -in cert.pem -inkey key.pem > server.p12 + +The next example converts the PKCS12 store into a Java keystore (JKS). It should prompt you for the keystore and key passwords for the destination keystore. You must use the master-secret for the keystore password and keep track of the password that you use for the key passphrase. + + keytool -importkeystore -srckeystore server.p12 -destkeystore gateway.jks -srcstoretype pkcs12 + +While using this approach a couple of important things to be aware of: + +1. the alias MUST be "gateway-identity". You may need to change it using keytool after the import of the PKCS12 store. You can use keytool to do this - for example: + + keytool -changealias -alias "1" -destalias "gateway-identity" -keystore gateway.jks -storepass {knoxpw} + +2. the name of the expected identity keystore for the gateway MUST be gateway.jks +3. the passwords for the keystore and the imported key may both be set to the master secret for the gateway install. You can change the key passphrase after import using keytool as well. You may need to do this in order to provision the password in the credential store as described later in this section. For example: + + keytool -keypasswd -alias gateway-identity -keystore gateway.jks + +NOTE: The password for the keystore as well as that of the imported key may be the master secret for the gateway instance or you may set the gateway-identity-passphrase alias using the Knox CLI to the actual key passphrase. See the Knox CLI section for details. + +The following will allow you to provision the passphrase for the private key that you set during keystore creation above - it will prompt you for the actual passphrase. + + bin/knoxcli.sh create-alias gateway-identity-passphrase + +##### Generating a self-signed cert for use in testing or development environments ##### + + keytool -genkey -keyalg RSA -alias gateway-identity -keystore gateway.jks \ + -storepass {master-secret} -validity 360 -keysize 2048 + +Keytool will prompt you for a number of elements used will comprise the distinguished name (DN) within your certificate. + +*NOTE:* When it prompts you for your First and Last name be sure to type in the hostname of the machine that your gateway instance will be running on. This is used by clients during hostname verification to ensure that the presented certificate matches the hostname that was used in the URL for the connection - so they need to match. + +*NOTE:* When it prompts for the key password just press enter to ensure that it is the same as the keystore password. Which, as was described earlier, must match the master secret for the gateway instance. Alternatively, you can set it to another passphrase - take note of it and set the gateway-identity-passphrase alias to that passphrase using the Knox CLI. + +See the Knox CLI section for descriptions of the command line utilities related to the management of the keystores. + +##### Using a CA Signed Key Pair ##### +For certain deployments a certificate key pair that is signed by a trusted certificate authority is required. There are a number of different ways in which these certificates are acquired and can be converted and imported into the Apache Knox keystore. + +The following steps have been used to do this and are provided here for guidance in your installation. +You may have to adjust according to your environment. + +General steps: + +1. Stop Knox gateway and back up all files in `{GATEWAY_HOME}/data/security/keystores` + + gateway.sh stop + +2. Create a new master key for Knox and persist it. The master key will be referred to in following steps as `$master-key` + + knoxcli.sh create-master -force + +3. Create identity keystore gateway.jks. cert in alias gateway-identity + + cd {GATEWAY_HOME}/data/security/keystore + keytool -genkeypair -alias gateway-identity -keyalg RSA -keysize 1024 -dname "CN=$fqdn_knox,OU=hdp,O=sdge" -keypass $keypass -keystore gateway.jks -storepass $master-key -validity 300 + + NOTE: `$fqdn_knox` is the hostname of the Knox host. Some may choose `$keypass` to be the same as `$master-key`. + +4. Create credential store to store the `$keypass` in step 3. This creates `__gateway-credentials.jceks` file + + knoxcli.sh create-alias gateway-identity-passphrase --value $keypass + +5. Generate a certificate signing request from the gateway.jks + + keytool -keystore gateway.jks -storepass $master-key -alias gateway-identity -certreq -file knox.csr + +6. Send the `knox.csr` file to the CA authority and get back the signed certificate (`knox.signed`). You also need the CA certificate, which normally can be requested through an openssl command or web browser or from the CA. + +7. Import both the CA authority certificate (referred as `corporateCA.cer`) and the signed Knox certificate back into `gateway.jks` + + keytool -keystore gateway.jks -storepass $master-key -alias $hwhq -import -file corporateCA.cer + keytool -keystore gateway.jks -storepass $master-key -alias gateway-identity -import -file knox.signed + + NOTE: Use any alias appropriate for the corporate CA. + +8. Restart Knox gateway. Check `gateway.log` to check whether the gateway started properly and clusters are deployed. You can check the timestamp on cluster deployment files + + ls -alrt {GATEWAY_HOME}/data/deployment + +9. Verify that clients can use the CA authority cert to access Knox (which is the goal of using public signed cert) using curl or a web browsers which has the CA certificate installed + + curl --cacert supwin12ad.cer -u hdptester:hadoop -X GET 'https://$fqdn_knox:8443/gateway/$topologyname/webhdfs/v1/tmp?op=LISTSTATUS' + +##### Credential Store ##### +Whenever you provide your own keystore with either a self-signed cert or an issued certificate signed by a trusted authority, you will need to set an alias for the gateway-identity-passphrase or create an empty credential store. This is necessary for the current release in order for the system to determine the correct password for the keystore and the key. + +The credential stores in Knox use the JCEKS keystore type as it allows for the storage of general secrets in addition to certificates. + +Keytool may be used to create credential stores but the Knox CLI section details how to create aliases. These aliases are managed within credential stores which are created by the CLI as needed. The simplest approach is to create the gateway-identity-passpharse alias with the Knox CLI. This will create the credential store if it doesn't already exist and add the key passphrase. + +See the Knox CLI section for descriptions of the command line utilities related to the management of the credential stores. + +##### Provisioning of Keystores ##### +Once you have created these keystores you must move them into place for the gateway to discover them and use them to represent its identity for SSL connections. This is done by copying the keystores to the `{GATEWAY_HOME}/data/security/keystores` directory for your gateway install. + +#### Summary of Secrets to be Managed #### + +1. Master secret - the same for all gateway instances in a cluster of gateways +2. All security related artifacts are protected with the master secret +3. Secrets used by the gateway itself are stored within the gateway credential store and are the same across all gateway instances in the cluster of gateways +4. Secrets used by providers within cluster topologies are stored in topology specific credential stores and are the same for the same topology across the cluster of gateway instances. + However, they are specific to the topology - so secrets for one hadoop cluster are different from those of another. + This allows for fail-over from one gateway instance to another even when encryption is being used while not allowing the compromise of one encryption key to expose the data for all clusters. + +NOTE: the SSL certificate will need special consideration depending on the type of certificate. Wildcard certs may be able to be shared across all gateway instances in a cluster. +When certs are dedicated to specific machines the gateway identity store will not be able to be blindly replicated as host name verification problems will ensue. +Obviously, trust-stores will need to be taken into account as well. Added: knox/trunk/books/0.14.0/config_advanced_ldap.md URL: http://svn.apache.org/viewvc/knox/trunk/books/0.14.0/config_advanced_ldap.md?rev=1809165&view=auto ============================================================================== --- knox/trunk/books/0.14.0/config_advanced_ldap.md (added) +++ knox/trunk/books/0.14.0/config_advanced_ldap.md Thu Sep 21 13:57:57 2017 @@ -0,0 +1,281 @@ +### Advanced LDAP Authentication + +The default configuration computes the bind DN for incoming user based on userDnTemplate. +This does not work in enterprises where users could belong to multiple branches of LDAP tree. +You could instead enable advanced configuration that would compute bind DN of incoming user with an LDAP search. + +#### Problem with userDnTemplate based Authentication + +UserDnTemplate based authentication uses configuration parameter ldapRealm.userDnTemplate. +Typical value of userDNTemplate would look like `uid={0},ou=people,dc=hadoop,dc=apache,dc=org`. + +To compute bind DN of the client, we swap the place holder {0} with login id provided by the client. +For example, if the login id provided by the client is "guest', +the computed bind DN would be `uid=guest,ou=people,dc=hadoop,dc=apache,dc=org`. + +This keeps configuration simple. + +However, this does not work if users belong to different branches of LDAP DIT. +For example, if there are some users under `ou=people,dc=hadoop,dc=apache,dc=org` +and some users under `ou=contractors,dc=hadoop,dc=apache,dc=org`, +we can not come up with userDnTemplate that would work for all the users. + +#### Using advanced LDAP Authentication + +With advanced LDAP authentication, we find the bind DN of the user by searching LDAP directory +instead of interpolating bind DN from userDNTemplate. + + +#### Example search filter to find the client bind DN + +Assuming + +* ldapRealm.userSearchAttributeName=uid +* ldapRealm.userObjectClass=person +* client specified login id = "guest" + +LDAP Filter for doing a search to find the bind DN would be + + (&(uid=guest)(objectclass=person)) + +This could find bind DN to be + + uid=guest,ou=people,dc=hadoop,dc=apache,dc=org + +Please note that the userSearchAttributeName need not be part of bindDN. + +For example, you could use + +* ldapRealm.userSearchAttributeName=email +* ldapRealm.userObjectClass=person +* client specified login id = "bill.clin...@gmail.com" + +LDAP Filter for doing a search to find the bind DN would be + + (&(email=bill.clin...@gmail.com)(objectclass=person)) + +This could find bind DN to be + + uid=billc,ou=contractors,dc=hadoop,dc=apache,dc=org + +#### Advanced LDAP configuration parameters +The table below provides a brief description and sample of the available advanced bind and search configuration parameters. + +| Parameter | Description | Default | Sample | +|-----------------------------|----------------------------------------------------------------|---------|--------------------------------------------------------------------| +| principalRegex | Parses the principal for insertion into templates via regex. | (.*) | (.\*?)\\\\(.\*) _(e.g. match US\tom: {0}=US\tom, {1}=US, {2}=tom)_ | +| userDnTemplate | Direct user bind DN template. | {0} | cn={2},dc={1},dc=qa,dc=company,dc=com | +| userSearchBase | Search based template. Used with config below. | none | dc={1},dc=qa,dc=company,dc=com | +| userSearchAttributeName | Attribute name for simplified search filter. | none | sAMAccountName | +| userSearchAttributeTemplate | Attribute template for simplified search filter. | {0} | {2} | +| userSearchFilter | Advanced search filter template. Note \& is \& in XML. | none | (\&(objectclass=person)(sAMAccountName={2})) | +| userSearchScope | Search scope: subtree, onelevel, object. | subtree | onelevel | + +#### Advanced LDAP configuration combinations +There are also only certain valid combinations of advanced LDAP configuration parameters. + +* User DN Template + * userDnTemplate (Required) + * principalRegex (Optional) +* User Search by Attribute + * userSearchBase (Required) + * userAttributeName (Required) + * userAttributeTemplate (Optional) + * userSearchScope (Optional) + * principalRegex (Optional) +* User Search by Filter + * userSearchBase (Required) + * userSearchFilter (Required) + * userSearchScope (Optional) + * principalRegex (Optional) + +#### Advanced LDAP configuration precedence +The presence of multiple configuration combinations should be avoided. +The rules below clarify which combinations take precedence when present. + +1. userSearchBase takes precedence over userDnTemplate +2. userSearchFilter takes precedence over userSearchAttributeName + +#### Example provider configuration to use advanced LDAP authentication + +The example configuration appears verbose due to the presence of liberal comments +and illustration of optional parameters and default values. +The configuration that you would use could be much shorter if you rely on default values. + + <provider> + + <role>authentication</role> + <name>ShiroProvider</name> + <enabled>true</enabled> + + <param> + <name>main.ldapRealm</name> + <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value> + </param> + + <param> + <name>main.ldapContextFactory</name> + <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value> + </param> + + <param> + <name>main.ldapRealm.contextFactory</name> + <value>$ldapContextFactory</value> + </param> + + <!-- update the value based on your ldap directory protocol, host and port --> + <param> + <name>main.ldapRealm.contextFactory.url</name> + <value>ldap://hdp.example.com:389</value> + </param> + + <!-- optional, default value: simple + Update the value based on mechanisms supported by your ldap directory --> + <param> + <name>main.ldapRealm.contextFactory.authenticationMechanism</name> + <value>simple</value> + </param> + + <!-- optional, default value: {0} + update the value based on your ldap DIT(directory information tree). + ignored if value is defined for main.ldapRealm.userSearchAttributeName --> + <param> + <name>main.ldapRealm.userDnTemplate</name> + <value>uid={0},ou=people,dc=hadoop,dc=apache,dc=org</value> + </param> + + <!-- optional, default value: null + If you specify a value for this attribute, useDnTemplate + specified above would be ignored and user bind DN would be computed using + ldap search + update the value based on your ldap DIT(directory information layout) + value of search attribute should identity the user uniquely --> + <param> + <name>main.ldapRealm.userSearchAttributeName</name> + <value>uid</value> + </param> + + <!-- optional, default value: false + If the value is true, groups in which user is a member are looked up + from LDAP and made available for service level authorization checks --> + <param> + <name>main.ldapRealm.authorizationEnabled</name> + <value>true</value> + </param> + + <!-- bind DN used to search for groups and user bind DN. + Required if a value is defined for main.ldapRealm.userSearchAttributeName + or if the value of main.ldapRealm.authorizationEnabled is true --> + <param> + <name>main.ldapRealm.contextFactory.systemUsername</name> + <value>uid=guest,ou=people,dc=hadoop,dc=apache,dc=org</value> + </param> + + <!-- password for systemUserName. + Required if a value is defined for main.ldapRealm.userSearchAttributeName + or if the value of main.ldapRealm.authorizationEnabled is true --> + <param> + <name>main.ldapRealm.contextFactory.systemPassword</name> + <value>${ALIAS=ldcSystemPassword}</value> + </param> + + <!-- optional, default value: simple + Update the value based on mechanisms supported by your ldap directory --> + <param> + <name>main.ldapRealm.contextFactory.systemAuthenticationMechanism</name> + <value>simple</value> + </param> + + <!-- optional, default value: person + Objectclass to identify user entries in ldap, used to build search + filter to search for user bind DN --> + <param> + <name>main.ldapRealm.userObjectClass</name> + <value>person</value> + </param> + + <!-- search base used to search for user bind DN and groups --> + <param> + <name>main.ldapRealm.searchBase</name> + <value>dc=hadoop,dc=apache,dc=org</value> + </param> + + <!-- search base used to search for user bind DN. + Defaults to the value of main.ldapRealm.searchBase. + If main.ldapRealm.userSearchAttributeName is defined, + value for main.ldapRealm.searchBase or main.ldapRealm.userSearchBase + should be defined --> + <param> + <name>main.ldapRealm.userSearchBase</name> + <value>dc=hadoop,dc=apache,dc=org</value> + </param> + + <!-- search base used to search for groups. + Defaults to the value of main.ldapRealm.searchBase. + If value of main.ldapRealm.authorizationEnabled is true, + value for main.ldapRealm.searchBase or main.ldapRealm.groupSearchBase should be defined --> + <param> + <name>main.ldapRealm.groupSearchBase</name> + <value>dc=hadoop,dc=apache,dc=org</value> + </param> + + <!-- optional, default value: groupOfNames + Objectclass to identify group entries in ldap, used to build search + filter to search for group entries --> + <param> + <name>main.ldapRealm.groupObjectClass</name> + <value>groupOfNames</value> + </param> + + <!-- optional, default value: member + If value is memberUrl, we treat found groups as dynamic groups --> + <param> + <name>main.ldapRealm.memberAttribute</name> + <value>member</value> + </param> + + <!-- optional, default value: uid={0} + Ignored if value is defined for main.ldapRealm.userSearchAttributeName --> + <param> + <name>main.ldapRealm.memberAttributeValueTemplate</name> + <value>uid={0},ou=people,dc=hadoop,dc=apache,dc=org</value> + </param> + + <!-- optional, default value: cn --> + <param> + <name>main.ldapRealm.groupIdAttribute</name> + <value>cn</value> + </param> + + <param> + <name>urls./**</name> + <value>authcBasic</value> + </param> + + <!-- optional, default value: 30min --> + <param> + <name>sessionTimeout</name> + <value>30</value> + </param> + + </provider> + +#### Special note on parameter main.ldapRealm.contextFactory.systemPassword + +The value for this could have one of the following 2 formats + +* plaintextpassword +* ${ALIAS=ldcSystemPassword} + +The first format specifies the password in plain text in the provider configuration. +Use of this format should be limited for testing and troubleshooting. + +We strongly recommend using the second format `${ALIAS=ldcSystemPassword}` in production. +This format uses an alias for the password stored in credential store. +In the example `${ALIAS=ldcSystemPassword}`, +ldcSystemPassword is the alias for the password stored in credential store. + +Assuming the plain text password is "hadoop", and your topology file name is "hdp.xml", +you would use following command to create the right password alias in credential store. + + {GATEWAY_HOME}/bin/knoxcli.sh create-alias ldcSystemPassword --cluster hdp --value hadoop Added: knox/trunk/books/0.14.0/config_audit.md URL: http://svn.apache.org/viewvc/knox/trunk/books/0.14.0/config_audit.md?rev=1809165&view=auto ============================================================================== --- knox/trunk/books/0.14.0/config_audit.md (added) +++ knox/trunk/books/0.14.0/config_audit.md Thu Sep 21 13:57:57 2017 @@ -0,0 +1,72 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +---> + +### Audit ### + +The Audit facility within the Knox Gateway introduces functionality for tracking actions that are executed by Knox per user's request or that are produced by Knox internal events like topology deploy, etc. +The Knox Audit module is based on [Apache log4j](http://logging.apache.org/log4j/1.2/). + +#### Configuration needed #### + +Out of the box, the Knox Gateway includes preconfigured auditing capabilities. To change its configuration please read the following sections. + +#### Where audit logs go #### + +The Audit module is preconfigured to write audit records to the log file `{GATEWAY_HOME}/log/gateway-audit.log`. + +This behavior can be changed in the `{GATEWAY_HOME}/conf/gateway-log4j.properties` file. `app.audit.file` can be used to change the location. The `log4j.appender.auditfile.*` properties can be used for further customization. For detailed information read the [Apache log4j](http://logging.apache.org/log4j/1.2/) documentation. + +#### Audit format #### + +Out of the box, the audit record format is defined by `org.apache.hadoop.gateway.audit.log4j.layout.AuditLayout`. +Its structure is as follows: + + EVENT_PUBLISHING_TIME ROOT_REQUEST_ID|PARENT_REQUEST_ID|REQUEST_ID|LOGGER_NAME|TARGET_SERVICE_NAME|USER_NAME|PROXY_USER_NAME|SYSTEM_USER_NAME|ACTION|RESOURCE_TYPE|RESOURCE_NAME|OUTCOME|LOGGING_MESSAGE + +The audit record format can be changed by setting `log4j.appender.auditfile.layout` property in `{GATEWAY_HOME}/conf/gateway-log4j.properties` to another class that extends `org.apache.log4j.Layout` or its subclasses. + +For detailed information read [Apache log4j](http://logging.apache.org/log4j/1.2/). + +##### How to interpret audit log ##### + +Component | Description +---------|----------- +EVENT_PUBLISHING_TIME|Time when audit record was published. +ROOT_REQUEST_ID|The root request ID if this is a sub-request. Currently it is empty. +PARENT_REQUEST_ID|The parent request ID if this is a sub-request. Currently it is empty. +REQUEST_ID|A unique value representing the current, active request. If the current request id value is different from the current parent request id value then the current request id value is moved to the parent request id before it is replaced by the provided request id. If the root request id is not set it will be set with the first non-null value of either the parent request id or the passed request id. +LOGGER_NAME|The name of the logger +TARGET_SERVICE_NAME|Name of Hadoop service. Can be empty if audit record is not linked to any Hadoop service, for example, audit record for topology deployment. +USER_NAME|Name of user that initiated session with Knox +PROXY_USER_NAME|Mapped user name. For detailed information read #[Identity Assertion]. +SYSTEM_USER_NAME|Currently is empty. +ACTION|Type of action that was executed. Following actions are defined: authentication, authorization, redeploy, deploy, undeploy, identity-mapping, dispatch, access. +RESOURCE_TYPE|Type of resource for which action was executed. Following resource types are defined: uri, topology, principal. +RESOURCE_NAME|Name of resource. For resource of type topology it is name of topology. For resource of type uri it is inbound or dispatch request path. For resource of type principal it is a name of mapped user. +OUTCOME|Action result type. Following outcomes are defined: success, failure, unavailable. +LOGGING_MESSAGE| Logging message. Contains additional tracking information. + +#### Audit log rotation #### + +Audit logging is preconfigured with `org.apache.log4j.DailyRollingFileAppender`. +[Apache log4j](http://logging.apache.org/log4j/1.2/) contains information about other Appenders. + +#### How to change the audit level or disable it #### + +All audit messages are logged at `INFO` level and this behavior can't be changed. + +Disabling auditing can be done by decreasing the log level for the Audit appender or setting it to `OFF`.
