Hi Larry, The gateway seems to be working now. Can you please help me with the below message.
[mapr@cldb-node-01 ~]$ curl -v "https://localhost:8444/gateway/single-node/WEBHDFS/v1?op=gethomedirectory&user.name=mapr"; * About to connect() to localhost port 8444 (#0) * Trying 127.0.0.1... connected * Connected to localhost (127.0.0.1) port 8444 (#0) * Initializing NSS with certpath: sql:/etc/pki/nssdb * CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none * Issuer certificate is invalid: 'CN=localhost,OU=Test,O=Hadoop,L=Test,ST=Test,C=US' * NSS error -8156 * Closing connection #0 * Peer certificate cannot be authenticated with known CA certificates curl: (60) Peer certificate cannot be authenticated with known CA certificates More details here: http://curl.haxx.se/docs/sslcerts.html curl performs SSL certificate verification by default, using a "bundle" of Certificate Authority (CA) public keys (CA certs). If the default bundle file isn't adequate, you can specify an alternate file using the --cacert option. If this HTTPS server uses a certificate signed by a CA represented in the bundle, the certificate verification probably failed due to a problem with the certificate (it might be expired, or the name might not match the domain name in the URL). If you'd like to turn off curl's verification of the certificate, use the -k (or --insecure) option. Regards Phani Kumar From: Phani Yadavilli -X (pyadavil) Sent: Tuesday, June 02, 2015 9:17 PM To: [email protected] Subject: RE: Unable to start the gateway Thank you,Larry. I will try these. Regards Phani Kumar From: larry mccay [mailto:[email protected]] Sent: Tuesday, June 02, 2015 8:05 PM To: [email protected]<mailto:[email protected]> Subject: Re: Unable to start the gateway I suggest that you take a look at: * the {GATEWAY_HOME}/samples directory * WebHDFS with ClientDSL section in http://knox.apache.org/books/knox-0-6-0/user-guide.html#WebHDFS * Tests for WebHDFS The samples directory contains groovy script examples for doing the sorts of things that you are talking about - look at: ExampleWebHdfsPutGet.groovy in particular. On Tue, Jun 2, 2015 at 10:23 AM, Phani Yadavilli -X (pyadavil) <[email protected]<mailto:[email protected]>> wrote: Hi Larry, I have a simple java api based program which tried to read a file(HDFS). When I run the jar file it can either go via the knox gateway or I should be able to include the gateway information within my java code so that when the program run it would be taken care. I wanted to know if there are any such API’s or examples for the use case. Also, let me know if there is any other way I can handle this. Regards Phani Kumar From: larry mccay [mailto:[email protected]<mailto:[email protected]>] Sent: Tuesday, June 02, 2015 5:00 PM To: [email protected]<mailto:[email protected]> Cc: [email protected]<mailto:[email protected]> Subject: Re: Unable to start the gateway Glad to hear that you are up and running now. Depending on what sort of configuration you are talking about, you can do two separate things: 1. You can shell out to the knoxcli.sh script for provisioning the master secret, more specific certs, etc 2. Topology related administration is available via REST API that you can read about at: http://knox.apache.org/books/knox-0-6-0/user-guide.html#Admin+API Hope that is useful. --larry On Tue, Jun 2, 2015 at 5:19 AM, Phani Yadavilli -X (pyadavil) <[email protected]<mailto:[email protected]>> wrote: Btw, do we have any JAVA API to configure knox from the code level. Regards Phani Kumar From: Phani Yadavilli -X (pyadavil) Sent: Tuesday, June 02, 2015 2:47 PM To: [email protected]<mailto:[email protected]> Cc: [email protected]<mailto:[email protected]> Subject: RE: Unable to start the gateway Hi Larry, The issue is resolved. Thanks for the help. Regards Phani Kumar From: larry mccay [mailto:[email protected]] Sent: Monday, June 01, 2015 6:26 PM To: Phani Yadavilli -X (pyadavil) Cc: [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]> Subject: Re: Unable to start the gateway That usually means that it is already running - or you have something else running on port 33389. Try "ps -ef|grep ldap" and you should see that it is already running. On Mon, Jun 1, 2015 at 7:52 AM, Phani Yadavilli -X (pyadavil) <[email protected]<mailto:[email protected]>> wrote: Hi Larry, I tried the below steps. This time the gateway started but the ladp is not starting. Please find the error logs as below [knox@cldb-node-01 knox-0.6.0]$ bin/ldap.sh start Starting LDAP failed. [knox@cldb-node-01 logs]$ cat ldap.err Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.gateway.launcher.Invoker.invokeMainMethod(Invoker.java:70) at org.apache.hadoop.gateway.launcher.Invoker.invoke(Invoker.java:39) at org.apache.hadoop.gateway.launcher.Command.run(Command.java:101) at org.apache.hadoop.gateway.launcher.Launcher.run(Launcher.java:69) at org.apache.hadoop.gateway.launcher.Launcher.main(Launcher.java:46) Caused by: java.net.BindException: Address already in use at java.net.PlainSocketImpl.socketBind(Native Method) at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376) at java.net.ServerSocket.bind(ServerSocket.java:376) at java.net.ServerSocket.<init>(ServerSocket.java:237) at java.net.ServerSocket.<init>(ServerSocket.java:128) at org.apache.hadoop.gateway.security.ldap.SimpleLdapDirectoryServer.main(SimpleLdapDirectoryServer.java:102) ... 9 more Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.gateway.launcher.Invoker.invokeMainMethod(Invoker.java:70) at org.apache.hadoop.gateway.launcher.Invoker.invoke(Invoker.java:39) at org.apache.hadoop.gateway.launcher.Command.run(Command.java:101) at org.apache.hadoop.gateway.launcher.Launcher.run(Launcher.java:69) at org.apache.hadoop.gateway.launcher.Launcher.main(Launcher.java:46) Caused by: java.net.BindException: Address already in use at java.net.PlainSocketImpl.socketBind(Native Method) at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376) at java.net.ServerSocket.bind(ServerSocket.java:376) at java.net.ServerSocket.<init>(ServerSocket.java:237) at java.net.ServerSocket.<init>(ServerSocket.java:128) at org.apache.hadoop.gateway.security.ldap.SimpleLdapDirectoryServer.main(SimpleLdapDirectoryServer.java:102) ... 9 more Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.gateway.launcher.Invoker.invokeMainMethod(Invoker.java:70) at org.apache.hadoop.gateway.launcher.Invoker.invoke(Invoker.java:39) at org.apache.hadoop.gateway.launcher.Command.run(Command.java:101) at org.apache.hadoop.gateway.launcher.Launcher.run(Launcher.java:69) at org.apache.hadoop.gateway.launcher.Launcher.main(Launcher.java:46) Caused by: java.net.BindException: Address already in use at java.net.PlainSocketImpl.socketBind(Native Method) at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376) at java.net.ServerSocket.bind(ServerSocket.java:376) at java.net.ServerSocket.<init>(ServerSocket.java:237) at java.net.ServerSocket.<init>(ServerSocket.java:128) at org.apache.hadoop.gateway.security.ldap.SimpleLdapDirectoryServer.main(SimpleLdapDirectoryServer.java:102) ... 9 more Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.gateway.launcher.Invoker.invokeMainMethod(Invoker.java:70) at org.apache.hadoop.gateway.launcher.Invoker.invoke(Invoker.java:39) at org.apache.hadoop.gateway.launcher.Command.run(Command.java:101) at org.apache.hadoop.gateway.launcher.Launcher.run(Launcher.java:69) at org.apache.hadoop.gateway.launcher.Launcher.main(Launcher.java:46) Caused by: java.net.BindException: Address already in use at java.net.PlainSocketImpl.socketBind(Native Method) at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376) at java.net.ServerSocket.bind(ServerSocket.java:376) at java.net.ServerSocket.<init>(ServerSocket.java:237) at java.net.ServerSocket.<init>(ServerSocket.java:128) at org.apache.hadoop.gateway.security.ldap.SimpleLdapDirectoryServer.main(SimpleLdapDirectoryServer.java:102) ... 9 more Regards Phani Kumar From: Phani Yadavilli -X (pyadavil) Sent: Monday, June 01, 2015 5:12 PM To: [email protected]<mailto:[email protected]> Cc: [email protected]<mailto:[email protected]> Subject: RE: Unable to start the gateway Thanks larry. I will try this. Regards Phani Kumar From: larry mccay [mailto:[email protected]] Sent: Monday, June 01, 2015 5:10 PM To: [email protected]<mailto:[email protected]> Cc: [email protected]<mailto:[email protected]> Subject: Re: Unable to start the gateway Phani - I have seen issues like this when the keystore was created with one JDK - like IBM - and the runtime tries to start with openjdk or oracle JDK. Check that those versions are the same. If you haven't provisioned your own certificate then you may want to try removing the keystores in {GATEWAY_HOME}/data/security/keystores and allowing them to be recreated. If the master secret that you provided doesn't match the one used to create them when you first started it then this would happen as well. Recreating them with the new master secret will fix that - if it is the case. HTH. --larry On Mon, Jun 1, 2015 at 2:32 AM, Phani Yadavilli -X (pyadavil) <[email protected]<mailto:[email protected]>> wrote: Thank you larry for the info. Vinay, I am still not able to start the gateway. I used the below process to change the password. Am I still missing something here. [knox@cldb-node-01 bin]$ ./knoxcli.sh create-master --force *************************************************************************************************** You have indicated that you would like to persist the master secret for this service instance. Be aware that this is less secure than manually entering the secret on startup. The persisted file will be encrypted and primarily protected through OS permissions. *************************************************************************************************** Enter master secret: Enter master secret again: Master secret has been persisted to disk. [knox@cldb-node-01 bin]$ ./gateway.sh start Starting Gateway failed. Regards Phani Kumar From: larry mccay [mailto:[email protected]<mailto:[email protected]>] Sent: Friday, May 29, 2015 10:27 PM To: [email protected]<mailto:[email protected]> Subject: Re: Unable to start the gateway Hi Phani - You should find that Knox WEBHDFS support is compatible with any implementation of the HDFS REST API. Things like HttpFS should work in place of WebHDFS if your distro is missing explicit support for it. Ranger service level authorization would still work with Knox in this case but may not have lower level authorization enforcement that requires NN - again depending on your distro's support. Hope this is helpful to you. --larry On Fri, May 29, 2015 at 11:22 AM, Phani Yadavilli -X (pyadavil) <[email protected]<mailto:[email protected]>> wrote: Hi Vinay, Thank you. I will try that. I have another question. We are using the MapR cluster where we do not have NameNode. Instead we have CLDB. Is there any workaround I can do to make the ranger and knox work with the cluster. Regards Phani Kumar From: Vinay Shukla [mailto:[email protected]<mailto:[email protected]>] Sent: Friday, May 29, 2015 8:29 PM To: [email protected]<mailto:[email protected]> Subject: Re: Unable to start the gateway You missed the below step needed once before starting gateway for the first time. bin/knoxcli.sh create-master Thx, Vinay On Fri, May 29, 2015 at 7:13 AM, Phani Yadavilli -X (pyadavil) <[email protected]<mailto:[email protected]>> wrote: Hi Kevin, Please find the log attached. Regards Phani Kumar From: Kevin Minder [mailto:[email protected]<mailto:[email protected]>] Sent: Friday, May 29, 2015 6:30 PM To: [email protected]<mailto:[email protected]> Subject: Re: Unable to start the gateway Hi Phani, Could you take a look at the log file (logs/gateway.log) and show us the content of that. With that we should be able to help. Kevin. From: "Phani Yadavilli -X (pyadavil)" <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Friday, May 29, 2015 at 2:54 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Unable to start the gateway Hi Team, I am a newbie to the knox. I am trying to setup knox in our environment as a part of POC. When I am trying to run the ./gateway.sh start Starting Gateway failed. I am unable to start the gateway. Could you please help me understand and debug the issue. Regards Phani Kumar
