I suggest that you take a look at:
* the {GATEWAY_HOME}/samples directory
* WebHDFS with ClientDSL section in
http://knox.apache.org/books/knox-0-6-0/user-guide.html#WebHDFS
* Tests for WebHDFSThe samples directory contains groovy script examples for doing the sorts of things that you are talking about - look at: ExampleWebHdfsPutGet.groovy in particular. On Tue, Jun 2, 2015 at 10:23 AM, Phani Yadavilli -X (pyadavil) < [email protected]> wrote: > Hi Larry, > > > > I have a simple java api based program which tried to read a file(HDFS). > When I run the jar file it can either go via the knox gateway or I should > be able to include the gateway information within my java code so that when > the program run it would be taken care. I wanted to know if there are any > such API’s or examples for the use case. > > > > Also, let me know if there is any other way I can handle this. > > > > Regards > > Phani Kumar > > > > *From:* larry mccay [mailto:[email protected]] > *Sent:* Tuesday, June 02, 2015 5:00 PM > > *To:* [email protected] > *Cc:* [email protected] > *Subject:* Re: Unable to start the gateway > > > > Glad to hear that you are up and running now. > > > > Depending on what sort of configuration you are talking about, you can do > two separate things: > > > > 1. You can shell out to the knoxcli.sh script for provisioning the master > secret, more specific certs, etc > > 2. Topology related administration is available via REST API that you can > read about at: > http://knox.apache.org/books/knox-0-6-0/user-guide.html#Admin+API > > > > Hope that is useful. > > > > --larry > > > > On Tue, Jun 2, 2015 at 5:19 AM, Phani Yadavilli -X (pyadavil) < > [email protected]> wrote: > > Btw, do we have any JAVA API to configure knox from the code level. > > > > Regards > > Phani Kumar > > > > *From:* Phani Yadavilli -X (pyadavil) > *Sent:* Tuesday, June 02, 2015 2:47 PM > > > *To:* [email protected] > *Cc:* [email protected] > *Subject:* RE: Unable to start the gateway > > > > Hi Larry, > > > > The issue is resolved. > > > > Thanks for the help. > > > > Regards > > Phani Kumar > > > > *From:* larry mccay [mailto:[email protected] <[email protected]>] > *Sent:* Monday, June 01, 2015 6:26 PM > *To:* Phani Yadavilli -X (pyadavil) > *Cc:* [email protected]; [email protected] > *Subject:* Re: Unable to start the gateway > > > > That usually means that it is already running - or you have something else > running on port 33389. > > > > Try "ps -ef|grep ldap" and you should see that it is already running. > > > > On Mon, Jun 1, 2015 at 7:52 AM, Phani Yadavilli -X (pyadavil) < > [email protected]> wrote: > > Hi Larry, > > > > I tried the below steps. This time the gateway started but the ladp is not > starting. Please find the error logs as below > > > > [knox@cldb-node-01 knox-0.6.0]$ bin/ldap.sh start > > Starting LDAP failed. > > > > [knox@cldb-node-01 logs]$ cat ldap.err > > Exception in thread "main" java.lang.reflect.InvocationTargetException > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > > at > org.apache.hadoop.gateway.launcher.Invoker.invokeMainMethod(Invoker.java:70) > > at > org.apache.hadoop.gateway.launcher.Invoker.invoke(Invoker.java:39) > > at org.apache.hadoop.gateway.launcher.Command.run(Command.java:101) > > at > org.apache.hadoop.gateway.launcher.Launcher.run(Launcher.java:69) > > at > org.apache.hadoop.gateway.launcher.Launcher.main(Launcher.java:46) > > Caused by: java.net.BindException: Address already in use > > at java.net.PlainSocketImpl.socketBind(Native Method) > > at > java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376) > > at java.net.ServerSocket.bind(ServerSocket.java:376) > > at java.net.ServerSocket.<init>(ServerSocket.java:237) > > at java.net.ServerSocket.<init>(ServerSocket.java:128) > > at > org.apache.hadoop.gateway.security.ldap.SimpleLdapDirectoryServer.main(SimpleLdapDirectoryServer.java:102) > > ... 9 more > > Exception in thread "main" java.lang.reflect.InvocationTargetException > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > > at > org.apache.hadoop.gateway.launcher.Invoker.invokeMainMethod(Invoker.java:70) > > at > org.apache.hadoop.gateway.launcher.Invoker.invoke(Invoker.java:39) > > at org.apache.hadoop.gateway.launcher.Command.run(Command.java:101) > > at > org.apache.hadoop.gateway.launcher.Launcher.run(Launcher.java:69) > > at > org.apache.hadoop.gateway.launcher.Launcher.main(Launcher.java:46) > > Caused by: java.net.BindException: Address already in use > > at java.net.PlainSocketImpl.socketBind(Native Method) > > at > java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376) > > at java.net.ServerSocket.bind(ServerSocket.java:376) > > at java.net.ServerSocket.<init>(ServerSocket.java:237) > > at java.net.ServerSocket.<init>(ServerSocket.java:128) > > at > org.apache.hadoop.gateway.security.ldap.SimpleLdapDirectoryServer.main(SimpleLdapDirectoryServer.java:102) > > ... 9 more > > Exception in thread "main" java.lang.reflect.InvocationTargetException > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > > at > org.apache.hadoop.gateway.launcher.Invoker.invokeMainMethod(Invoker.java:70) > > at > org.apache.hadoop.gateway.launcher.Invoker.invoke(Invoker.java:39) > > at org.apache.hadoop.gateway.launcher.Command.run(Command.java:101) > > at > org.apache.hadoop.gateway.launcher.Launcher.run(Launcher.java:69) > > at > org.apache.hadoop.gateway.launcher.Launcher.main(Launcher.java:46) > > Caused by: java.net.BindException: Address already in use > > at java.net.PlainSocketImpl.socketBind(Native Method) > > at > java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376) > > at java.net.ServerSocket.bind(ServerSocket.java:376) > > at java.net.ServerSocket.<init>(ServerSocket.java:237) > > at java.net.ServerSocket.<init>(ServerSocket.java:128) > > at > org.apache.hadoop.gateway.security.ldap.SimpleLdapDirectoryServer.main(SimpleLdapDirectoryServer.java:102) > > ... 9 more > > Exception in thread "main" java.lang.reflect.InvocationTargetException > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > > at > org.apache.hadoop.gateway.launcher.Invoker.invokeMainMethod(Invoker.java:70) > > at > org.apache.hadoop.gateway.launcher.Invoker.invoke(Invoker.java:39) > > at org.apache.hadoop.gateway.launcher.Command.run(Command.java:101) > > at > org.apache.hadoop.gateway.launcher.Launcher.run(Launcher.java:69) > > at > org.apache.hadoop.gateway.launcher.Launcher.main(Launcher.java:46) > > Caused by: java.net.BindException: Address already in use > > at java.net.PlainSocketImpl.socketBind(Native Method) > > at > java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376) > > at java.net.ServerSocket.bind(ServerSocket.java:376) > > at java.net.ServerSocket.<init>(ServerSocket.java:237) > > at java.net.ServerSocket.<init>(ServerSocket.java:128) > > at > org.apache.hadoop.gateway.security.ldap.SimpleLdapDirectoryServer.main(SimpleLdapDirectoryServer.java:102) > > ... 9 more > > > > Regards > > Phani Kumar > > > > *From:* Phani Yadavilli -X (pyadavil) > *Sent:* Monday, June 01, 2015 5:12 PM > *To:* [email protected] > *Cc:* [email protected] > *Subject:* RE: Unable to start the gateway > > > > Thanks larry. I will try this. > > > > Regards > > Phani Kumar > > > > *From:* larry mccay [mailto:[email protected] <[email protected]>] > *Sent:* Monday, June 01, 2015 5:10 PM > *To:* [email protected] > *Cc:* [email protected] > *Subject:* Re: Unable to start the gateway > > > > Phani - > > > > I have seen issues like this when the keystore was created with one JDK - > like IBM - and the runtime tries to start with openjdk or oracle JDK. > > Check that those versions are the same. > > > > If you haven't provisioned your own certificate then you may want to try > removing the keystores in {GATEWAY_HOME}/data/security/keystores and > allowing them to be recreated. > > > > If the master secret that you provided doesn't match the one used to > create them when you first started it then this would happen as well. > > Recreating them with the new master secret will fix that - if it is the > case. > > > > HTH. > > > > --larry > > > > > > On Mon, Jun 1, 2015 at 2:32 AM, Phani Yadavilli -X (pyadavil) < > [email protected]> wrote: > > Thank you larry for the info. > > > > Vinay, > > > > I am still not able to start the gateway. I used the below process to > change the password. Am I still missing something here. > > > > [knox@cldb-node-01 bin]$ ./knoxcli.sh create-master --force > > > *************************************************************************************************** > > You have indicated that you would like to persist the master secret for > this service instance. > > Be aware that this is less secure than manually entering the secret on > startup. > > The persisted file will be encrypted and primarily protected through OS > permissions. > > > *************************************************************************************************** > > Enter master secret: > > Enter master secret again: > > Master secret has been persisted to disk. > > [knox@cldb-node-01 bin]$ ./gateway.sh start > > Starting Gateway failed. > > > > Regards > > Phani Kumar > > > > *From:* larry mccay [mailto:[email protected]] > *Sent:* Friday, May 29, 2015 10:27 PM > > > *To:* [email protected] > *Subject:* Re: Unable to start the gateway > > > > Hi Phani - > > > > You should find that Knox WEBHDFS support is compatible with any > implementation of the HDFS REST API. > > Things like HttpFS should work in place of WebHDFS if your distro is > missing explicit support for it. > > > > Ranger service level authorization would still work with Knox in this case > but may not have lower level authorization enforcement that requires NN - > again depending on your distro's support. > > > > Hope this is helpful to you. > > > > --larry > > > > On Fri, May 29, 2015 at 11:22 AM, Phani Yadavilli -X (pyadavil) < > [email protected]> wrote: > > Hi Vinay, > > > > Thank you. I will try that. I have another question. > > We are using the MapR cluster where we do not have NameNode. Instead we > have CLDB. Is there any workaround I can do to make the ranger and knox > work with the cluster. > > > > Regards > > Phani Kumar > > > > *From:* Vinay Shukla [mailto:[email protected]] > *Sent:* Friday, May 29, 2015 8:29 PM > > > *To:* [email protected] > *Subject:* Re: Unable to start the gateway > > > > You missed the below step needed once before starting gateway for the > first time. > > > > bin/knoxcli.sh create-master > > > > Thx, > > Vinay > > > > On Fri, May 29, 2015 at 7:13 AM, Phani Yadavilli -X (pyadavil) < > [email protected]> wrote: > > Hi Kevin, > > > > Please find the log attached. > > > > Regards > > Phani Kumar > > > > *From:* Kevin Minder [mailto:[email protected]] > *Sent:* Friday, May 29, 2015 6:30 PM > *To:* [email protected] > *Subject:* Re: Unable to start the gateway > > > > Hi Phani, > > Could you take a look at the log file (logs/gateway.log) and show us the > content of that. With that we should be able to help. > > Kevin. > > > > *From: *"Phani Yadavilli -X (pyadavil)" <[email protected]> > *Reply-To: *"[email protected]" <[email protected]> > *Date: *Friday, May 29, 2015 at 2:54 AM > *To: *"[email protected]" <[email protected]> > *Subject: *Unable to start the gateway > > > > Hi Team, > > > > I am a newbie to the knox. I am trying to setup knox in our environment as > a part of POC. When I am trying to run the > > > > ./gateway.sh start > > Starting Gateway failed. > > > > I am unable to start the gateway. Could you please help me understand and > debug the issue. > > > > Regards > > Phani Kumar > > > > > > > > > > >
