Thank you so much Larry. Thanks for your help. Actually i feel the documentation and logging is very confusing.
On Fri, Jul 3, 2015 at 3:13 AM, larry mccay <[email protected]> wrote: > Oops - hit send too early... > > At that point, you should be good to try the curl command - you have to > make sure that you replace the "sandbox" path element with "cluster1" or > whatever your topology is. > > Hope that gets you going. > > --larry > > On Thu, Jul 2, 2015 at 6:11 PM, larry mccay <[email protected]> wrote: > >> You aren't asking any foolish questions. >> If you don't have proper instructions to follow then you will find >> yourself in a jam. >> >> You have actually already contributed to Knox by finding a bug that we >> will fix in https://issues.apache.org/jira/browse/KNOX-564 - all >> contributions from the community are valuable. >> >> Now.... >> >> Let's step back and ensure that you have a solid foundation for using >> Knox in your environment. >> >> 1. You need a Hadoop cluster install that you can point your instance of >> Knox at - this is available from distro vendors as VMs that you can >> download, etc >> 2. You need to have Knox installed in the VMs or external to the VMs but >> somewhere that it can see the Hadoop cluster in the VMs. >> 3. Since you have had Knox running already - you don't need to create the >> master secret but if it were a brand new install then you would and you >> could follow the instructions in the apache docs for QuickStart >> 4. The sandbox.xml topology is intended to be used with such VMs - you >> can copy it to cluster1 or whatever you like >> 5. The authentication provider in sandbox.xml is the Shiro provider with >> configuration for HTTP Basic authentication against an LDAP server >> - since it is expecting to authenticate against the demo ldap >> server, you will need to start the Demo LDAP Server - see step 6 in the >> QuickStart section of the apache docs. >> >> cd {GATEWAY_HOME}bin/ldap.sh start >> >> >> - since it is expecting HTTP Basic authentication, you need to >> provide username and password using the -u username:password switch in curl >> - see examples in the apache docs for the exact command line. >> >> curl -i -k *-u guest:guest-password* -X GET >> 'https://localhost:8443/gateway/sandbox/webhdfs/v1/?op=LISTSTATUS' >> >> 6. >> >> >> On Thu, Jul 2, 2015 at 5:51 PM, Aneela Saleem <[email protected]> >> wrote: >> >>> I'm very much confused with the documentation and could not getting >>> points related to configuration. >>> I'm so sorry if I am posting foolish questions >>> >>> >>> On Fri, Jul 3, 2015 at 2:49 AM, Aneela Saleem <[email protected]> >>> wrote: >>> >>>> Thanks Larry. >>>> >>>> On Fri, Jul 3, 2015 at 2:41 AM, larry mccay <[email protected]> >>>> wrote: >>>> >>>>> We need a separate jira for the NullPointerException. >>>>> What you have in KNOX-563 should be discussed on the user@ list. >>>>> As I indicated in the jira - you need to supply credentials and make >>>>> sure that the LDAP server is running. >>>>> >>>>> I will close KNOX-563 and we can continue to discuss your problems >>>>> here on the mailing list. >>>>> >>>>> On Thu, Jul 2, 2015 at 4:49 PM, Aneela Saleem <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Kevin, >>>>>> >>>>>> Please follow this URL >>>>>> >>>>>> https://issues.apache.org/jira/browse/KNOX-563 >>>>>> >>>>>> On Fri, Jul 3, 2015 at 1:22 AM, Kevin Minder < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Anella or Larry, >>>>>>> Can one of you please file a jira about the NPE. Knox should >>>>>>> certainly handle that much more gracefully. That would be very much >>>>>>> appreciated. Please attache the topology file (e.g. Sandbox.xml) that >>>>>>> resulted in this error if possible. >>>>>>> Kevin. >>>>>>> >>>>>>> From: larry mccay <[email protected]> >>>>>>> Reply-To: "[email protected]" <[email protected]> >>>>>>> Date: Thursday, July 2, 2015 at 3:37 PM >>>>>>> To: "[email protected]" <[email protected]> >>>>>>> Subject: Re: Knox - Hostname was NOT found >>>>>>> >>>>>>> You are probably getting far enough now to be hitting errors due >>>>>>> to your lack of providers. Start with sandbox.xml config and start the >>>>>>> ldap >>>>>>> server. See the apache docs for details. >>>>>>> On Jul 2, 2015 3:14 PM, "Aneela Saleem" <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hey Larry, >>>>>>>> >>>>>>>> i got the following error >>>>>>>> >>>>>>>> <title>Error 500 Server Error</title> >>>>>>>> </head> >>>>>>>> <body><h2>HTTP ERROR 500</h2> >>>>>>>> <p>Problem accessing /gateway/cluster1/webhdfs/v1/. >>>>>>>> >>>>>>>> And the gateway.log file contains: >>>>>>>> >>>>>>>> 2015-07-03 00:09:04,560 ERROR hadoop.gateway >>>>>>>> (AbstractGatewayFilter.java:doFilter(69)) - Failed to execute filter: >>>>>>>> java.lang.NullPointerException >>>>>>>> 2015-07-03 00:09:04,560 ERROR hadoop.gateway >>>>>>>> (GatewayFilter.java:doFilter(135)) - Gateway processing failed: >>>>>>>> javax.servlet.ServletException: java.lang.NullPointerException >>>>>>>> javax.servlet.ServletException: java.lang.NullPointerException >>>>>>>> at >>>>>>>> org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:70) >>>>>>>> at >>>>>>>> org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:315) >>>>>>>> at >>>>>>>> org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:215) >>>>>>>> at >>>>>>>> org.apache.hadoop.gateway.GatewayFilter.doFilter(GatewayFilter.java:129) >>>>>>>> at >>>>>>>> org.apache.hadoop.gateway.GatewayServlet.service(GatewayServlet.java:121) >>>>>>>> at >>>>>>>> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) >>>>>>>> at >>>>>>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501) >>>>>>>> at >>>>>>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) >>>>>>>> at >>>>>>>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) >>>>>>>> at >>>>>>>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) >>>>>>>> at >>>>>>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) >>>>>>>> at >>>>>>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) >>>>>>>> at >>>>>>>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) >>>>>>>> at >>>>>>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) >>>>>>>> at >>>>>>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) >>>>>>>> at >>>>>>>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) >>>>>>>> at >>>>>>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) >>>>>>>> at org.eclipse.jetty.server.Server.handle(Server.java:370) >>>>>>>> at >>>>>>>> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) >>>>>>>> at >>>>>>>> org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971) >>>>>>>> at >>>>>>>> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033) >>>>>>>> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) >>>>>>>> at >>>>>>>> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) >>>>>>>> at >>>>>>>> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) >>>>>>>> at >>>>>>>> org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196) >>>>>>>> at >>>>>>>> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667) >>>>>>>> at >>>>>>>> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) >>>>>>>> at >>>>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) >>>>>>>> at >>>>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) >>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>> Caused by: java.lang.NullPointerException >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Jul 3, 2015 at 12:07 AM, larry mccay <[email protected] >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> Try lower case webhdfs as in: >>>>>>>>> >>>>>>>>> >>>>>>>>> 'https://localhost:8443/gateway/cluster1/webhdfs/v1/?op=GETHOMEDIRECTORY' >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Jul 2, 2015 at 2:59 PM, Aneela Saleem < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Dear larry, >>>>>>>>>> >>>>>>>>>> The gateway.log file contains the following logs: >>>>>>>>>> >>>>>>>>>> 2015-07-02 23:43:16,391 INFO hadoop.gateway >>>>>>>>>> (GatewayServer.java:handleCreateDeployment(427)) - Deploying topology >>>>>>>>>> cluster1 to >>>>>>>>>> /home/hduser/knox-0.6.0/bin/../data/deployments/cluster1.war.14e501479a0 >>>>>>>>>> 2015-07-02 23:43:16,392 INFO hadoop.gateway >>>>>>>>>> (DeploymentFactory.java:createDeployment(82)) - Configured services >>>>>>>>>> directory is /home/hduser/knox-0.6.0/bin/../data/services >>>>>>>>>> 2015-07-02 23:43:16,428 INFO hadoop.gateway >>>>>>>>>> (DefaultGatewayServices.java:initializeContribution(180)) - >>>>>>>>>> Credential >>>>>>>>>> store found for the cluster: cluster1 - no need to create one. >>>>>>>>>> 2015-07-02 23:44:40,453 WARN hadoop.gateway >>>>>>>>>> (GatewayFilter.java:doFilter(152)) - Failed to match path >>>>>>>>>> /WEBHDFS/v1?op=GETHOMEDIRECTORY >>>>>>>>>> 2015-07-02 23:44:51,722 WARN hadoop.gateway >>>>>>>>>> (GatewayFilter.java:doFilter(152)) - Failed to match path >>>>>>>>>> /WEBHDFS/v1?op=GETHOMEDIRECTORY >>>>>>>>>> 2015-07-02 23:45:30,783 WARN hadoop.gateway >>>>>>>>>> (GatewayFilter.java:doFilter(152)) - Failed to match path >>>>>>>>>> /WEBHDFS/v1?op=GETHOMEDIRECTORY >>>>>>>>>> >>>>>>>>>> And i'm just a beginner. I'm trying to configure Knox Topology >>>>>>>>>> to connect to HDFS, using default services provided. And i'm >>>>>>>>>> validating >>>>>>>>>> connection to services. >>>>>>>>>> >>>>>>>>>> On Thu, Jul 2, 2015 at 11:50 PM, larry mccay < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Please check the {GATEWAY_HOME}/logs/gateway.log file for errors >>>>>>>>>>> during deployment. >>>>>>>>>>> >>>>>>>>>>> I notice that you have no providers described in cluster1.xml >>>>>>>>>>> - this may be the root of your problem. >>>>>>>>>>> I don't think that I have ever even tried that. >>>>>>>>>>> >>>>>>>>>>> What are you expecting the authentication behavior to be there? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Thu, Jul 2, 2015 at 2:27 PM, Aneela Saleem < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> ${KNOX_HOME}/knox/conf/topologies has the following contents: >>>>>>>>>>>> >>>>>>>>>>>> admin.xml cluster1.xml README sandbox.xml >>>>>>>>>>>> >>>>>>>>>>>> cluster1.xml is the topology descriptor file created by me. >>>>>>>>>>>> >>>>>>>>>>>> cluster1.xml has following contents: >>>>>>>>>>>> >>>>>>>>>>>> <topology> >>>>>>>>>>>> <gateway> >>>>>>>>>>>> </gateway> >>>>>>>>>>>> <service> >>>>>>>>>>>> <role>NAMENODE</role> >>>>>>>>>>>> <url>hdfs:// namenode-host :8020</url> >>>>>>>>>>>> </service> >>>>>>>>>>>> >>>>>>>>>>>> <service> >>>>>>>>>>>> <role>JOBTRACKER</role> >>>>>>>>>>>> <url>rpc:// jobtracker-host :8050</url> >>>>>>>>>>>> </service> >>>>>>>>>>>> >>>>>>>>>>>> <service> >>>>>>>>>>>> <role>RESOURCEMANAGER</role> >>>>>>>>>>>> <url>http://red3:8088/ws</url> >>>>>>>>>>>> </service> >>>>>>>>>>>> >>>>>>>>>>>> <service> >>>>>>>>>>>> <role>WEBHDFS</role> >>>>>>>>>>>> <url>http://localhost:50070/webhdfs</url> >>>>>>>>>>>> </service> >>>>>>>>>>>> >>>>>>>>>>>> <service> >>>>>>>>>>>> <role>WEBHCAT</role> >>>>>>>>>>>> <url>http://webcat-host :50111/templeton</url> >>>>>>>>>>>> </service> >>>>>>>>>>>> >>>>>>>>>>>> <service> >>>>>>>>>>>> <role>OOZIE</role> >>>>>>>>>>>> <url>http://oozie-host :11000/oozie</url> >>>>>>>>>>>> </service> >>>>>>>>>>>> >>>>>>>>>>>> <service> >>>>>>>>>>>> <role>WEBHBASE</role> >>>>>>>>>>>> <url>http://webhbase-host :60080</url> >>>>>>>>>>>> </service> >>>>>>>>>>>> </topology> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> admin.xml has default contents. I did not make any changes. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Jul 2, 2015 at 11:15 PM, Steve Howard < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> To be clear, the error isn't hostname not found (that just >>>>>>>>>>>>> means it wasn't in the DNS cache on your computer), it's that the >>>>>>>>>>>>> URL can't >>>>>>>>>>>>> be found by the knox server. >>>>>>>>>>>>> >>>>>>>>>>>>> Can you post the contents of your >>>>>>>>>>>>> ${KNOX_HOME}/knox/conf/topologies directory? >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Jul 2, 2015 at 1:50 PM, Aneela Saleem < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm trying to connect to HDFS through Knox gateway. I run >>>>>>>>>>>>>> the following command: >>>>>>>>>>>>>> >>>>>>>>>>>>>> curl -vk >>>>>>>>>>>>>> https://localhost:8443/gateway/cluster1/WEBHDFS/v1?op=GETHOMEDIRECTORY >>>>>>>>>>>>>> >>>>>>>>>>>>>> when i run this command i get the following error: >>>>>>>>>>>>>> >>>>>>>>>>>>>> * Hostname was NOT found in DNS cache >>>>>>>>>>>>>> * Trying 127.0.0.1... >>>>>>>>>>>>>> * Connected to localhost (127.0.0.1) port 8443 (#0) >>>>>>>>>>>>>> * successfully set certificate verify locations: >>>>>>>>>>>>>> * CAfile: none >>>>>>>>>>>>>> CApath: /etc/ssl/certs >>>>>>>>>>>>>> * SSLv3, TLS handshake, Client hello (1): >>>>>>>>>>>>>> * SSLv3, TLS handshake, Server hello (2): >>>>>>>>>>>>>> * SSLv3, TLS handshake, CERT (11): >>>>>>>>>>>>>> * SSLv3, TLS handshake, Server key exchange (12): >>>>>>>>>>>>>> * SSLv3, TLS handshake, Server finished (14): >>>>>>>>>>>>>> * SSLv3, TLS handshake, Client key exchange (16): >>>>>>>>>>>>>> * SSLv3, TLS change cipher, Client hello (1): >>>>>>>>>>>>>> * SSLv3, TLS handshake, Finished (20): >>>>>>>>>>>>>> * SSLv3, TLS change cipher, Client hello (1): >>>>>>>>>>>>>> * SSLv3, TLS handshake, Finished (20): >>>>>>>>>>>>>> * SSL connection using ECDHE-RSA-DES-CBC3-SHA >>>>>>>>>>>>>> * Server certificate: >>>>>>>>>>>>>> * subject: C=US; ST=Test; L=Test; O=Hadoop; OU=Test; >>>>>>>>>>>>>> CN=localhost >>>>>>>>>>>>>> * start date: 2015-06-29 21:39:18 GMT >>>>>>>>>>>>>> * expire date: 2016-06-28 21:39:18 GMT >>>>>>>>>>>>>> * issuer: C=US; ST=Test; L=Test; O=Hadoop; OU=Test; >>>>>>>>>>>>>> CN=localhost >>>>>>>>>>>>>> * SSL certificate verify result: self signed certificate >>>>>>>>>>>>>> (18), continuing anyway. >>>>>>>>>>>>>> > GET /gateway/cluster1/WEBHDFS/v1?op=GETHOMEDIRECTORY >>>>>>>>>>>>>> HTTP/1.1 >>>>>>>>>>>>>> > User-Agent: curl/7.35.0 >>>>>>>>>>>>>> > Host: localhost:8443 >>>>>>>>>>>>>> > Accept: */* >>>>>>>>>>>>>> > >>>>>>>>>>>>>> < HTTP/1.1 404 Not Found >>>>>>>>>>>>>> < Content-Length: 0 >>>>>>>>>>>>>> * Server Jetty(8.1.14.v20131031) is not blacklisted >>>>>>>>>>>>>> < Server: Jetty(8.1.14.v20131031) >>>>>>>>>>>>>> < >>>>>>>>>>>>>> * Connection #0 to host localhost left intact >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Can anyone please help me in tackling this issue? >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>>>> >>>> >>> >> >
