Venkat, prism configs need to have prism as the domain in startup.properties.
For example: prism.application.services=org.apache.falcon.entity.store.ConfigurationStore prism.falcon.enableTLS=false You need to add these for prism specific configs. Also, may be the user authentication between falcon and prism might not be working. On Wed, Jul 16, 2014 at 4:54 AM, Venkatr <[email protected]> wrote: > Looking at the code, Prism uses HTTPS to talk to other Falcon servers in > distributed mode. So, I presume I need to set enableTLS = true (i'm setting > it to false) in the startup like below: > > *.falcon.enableTLS=true > *.keystore.file=/export/apps/falcon/latest/conf/prism.keystore > *.keystore.password=password > > So, > > 1. my FALCON_URL will be https -- correct? > 2. I need to use the same keystore file for both of my Falcon instances > 3. is this the right way to generate the keystore? > $keytool -genkey -alias tomcat -keyalg RSA -keystore > conf/prism.keystore > <password is password> > > Venkat > > > On Tue, Jul 15, 2014 at 2:49 PM, Venkatr <[email protected]> wrote: > > > Will do. One more issue after the Prism comes up (this should be > simple), > > I run the following CLI and that throws the HTTP ERROR CODE 401 > > (authorization). > > I think Prism talks to under lying Falcon servers using HTTPS. > > is there any config I need to do (like the keystore) before it can work? > > > > > > > > $ falcon entity -url $FALCON_URL -type cluster -file > > falconChurnDemo/primaryCluster-atlanta.xml -submit > > > > Error: Bad > > > Request;eat-1/org.apache.falcon.FalconException::org.apache.falcon.FalconException: > > <html> > > > > <head> > > <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/> > > <title>Error 401 </title> > > </head> > > <body><h2>HTTP ERROR 401</h2> > > <p>Problem accessing /secure/sync/submit/cluster. Reason: > > <pre> </pre></p><hr /><i><small>Powered by Jetty://</small></i><br/> > > </body> > > </html> > > > > > lva-1/org.apache.falcon.FalconException::org.apache.falcon.FalconException: > > <html> > > > > <head> > > <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/> > > <title>Error 401 </title> > > </head> > > <body><h2>HTTP ERROR 401</h2> > > <p>Problem accessing /secure/sync/submit/cluster. Reason: > > <pre> </pre></p><hr /><i><small>Powered by Jetty://</small></i><br/> > > </body> > > </html> > > > > *Log from the prism.application.log* > > > > 2014-07-15 21:41:38,603 DEBUG - [683347213@qtp-1401888126-0 > :veramach:POST//entities/submit/cluster > > 6ed34600-f74b-416d-8277-3a000dc5714f] ~ Executing > > > http://eat1-hcl0758.grid.linkedin.com:15000/secure/sync/submit/cluster?colo=eat-1&; > > (HTTPChannel:82) > > 2014-07-15 21:41:38,603 INFO - [683347213@qtp-1401888126-0 > :veramach:POST//entities/submit/cluster > > 6ed34600-f74b-416d-8277-3a000dc5714f] ~ Configuring client with > > > /export/apps/falcon/falcon-0.6-incubating-SNAPSHOT.debug.nertz.current/conf/prism.keystore > > (SecureHTTPChannel:56) > > 2014-07-15 21:41:38,677 ERROR - [683347213@qtp-1401888126-0 > :veramach:POST//entities/submit/cluster > > 6ed34600-f74b-416d-8277-3a000dc5714f] ~ Request failed: 401 > > (HTTPChannel:107) > > 2014-07-15 21:41:38,678 ERROR - [683347213@qtp-1401888126-0 > :veramach:POST//entities/submit/cluster > > 6ed34600-f74b-416d-8277-3a000dc5714f] ~ Request failed (HTTPChannel:111) > > org.apache.falcon.FalconException: <html> > > > > <head> > > <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/> > > <title>Error 401 </title> > > </head> > > <body><h2>HTTP ERROR 401</h2> > > <p>Problem accessing /secure/sync/submit/cluster. Reason: > > <pre> </pre></p><hr /><i><small>Powered by Jetty://</small></i><br/> > > <br/> > > <br/> > > <br/> > > <br/> > > <br/> > > <br/> > > <br/> > > <br/> > > <br/> > > <br/> > > <br/> > > <br/> > > <br/> > > <br/> > > <br/> > > <br/> > > <br/> > > <br/> > > <br/> > > > > </body> > > </html> > > > > at > > > org.apache.falcon.resource.channel.HTTPChannel.invoke(HTTPChannel.java:108) > > at > > > org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$1.doExecute(SchedulableEntityManagerProxy.java:124) > > at > > > org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$EntityProxy.execute(SchedulableEntityManagerProxy.java:416) > > at > > > org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.submit_aroundBody0(SchedulableEntityManagerProxy.java:126) > > at > > > org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$AjcClosure1.run(SchedulableEntityManagerProxy.java:1) > > at > > org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) > > at > > > org.apache.falcon.aspect.AbstractFalconAspect.logAround(AbstractFalconAspect.java:51) > > at > > > org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.submit(SchedulableEntityManagerProxy.java:107) > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > at java.lang.reflect.Method.invoke(Method.java:597) > > at > > > com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) > > at > > > com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185) > > at > > > com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) > > at > > > com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) > > at > > > com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) > > at > > > com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) > > at > > > com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) > > at > > > com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) > > at > > > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) > > > > > > > > On Tue, Jul 15, 2014 at 2:38 PM, Arpit Gupta <[email protected]> > > wrote: > > > >> Yes that is a bug. Go ahead and log a jira for it. We had logged one for > >> falcon and we are not testing Prism so never ran into it. > >> > >> -- > >> Arpit Gupta > >> Hortonworks Inc. > >> http://hortonworks.com/ > >> > >> On Jul 15, 2014, at 2:36 PM, Venkatr <[email protected]> wrote: > >> > >> > Hi Arpit, > >> > > >> > Looks like Prism does not expand _HOST in the principals, where as > >> Falcon > >> > expands it -- see the security audit logs. > >> > When I switched the principal to use machine name instead of _HOST, > the > >> > server starts up and I'm able to ping port 16000 via browser. > >> > > >> > Wondering both Falcon and Prism should use the same code for hadoop > >> > authentication. if so, why does it behave this way? is this a bug? > >> > > >> > Thanks > >> > Venkat > >> > > >> > [dm@eat1-hcl0758 logs]$ cat falcon.security.audit.log > >> > 2014-07-15 20:56:00,778 Login successful for user dm/ > >> > [email protected] using keytab file > >> > /export/apps/hadoop/keytabs/dm.keytab > >> > 2014-07-15 20:56:02,134 Login using keytab > >> > /export/apps/hadoop/keytabs/dm.keytab, for principal HTTP/ > >> > [email protected] > >> > > >> > [dm@eat1-hcl0758 logs]$ cat prism.security.audit.log > >> > 2014-07-15 20:56:02,138 Initialized, principal [HTTP/ > >> > [email protected]] from keytab > >> > [/export/apps/hadoop/keytabs/dm.keytab] > >> > 2014-07-15 20:49:53,427 Login using keytab > >> > /export/apps/hadoop/keytabs/dm.keytab, for principal HTTP/_ > >> > [email protected] > >> > > >> > > >> > > >> > On Tue, Jul 15, 2014 at 1:46 PM, Arpit Gupta <[email protected]> > >> wrote: > >> > > >> >> Can you make the same curl call to port 15000 then? > >> >> > >> >> -- > >> >> Arpit Gupta > >> >> Hortonworks Inc. > >> >> http://hortonworks.com/ > >> >> > >> >> On Jul 15, 2014, at 1:09 PM, Venkat R <[email protected]> > >> wrote: > >> >> > >> >>> Prism and Falcon for colo-1 are running on the same machine and > Falcon > >> >> for colo-2 is running on a different machine. > >> >>> > >> >>> So, I'm sharing the config files with Prisim and Falcon colo-1. > >> >>> I think it should be okay? > >> >>> > >> >>> > >> >>> On Tuesday, July 15, 2014 1:03 PM, Arpit Gupta < > [email protected] > >> > > >> >> wrote: > >> >>> > >> >>> > >> >>> > >> >>> you cant use the same config for falcon and prism servers they are > >> >> running on different hosts at least from the hostname you mention. > >> >>> > >> >>> The falcon service principal and spnego principal both have to have > >> >> hostnames as part of them. For example if your host is " > >> >> eat1-server1.grid.example.com" > >> >>> > >> >>> then your falcon service principal would be > >> >> "falcon/eat1-server1.grid.example.com@REALM" and spnego would be > >> >> "HTTP/eat1-server1.grid.example.com@REALM" > >> >>> > >> >>> > >> >>> If you are using _HOST in the configs instead of the real hostname > >> then > >> >> you have to make sure the appropriate principal's are available in > >> keytabs. > >> >>> > >> >>> -- > >> >>> Arpit Gupta > >> >>> Hortonworks Inc. > >> >>> http://hortonworks.com/ > >> >>> > >> >>> On Jul 15, 2014, at 12:16 PM, Venkat R <[email protected] > > > >> >> wrote: > >> >>> > >> >>>> Hi Arpit, > >> >>>> > >> >>>> curl --negotiate -u : "http://eat1-server1.grid.example.com:16000/ > " > >> >>>> <html> > >> >>>> <head> > >> >>>> <meta http-equiv="Content-Type" content="text/html; > >> >> charset=ISO-8859-1"/> > >> >>>> <title>Error 503 SERVICE_UNAVAILABLE</title> > >> >>>> </head> > >> >>>> <body> > >> >>>> <h2>HTTP ERROR: 503</h2> > >> >>>> <p>Problem accessing /. Reason: > >> >>>> <pre> SERVICE_UNAVAILABLE</pre></p> > >> >>>> <hr /><i><small>Powered by Jetty://</small></i> > >> >>>> </body> > >> >>>> </html> > >> >>>> > >> >>>> The startup.properties points to the correct keytabs containing > both > >> >> the falcon user and HTTP principals. The Falcon server starts without > >> any > >> >> issue (or exception). > >> >>>> > >> >>>> Command to start prism: > >> >>>> $ bin/prism-start -port 16000 > >> >>>> $ bin/prism-status > >> >>>> Hadoop is installed, adding hadoop classpath to falcon classpath > >> >>>> Falcon server is running (on > >> >> http://eat1-hcl0758.grid.linkedin.com:15000/) > >> >>>> > >> >>>> runtime.properties > >> >>>> > >> >>>> *.all.colos=eat-1, lva-1 > >> >>>> *.falcon.eat-1.endpoint=http://eat1-server1.grid.example.com:15000 > >> >>>> *.falcon.lva-1.endpoint=http://lva1-server1.grid.example.com:15000 > >> >>>> #falcon server should have the following properties > >> >>>> falcon.current.colo=eat-1 > >> >>>> ######### Authentication Properties ######### > >> >>>> falcon.enableTLS=false > >> >>>> > >> >>>> The startup properties remains the same as the one I used for > >> >> standalone version (nothing changed). > >> >>>> > >> >>>> is there something else in the config I'm missing? > >> >>>> > >> >>>> Thanks > >> >>>> > >> >>>> > >> >>>> > >> >>>> On Tuesday, July 15, 2014 9:17 AM, Arpit Gupta < > >> [email protected]> > >> >> wrote: > >> >>>> > >> >>>> > >> >>>> > >> >>>> Then check your service principal and spnego principal properties > and > >> >> make sure the keytab location and the principal configured are > correct. > >> >>>> > >> >>>> From the exception it could not log in using the keytab provided. > >> >>>> > >> >>>> -- > >> >>>> Arpit Gupta > >> >>>> Hortonworks Inc. > >> >>>> http://hortonworks.com/ > >> >>>> > >> >>>> On Jul 15, 2014, at 9:14 AM, [email protected] > >> >> <[email protected]> wrote: > >> >>>> > >> >>>>> Arpit > >> >>>>> > >> >>>>> Will try, but the exception I see is in the prism.application.log > >> and > >> >> so the service is not up. > >> >>>>> > >> >>>>> Sent from my HTC > >> >>>>> > >> >>>>> ----- Reply message ----- > >> >>>>> From: "Arpit Gupta" <[email protected]> > >> >>>>> To: "[email protected]" < > >> [email protected]>, > >> >> "Venkat R" <[email protected]> > >> >>>>> Subject: Prism server setup > >> >>>>> Date: Tue, Jul 15, 2014 8:46 AM > >> >>>>> > >> >>>>> If you are running secure falcon than the browser will need spnego > >> >> support > >> >>>>> in order to show the UI. The error message the user sees can be > >> >> improved > >> >>>>> but you will need to configure your browser to do spnego > negotiate. > >> >>>>> > >> >>>>> After kinit run the following call > >> >>>>> > >> >>>>> curl --negotiate -u : " > http://eat1-hcl0758.grid.linkedin.com:16000/ > >> " > >> >> and > >> >>>>> see if it goes through. > >> >>>>> > >> >>>>> Arpit > >> >>>>> > >> >>>>> > >> >>>>> On Mon, Jul 14, 2014 at 6:28 PM, Venkat R > >> <[email protected] > >> >>> > >> >>>>> wrote: > >> >>>>> > >> >>>>>> Hi All, > >> >>>>>> > >> >>>>>> I followed the instructions here > >> >>>>>> > >> >> > >> > https://blogs.apache.org/falcon/entry/starting_falcon_in_distributed_mode > >> >> and > >> >>>>>> made the necessary changes to the conf/runtime.properties as > below: > >> >>>>>> > >> >>>>>> <verbatim> > >> >>>>>> > >> >>>>>> *.all.colos=eat-1, lva-1 > >> >>>>>> *.falcon.eat-1.endpoint= > http://eat1-server1.grid.example.com:15000 > >> >>>>>> *.falcon.lva-1.endpoint= > http://lva1-server2.grid.example.com:15000 > >> >>>>>> > >> >>>>>> #falcon server should have the following properties > >> >>>>>> falcon.current.colo=eat-1 > >> >>>>>> > >> >>>>>> </verbatim> > >> >>>>>> > >> >>>>>> I started the prism server as follows: > >> >>>>>> > >> >>>>>> bin/prism-start -port 16000 > >> >>>>>> > >> >>>>>> and the status report ok. But browser reports error when I try to > >> >> access > >> >>>>>> http://eat1-hcl0758.grid.linkedin.com:16000/ > >> >>>>>> > >> >>>>>> return ERROR 503. > >> >>>>>> > >> >>>>>> And the prims log has the following exception: > >> >>>>>> > >> >>>>>> Not sure what this password being asked. > >> >>>>>> > >> >>>>>> The use lannching the Prism server has kerberos TGT in the cache. > >> >>>>>> > >> >>>>>> Thanks > >> >>>>>> --Venkat > >> >>>>>> > >> >>>>>> > >> >>>>>> 2014-07-15 01:19:21,426 WARN - [main:] ~ Nested in > >> >>>>>> javax.servlet.ServletException: > >> >> javax.security.auth.login.LoginException: > >> >>>>>> Unable to obtain password from user > >> >>>>>> : (log:89) > >> >>>>>> javax.security.auth.login.LoginException: Unable to obtain > password > >> >> from > >> >>>>>> user > >> >>>>>> > >> >>>>>> at > >> >>>>>> > >> >> > >> > com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:789) > >> >>>>>> at > >> >>>>>> > >> >> > >> > com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:654) > >> >>>>>> at > >> >>>>>> > >> >> > >> > com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:542) > >> >>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > >> >> Method) > >> >>>>>> at > >> >>>>>> > >> >> > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > >> >>>>>> at > >> >>>>>> > >> >> > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > >> >>>>>> at java.lang.reflect.Method.invoke(Method.java:597) > >> >>>>>> at > >> >>>>>> > >> javax.security.auth.login.LoginContext.invoke(LoginContext.java:769) > >> >>>>>> at > >> >>>>>> > >> >> > >> javax.security.auth.login.LoginContext.access$000(LoginContext.java:186) > >> >>>>>> at > >> >>>>>> > javax.security.auth.login.LoginContext$5.run(LoginContext.java:706) > >> >>>>>> at java.security.AccessController.doPrivileged(Native > >> Method) > >> >>>>>> at > >> >>>>>> > >> >> > >> > javax.security.auth.login.LoginContext.invokeCreatorPriv(LoginContext.java:703) > >> >>>>>> at > >> >>>>>> > javax.security.auth.login.LoginContext.login(LoginContext.java:575) > >> >>>>>> at > >> >>>>>> > >> >> > >> > org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.init(KerberosAuthenticationHandler.java:187) > >> >>>>>> at > >> >>>>>> > >> >> > >> > org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:146) > >> >>>>>> at > >> >>>>>> > >> >> > >> org.apache.falcon.security.BasicAuthFilter.init(BasicAuthFilter.java:82) > >> >>>>>> at > >> >>>>>> > >> org.mortbay.jetty.servlet.FilterHolder.doStart(FilterHolder.java:97) > >> >>>>>> at > >> >>>>>> > >> >> > >> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > >> >>>>>> at > >> >>>>>> > >> >> > >> > org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:713) > >> >>>>>> at > >> >> org.mortbay.jetty.servlet.Context.startContext(Context.java:140) > >> >>>>>> at > >> >>>>>> > >> >> > >> > org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282) > >> >>>>>> > >> >>>>> > >> >>>>> -- > >> >>>>> CONFIDENTIALITY NOTICE > >> >>>>> NOTICE: This message is intended for the use of the individual or > >> >> entity to > >> >>>>> which it is addressed and may contain information that is > >> confidential, > >> >>>>> privileged and exempt from disclosure under applicable law. If the > >> >> reader > >> >>>>> of this message is not the intended recipient, you are hereby > >> notified > >> >> that > >> >>>>> any printing, copying, dissemination, distribution, disclosure or > >> >>>>> forwarding of this communication is strictly prohibited. If you > have > >> >>>>> received this communication in error, please contact the sender > >> >> immediately > >> >>>>> and delete it from your system. Thank You. > >> >>> > >> >>>> > >> >>>> > >> >>>> > >> >>>> -- > >> >>>> CONFIDENTIALITY NOTICE > >> >>>> NOTICE: This message is intended for the use of the individual or > >> >> entity to > >> >>>> which it is addressed and may contain information that is > >> confidential, > >> >>>> privileged and exempt from disclosure under applicable law. If the > >> >> reader > >> >>>> of this message is not the intended recipient, you are hereby > >> notified > >> >> that > >> >>>> any printing, copying, dissemination, distribution, disclosure or > >> >>>> forwarding of this communication is strictly prohibited. If you > have > >> >>>> received this communication in error, please contact the sender > >> >> immediately > >> >>>> and delete it from your system. Thank You. > >> >>> > >> >>> > >> >>> -- > >> >>> CONFIDENTIALITY NOTICE > >> >>> NOTICE: This message is intended for the use of the individual or > >> entity > >> >> to > >> >>> which it is addressed and may contain information that is > >> confidential, > >> >>> privileged and exempt from disclosure under applicable law. If the > >> reader > >> >>> of this message is not the intended recipient, you are hereby > notified > >> >> that > >> >>> any printing, copying, dissemination, distribution, disclosure or > >> >>> forwarding of this communication is strictly prohibited. If you have > >> >>> received this communication in error, please contact the sender > >> >> immediately > >> >>> and delete it from your system. Thank You. > >> >> > >> >> > >> >> -- > >> >> CONFIDENTIALITY NOTICE > >> >> NOTICE: This message is intended for the use of the individual or > >> entity to > >> >> which it is addressed and may contain information that is > confidential, > >> >> privileged and exempt from disclosure under applicable law. If the > >> reader > >> >> of this message is not the intended recipient, you are hereby > notified > >> that > >> >> any printing, copying, dissemination, distribution, disclosure or > >> >> forwarding of this communication is strictly prohibited. If you have > >> >> received this communication in error, please contact the sender > >> immediately > >> >> and delete it from your system. Thank You. > >> >> > >> > >> > >> -- > >> CONFIDENTIALITY NOTICE > >> NOTICE: This message is intended for the use of the individual or entity > >> to > >> which it is addressed and may contain information that is confidential, > >> privileged and exempt from disclosure under applicable law. If the > reader > >> of this message is not the intended recipient, you are hereby notified > >> that > >> any printing, copying, dissemination, distribution, disclosure or > >> forwarding of this communication is strictly prohibited. If you have > >> received this communication in error, please contact the sender > >> immediately > >> and delete it from your system. Thank You. > >> > > > > > -- Regards, Venkatesh “Perfection (in design) is achieved not when there is nothing more to add, but rather when there is nothing more to take away.” - Antoine de Saint-Exupéry
