Hello Venkat, Thanks for quick response. Could you please guide to any contacts in HWX or any online server where we can find the patch / answer.
Do you know in which version this issue is being resolved. Thanks in advance. Regards, Vishal. On Thu, Dec 22, 2016 at 1:34 PM, Venkat Ranganathan < [email protected]> wrote: > > Caused by: AUTHENTICATION : Could not authenticate, GSSException: No > valid credentials provided (Mechanism level: Failed to find any Kerberos > tgt) > > This shows that the Falcon kerberos context become invalidated. There > was a fix in later versions and patch should be available from HWX support > for the older versions > > Thanks > Venkat > ________________________________________ > From: Vishal Gupta <[email protected]> > Sent: Wednesday, December 21, 2016 9:19 PM > To: [email protected] > Subject: Re: Falcon issue after Kerberos implementation > > Cluster is HDP 2.3.2 > > On Thu, Dec 22, 2016 at 1:17 PM, Vishal Gupta <[email protected]> wrote: > > > Hello Team, > > We are implementing Kerberos at one the clusters. > > > > After Kerberos implementation looks like Falcon while using SPNEGO to > > authenticate the user, is failing while authenticating and connecting to > > Oozie. > > > > > > > > Any suggestions please. Details below.. > > > > > > 2016-12-21 07:52:42,597 ERROR - [1529601858@qtp-708533063-243 - > > 737162dd-6235-494c-a0b4-c8e132fc5491:ifrsdev:DELETE// > > entities/delete/process/edmhdpif-oozie-icbdev-th] ~ Unable to reach > > workflow engine for deletion or deletion failed > (AbstractEntityManager:265) > > > > org.apache.falcon.FalconException: IO_ERROR : java.io.IOException: Error > > while connecting Oozie server. No of retries = 1. Exception = null > > > > at org.apache.falcon.workflow. > engine.OozieWorkflowEngine. > > findBundles(OozieWorkflowEngine.java:303) > > > > at org.apache.falcon.workflow. > engine.OozieWorkflowEngine. > > doBundleAction(OozieWorkflowEngine.java:377) > > > > at org.apache.falcon.workflow. > engine.OozieWorkflowEngine. > > doBundleAction(OozieWorkflowEngine.java:371) > > > > at org.apache.falcon.workflow. > engine.OozieWorkflowEngine. > > delete(OozieWorkflowEngine.java:355) > > > > at org.apache.falcon.resource. > > AbstractEntityManager.delete(AbstractEntityManager.java:252) > > > > at org.apache.falcon.resource.ConfigSyncService.delete( > > ConfigSyncService.java:62) > > > > at sun.reflect.GeneratedMethodAccessor88.invoke(Unknown > > Source) > > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > > DelegatingMethodAccessorImpl.java:43) > > > > at java.lang.reflect.Method.invoke(Method.java:498) > > > > at org.apache.falcon.resource.channel.IPCChannel.invoke( > > IPCChannel.java:49) > > > > at org.apache.falcon.resource.proxy. > > SchedulableEntityManagerProxy$3.doExecute(SchedulableEntityManagerProxy. > > java:230) > > > > at org.apache.falcon.resource.proxy. > > SchedulableEntityManagerProxy$EntityProxy.execute( > > SchedulableEntityManagerProxy.java:575) > > > > at org.apache.falcon.resource.proxy. > > SchedulableEntityManagerProxy$3.execute(SchedulableEntityManagerProxy. > > java:219) > > > > at org.apache.falcon.resource.proxy. > > SchedulableEntityManagerProxy.delete_aroundBody2( > > SchedulableEntityManagerProxy.java:232) > > > > at org.apache.falcon.resource.proxy. > > SchedulableEntityManagerProxy$AjcClosure3.run( > > SchedulableEntityManagerProxy.java:1) > > > > at org.aspectj.runtime.reflect.JoinPointImpl.proceed( > > JoinPointImpl.java:149) > > > > at org.apache.falcon.aspect.AbstractFalconAspect. > > logAroundMonitored(AbstractFalconAspect.java:51) > > > > at org.apache.falcon.resource.proxy. > > SchedulableEntityManagerProxy.delete(SchedulableEntityManagerProxy. > > java:206) > > > > at sun.reflect.GeneratedMethodAccessor87.invoke(Unknown > > Source) > > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > > DelegatingMethodAccessorImpl.java:43) > > > > at java.lang.reflect.Method.invoke(Method.java:498) > > > > at com.sun.jersey.spi.container. > > JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) > > > > at com.sun.jersey.server.impl.model.method.dispatch. > > AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch( > > AbstractResourceMethodDispatchProvider.java:185) > > > > at com.sun.jersey.server.impl.model.method.dispatch. > > ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher. > > java:75) > > > > at com.sun.jersey.server.impl.uri.rules.HttpMethodRule. > > accept(HttpMethodRule.java:288) > > > > at com.sun.jersey.server.impl. > uri.rules.RightHandPathRule. > > accept(RightHandPathRule.java:147) > > > > at com.sun.jersey.server.impl. > uri.rules.ResourceClassRule. > > accept(ResourceClassRule.java:108) > > > > at com.sun.jersey.server.impl. > uri.rules.RightHandPathRule. > > accept(RightHandPathRule.java:147) > > > > at com.sun.jersey.server.impl.uri.rules. > > RootResourceClassesRule.accept(RootResourceClassesRule.java:84) > > > > at com.sun.jersey.server.impl.application. > > WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) > > > > at com.sun.jersey.server.impl.application. > > WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) > > > > at com.sun.jersey.server.impl.application. > > WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) > > > > at com.sun.jersey.server.impl.application. > > WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) > > > > at com.sun.jersey.spi.container. > > servlet.WebComponent.service(WebComponent.java:416) > > > > at com.sun.jersey.spi.container. > servlet.ServletContainer. > > service(ServletContainer.java:537) > > > > at com.sun.jersey.spi.container. > servlet.ServletContainer. > > service(ServletContainer.java:699) > > > > at javax.servlet.http.HttpServlet.service( > > HttpServlet.java:820) > > > > at org.mortbay.jetty.servlet.ServletHolder.handle( > > ServletHolder.java:511) > > > > at org.mortbay.jetty.servlet.ServletHandler$CachedChain. > > doFilter(ServletHandler.java:1221) > > > > at org.apache.falcon.security.FalconAuthorizationFilter. > > doFilter(FalconAuthorizationFilter.java:106) > > > > at org.mortbay.jetty.servlet.ServletHandler$CachedChain. > > doFilter(ServletHandler.java:1212) > > > > at org.apache.falcon.security. > > FalconAuthenticationFilter$2.doFilter(FalconAuthenticationFilter. > java:184) > > > > at org.apache.hadoop.security.authentication.server. > > AuthenticationFilter.doFilter(AuthenticationFilter.java:595) > > > > at org.apache.hadoop.security.authentication.server. > > AuthenticationFilter.doFilter(AuthenticationFilter.java:554) > > > > at org.apache.falcon.security. > FalconAuthenticationFilter. > > doFilter(FalconAuthenticationFilter.java:193) > > > > at org.mortbay.jetty.servlet.ServletHandler$CachedChain. > > doFilter(ServletHandler.java:1212) > > > > at org.apache.falcon.security. > FalconAuditFilter.doFilter( > > FalconAuditFilter.java:64) > > > > at org.mortbay.jetty.servlet.ServletHandler$CachedChain. > > doFilter(ServletHandler.java:1212) > > > > at org.mortbay.jetty.servlet.ServletHandler.handle( > > ServletHandler.java:399) > > > > at org.mortbay.jetty.security.SecurityHandler.handle( > > SecurityHandler.java:216) > > > > at org.mortbay.jetty.servlet.SessionHandler.handle( > > SessionHandler.java:182) > > > > at org.mortbay.jetty.handler.ContextHandler.handle( > > ContextHandler.java:767) > > > > at org.mortbay.jetty.webapp.WebAppContext.handle( > > WebAppContext.java:450) > > > > at org.mortbay.jetty.handler.HandlerWrapper.handle( > > HandlerWrapper.java:152) > > > > at org.mortbay.jetty.Server.handle(Server.java:326) > > > > at org.mortbay.jetty.HttpConnection.handleRequest( > > HttpConnection.java:542) > > > > at org.mortbay.jetty.HttpConnection$RequestHandler. > > headerComplete(HttpConnection.java:928) > > > > at org.mortbay.jetty.HttpParser. > > parseNext(HttpParser.java:549) > > > > at org.mortbay.jetty.HttpParser. > parseAvailable(HttpParser. > > java:212) > > > > at org.mortbay.jetty.HttpConnection.handle( > > HttpConnection.java:404) > > > > at org.mortbay.jetty.bio.SocketConnector$Connection. > > run(SocketConnector.java:228) > > > > at org.mortbay.thread.QueuedThreadPool$PoolThread. > > run(QueuedThreadPool.java:582) > > > > Caused by: IO_ERROR : java.io.IOException: Error while connecting Oozie > > server. No of retries = 1. Exception = null > > > > at org.apache.oozie.client. > OozieClient.validateWSVersion( > > OozieClient.java:374) > > > > at org.apache.oozie.client.OozieClient.createURL( > > OozieClient.java:459) > > > > at org.apache.oozie.client.OozieClient.access$000( > > OozieClient.java:80) > > > > at org.apache.oozie.client.OozieClient$ClientCallable. > > call(OozieClient.java:555) > > > > at org.apache.oozie.client. > OozieClient.getBundleJobsInfo( > > OozieClient.java:2149) > > > > at org.apache.oozie.client.ProxyOozieClient.access$2901( > > ProxyOozieClient.java:48) > > > > at org.apache.oozie.client.ProxyOozieClient$29.call( > > ProxyOozieClient.java:598) > > > > at org.apache.oozie.client.ProxyOozieClient$29.call( > > ProxyOozieClient.java:596) > > > > at org.apache.oozie.client.OozieClient.doAs(OozieClient. > > java:244) > > > > at org.apache.oozie.client.ProxyOozieClient. > > getBundleJobsInfo(ProxyOozieClient.java:596) > > > > at org.apache.falcon.workflow. > engine.OozieWorkflowEngine. > > findBundles(OozieWorkflowEngine.java:290) > > > > ... 61 more > > > > Caused by: java.io.IOException: Error while connecting Oozie server. No > of > > retries = 1. Exception = null > > > > at org.apache.oozie.client.retry. > > ConnectionRetriableClient.execute(ConnectionRetriableClient.java:66) > > > > at org.apache.oozie.client.OozieClient. > > createRetryableConnection(OozieClient.java:504) > > > > at org.apache.oozie.client.OozieClient. > > getSupportedProtocolVersions(OozieClient.java:384) > > > > at org.apache.oozie.client. > OozieClient.validateWSVersion( > > OozieClient.java:344) > > > > ... 71 more > > > > Caused by: java.lang.reflect.UndeclaredThrowableException > > > > at org.apache.hadoop.security.UserGroupInformation.doAs( > > UserGroupInformation.java:1672) > > > > at org.apache.oozie.client.ProxyOozieClient. > > createConnection(ProxyOozieClient.java:79) > > > > at org.apache.oozie.client.OozieClient$1.doExecute( > > OozieClient.java:507) > > > > at org.apache.oozie.client.retry. > > ConnectionRetriableClient.execute(ConnectionRetriableClient.java:44) > > > > ... 74 more > > > > Caused by: AUTHENTICATION : Could not authenticate, GSSException: No > valid > > credentials provided (Mechanism level: Failed to find any Kerberos tgt) > > > > at org.apache.oozie.client.AuthOozieClient. > > createConnection(AuthOozieClient.java:136) > > > > at org.apache.oozie.client.ProxyOozieClient.access$001( > > ProxyOozieClient.java:48) > > > > at org.apache.oozie.client.ProxyOozieClient$1.run( > > ProxyOozieClient.java:81) > > > > at org.apache.oozie.client.ProxyOozieClient$1.run( > > ProxyOozieClient.java:79) > > > > at java.security.AccessController.doPrivileged(Native > > Method) > > > > at javax.security.auth.Subject.doAs(Subject.java:422) > > > > at org.apache.hadoop.security.UserGroupInformation.doAs( > > UserGroupInformation.java:1657) > > > > ... 77 more > > > > Caused by: org.apache.hadoop.security.authentication.client. > AuthenticationException: > > GSSException: No valid credentials provided (Mechanism level: Failed to > > find any Kerberos tgt) > > > > at org.apache.hadoop.security.authentication.client. > > KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:332) > > > > at org.apache.hadoop.security.authentication.client. > > KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205) > > > > at org.apache.hadoop.security.authentication.client. > > AuthenticatedURL.openConnection(AuthenticatedURL.java:215) > > > > at org.apache.oozie.client.AuthOozieClient. > > createConnection(AuthOozieClient.java:127) > > > > ... 83 more > > > > Caused by: GSSException: No valid credentials provided (Mechanism level: > > Failed to find any Kerberos tgt) > > > > at sun.security.jgss.krb5.Krb5InitCredential. > getInstance( > > Krb5InitCredential.java:147) > > > > at sun.security.jgss.krb5.Krb5MechFactory. > > getCredentialElement(Krb5MechFactory.java:122) > > > > at sun.security.jgss.krb5.Krb5MechFactory. > > getMechanismContext(Krb5MechFactory.java:187) > > > > at sun.security.jgss.GSSManagerImpl.getMechanismContext( > > GSSManagerImpl.java:224) > > > > at sun.security.jgss.GSSContextImpl.initSecContext( > > GSSContextImpl.java:212) > > > > at sun.security.jgss.GSSContextImpl.initSecContext( > > GSSContextImpl.java:179) > > > > at org.apache.hadoop.security.authentication.client. > > KerberosAuthenticator$1.run(KerberosAuthenticator.java:311) > > > > at org.apache.hadoop.security.authentication.client. > > KerberosAuthenticator$1.run(KerberosAuthenticator.java:287) > > > > at java.security.AccessController.doPrivileged(Native > > Method) > > > > at javax.security.auth.Subject.doAs(Subject.java:422) > > > > at org.apache.hadoop.security.authentication.client. > > KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:287) > > > > ... 86 more > > > > > > > > > > >
