MariaCarrie created PHOENIX-5145: ------------------------------------ Summary: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) Key: PHOENIX-5145 URL: https://issues.apache.org/jira/browse/PHOENIX-5145 Project: Phoenix Issue Type: Bug Affects Versions: 5.0.0 Environment: >HDP 3.0.0
>Phoenix 5.0.0 >HBase 2.0.0 >Spark 2.3.1 >Hadoop 3.0.1 Reporter: MariaCarrie I can successfully read the data using the local mode. Here is my code: ^val sqlContext: SQLContext = missionSession.app.ss.sqlContext^ ^System.setProperty("sun.security.krb5.debug", "true")^ ^System.setProperty("sun.security.spnego.debug", "true")^ ^UserGroupInformation.loginUserFromKeytab("d...@devdip.org", "devdmp.keytab")^ ^// Load as a DataFrame directly using a Configuration object^ ^val df: DataFrame = sqlContext.phoenixTableAsDataFrame(missionSession.config.tableName, Seq("ID"), zkUrl = Some(missionSession.config.zkUrl))^ ^df.show(5)^ But when I submit this to YARN for execution, an exception will be thrown. The following is the exception information: ^Tue Feb 19 13:07:53 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^ ^Tue Feb 19 13:07:53 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^ ^Tue Feb 19 13:07:53 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^ ^Tue Feb 19 13:07:54 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^ ^Tue Feb 19 13:07:54 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^ ^Tue Feb 19 13:07:55 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^ ^Tue Feb 19 13:07:57 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^ ^Tue Feb 19 13:08:01 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:08:11 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:08:21 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:08:31 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:08:41 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:09:01 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:09:22 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:09:42 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:10:02 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:10:22 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:10:42 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:11:02 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:11:22 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:11:42 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:12:02 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:12:23 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:12:43 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:13:03 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:13:23 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:13:43 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:14:03 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:14:23 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:14:43 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:15:03 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:15:23 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:15:44 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:16:04 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:16:24 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:16:44 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:145)^ ^at org.apache.hadoop.hbase.client.HTable.get(HTable.java:386)^ ^at org.apache.hadoop.hbase.client.HTable.get(HTable.java:360)^ ^at org.apache.hadoop.hbase.MetaTableAccessor.getTableState(MetaTableAccessor.java:1076)^ ^at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:399)^ ^at org.apache.hadoop.hbase.client.HBaseAdmin$6.rpcCall(HBaseAdmin.java:437)^ ^at org.apache.hadoop.hbase.client.HBaseAdmin$6.rpcCall(HBaseAdmin.java:434)^ ^at org.apache.hadoop.hbase.client.RpcRetryingCallable.call(RpcRetryingCallable.java:58)^ ^at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)^ ^... 55 more^ ^Caused by: java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180)^ ^at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390)^ ^at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95)^ ^at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410)^ ^at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406)^ ^at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103)^ ^at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118)^ ^at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:92)^ ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)^ ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)^ ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)^ ^at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:108)^ ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)^ ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)^ ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)^ ^at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:108)^ ^at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:353)^ ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)^ ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)^ ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)^ ^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1377)^ ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)^ ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)^ ^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:929)^ ^at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:179)^ ^at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$500(NettyRpcConnection.java:71)^ ^at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:247)^ ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)^ ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)^ ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)^ ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:163)^ ^at org.apache.hadoop.hbase.ipc.NettyRpcConnection.saslNegotiate(NettyRpcConnection.java:201)^ ^at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$800(NettyRpcConnection.java:71)^ ^at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:273)^ ^at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:261)^ ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)^ ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)^ ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)^ ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)^ ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)^ ^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)^ ^at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:306)^ ^at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:341)^ ^at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633)^ ^at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)^ ^at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)^ ^at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)^ ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)^ ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)^ ^... 1 more^ ^Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)^ ^at org.apache.hadoop.hbase.security.AbstractHBaseSaslRpcClient.getInitialResponse(AbstractHBaseSaslRpcClient.java:131)^ ^at org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler$1.run(NettyHBaseSaslRpcClientHandler.java:110)^ ^at org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler$1.run(NettyHBaseSaslRpcClientHandler.java:106)^ ^at java.security.AccessController.doPrivileged(Native Method)^ ^at javax.security.auth.Subject.doAs(Subject.java:422)^ ^at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)^ ^at org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.handlerAdded(NettyHBaseSaslRpcClientHandler.java:106)^ ^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:606)^ ^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:187)^ ^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:380)^ ^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:359)^ ^at org.apache.hadoop.hbase.ipc.NettyRpcConnection.saslNegotiate(NettyRpcConnection.java:200)^ ^... 18 more^ ^Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)^ ^at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)^ ^at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)^ ^at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)^ ^at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)^ ^at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)^ ^at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)^ ^at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)^ ^... 30 more^ The following are my submission parameters: ^spark-submit^ ^--master yarn^ ^--name PHOENIX_SPARK_PLUGIN^ ^--deploy-mode cluster^ ^--keytab /tesdmp/keytabs/nnjKorRc37PPPjLf/dmp/testdmp.keytab^ ^--principal d...@testdip.org^ ^--jars /usr/hdp/3.0.0.0-1634/phoenix/phoenix-5.0.0.3.0.0.0-1634-client.jar^ ^--conf spark.yarn.maxAppAttempts=1^ ^/opt/workspace/plugin/phoenix-spark-plugin-example-1.10.0-SNAPSHOT-jar-with-dependencies.jar "DMP_CONF=\{\"spark\":{\"sparkMaster\":\"yarn\"},\"zkUrl\":\"jdbc:phoenix:test-dmp5.fengdai.org,test-dmp3.fengdai.org,test-dmp4.fengdai.org:2181/hbase-secure\",\"tableName\":\"DMP_TEST\" ,\"isDS\":true}"^ This program runs on YARN. Do I need to do something extra? Can anyone give me some advice? -- This message was sent by Atlassian JIRA (v7.6.3#76005)