[
https://issues.apache.org/jira/browse/NIFI-7831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264758#comment-17264758
]
Rastislav Krist commented on NIFI-7831:
---------------------------------------
we have made some experiments - the message starts appearing in the logs
shortly before Kerberos ticket lifetime expires, but the only problem seems to
be the message in logs as the requests to HBase (and HDFS too) seems to keep
working.
I have examined the workaround from NIFI-7954 slightly - it seems to me that it
attempts to do each request to HBase with a fresh TGT (not sure, but this is
what UserGroupInformation.doAs does, I guess). The error message is coming from
HDFS client thread which is attempting to renew the original TGT, which doesn't
seems to be used after this patch and the error message can be thus ignored.
But this is just what I have seen so far.
>From my point of view, if the only problem is the error message in logs (while
>the requests keep working), I can live with it. Can anyone confirm, that the
>my observations are (roughly) correct and the error message can be ignored?
> KeytabCredentialsService not working with HBase Clients
> -------------------------------------------------------
>
> Key: NIFI-7831
> URL: https://issues.apache.org/jira/browse/NIFI-7831
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 1.12.0
> Reporter: Manuel Navarro
> Assignee: Tamas Palfy
> Priority: Major
> Fix For: 1.13.0
>
>
> HBase Client (both 1.x and 2.x) is not able to renew ticket after expiration
> with KeytabCredentialsService configured (same behaviour with principal and
> password configured directly in the controller service). The same
> KeytabCredentialsService works ok with Hive and Hbase clients configured in
> the same NIFI cluster.
> Note that the same configuration works ok in version 1.11 (error start to
> appear after upgrade from 1.11 to 1.12).
> After 24hours (time renewal period in our case), the following error appears
> using HBase_2_ClientServices + HBase_2_ClientMapCacheService :
> {code:java}
> 2020-09-17 09:00:27,014 ERROR [Relogin service.Chore.1]
> org.apache.hadoop.hbase.AuthUtil Got exception while trying to refresh
> credentials: loginUserFromKeyTab must be done first java.io.IOException:
> loginUserFromKeyTab must be done first at
> org.apache.hadoop.security.UserGroupInformation.reloginFromKeytab(UserGroupInformation.java:1194)
> at
> org.apache.hadoop.security.UserGroupInformation.checkTGTAndReloginFromKeytab(UserGroupInformation.java:1125)
> at org.apache.hadoop.hbase.AuthUtil$1.chore(AuthUtil.java:206) at
> org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186) at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
>
> With HBase_1_1_2_ClientServices + HBase_1_1_2_ClientMapCacheService the
> following error appears:
>
> {code:java}
> 2020-09-22 12:18:37,184 WARN [hconnection-0x55d9d8d1-shared--pool3-t769]
> o.a.hadoop.hbase.ipc.AbstractRpcClient Exception encountered while connecting
> to the server : javax.security.sasl.SaslException: GSS initiate failed
> [Caused by GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)] 2020-09-22 12:18:37,197 ERROR
> [hconnection-0x55d9d8d1-shared--pool3-t769]
> o.a.hadoop.hbase.ipc.AbstractRpcClient SASL authentication failed. The most
> likely cause is missing or invalid credentials. Consider 'kinit'.
> javax.security.sasl.SaslException: GSS initiate failed at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:612)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$600(RpcClientImpl.java:157)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:738)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:735)
> at java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:735)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:897)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:866)
> at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1208)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:223)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:328)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.multi(ClientProtos.java:32879)
> at
> org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:128)
> at
> org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:53)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:723)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) Caused by:
> org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)
> {code}
>
> Environment: Apache NIFI 1.12, RHEL 7.7, openjdk version "1.8.0_222-ea"
> Regards!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
