[
https://issues.apache.org/jira/browse/NIFI-3472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16272821#comment-16272821
]
Jeff Storck commented on NIFI-3472:
-----------------------------------
I've reproduced the tgt expiration issue outside of NiFi in a small test
utility that I wrote. The Hadoop team is looking into this issue. There are a
few Hadoop JIRAs regarding the UGI class that are being worked on concurrently,
so we should see a fix for this issue once those JIRAs have been resolved.
[~jomach] After a quick look at the code, spawnAutoRenewalThreadForUserCreds is
meant for renewing credentials from the ticket cache, and the thread doesn't
get started if keytabs are being used. Since it's a private method, NiFi
wouldn't be able to make use of this explicitly, and we still need to find a
way to renew the TGT whether the ticket cache or keytabs are being used.
Hopefully, this will be transparent to NiFi. hadoop-client will implicitly
relogin if an operation is performed before expiration of the TGT, but won't
acquire another TGT if the current one has expired. Hopefully that'll be part
of the fix provided by the Hadoop team.
> Kerberos relogin not working (tgt) after ticket expires for HDFS/Hive/HBase
> processors
> --------------------------------------------------------------------------------------
>
> Key: NIFI-3472
> URL: https://issues.apache.org/jira/browse/NIFI-3472
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Affects Versions: 1.0.0, 1.1.0, 1.1.1, 1.0.1
> Reporter: Jeff Storck
> Assignee: Jeff Storck
>
> PutHDFS is not able to relogin if the ticket expires.
> NiFi, running locally as standalone, was sending files to HDFS. After
> suspending the system for the weekend, when the flow attempted to continue to
> process flowfiles, the following exception occurred:
> {code}2017-02-13 11:59:53,460 WARN [Timer-Driven Process Thread-10]
> org.apache.hadoop.ipc.Client Exception encountered while connecting to the
> server : javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> 2017-02-13 11:59:53,463 INFO [Timer-Driven Process Thread-10]
> o.a.h.io.retry.RetryInvocationHandler Exception while invoking getFileInfo of
> class ClientNamenodeProtocolTranslatorPB over [host:port] after 3 fail over
> attempts. Trying to fail over immediately.
> java.io.IOException: Failed on local exception: java.io.IOException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]; Host Details : local host is: "[host:port]"; destination
> host is: [host:port];
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
> ~[hadoop-common-2.7.3.jar:na]
> at org.apache.hadoop.ipc.Client.call(Client.java:1479)
> ~[hadoop-common-2.7.3.jar:na]
> at org.apache.hadoop.ipc.Client.call(Client.java:1412)
> ~[hadoop-common-2.7.3.jar:na]
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> ~[hadoop-common-2.7.3.jar:na]
> at com.sun.proxy.$Proxy136.getFileInfo(Unknown Source) ~[na:na]
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
> ~[hadoop-hdfs-2.7.3.jar:na]
> at sun.reflect.GeneratedMethodAccessor386.invoke(Unknown Source)
> ~[na:na]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_102]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_102]
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
> ~[hadoop-common-2.7.3.jar:na]
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> ~[hadoop-common-2.7.3.jar:na]
> at com.sun.proxy.$Proxy137.getFileInfo(Unknown Source) [na:na]
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108)
> [hadoop-hdfs-2.7.3.jar:na]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
> [hadoop-hdfs-2.7.3.jar:na]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> [hadoop-hdfs-2.7.3.jar:na]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> [hadoop-common-2.7.3.jar:na]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
> [hadoop-hdfs-2.7.3.jar:na]
> at org.apache.nifi.processors.hadoop.PutHDFS$1.run(PutHDFS.java:262)
> [nifi-hdfs-processors-1.1.1.jar:1.1.1]
> at java.security.AccessController.doPrivileged(Native Method)
> [na:1.8.0_102]
> at javax.security.auth.Subject.doAs(Subject.java:360) [na:1.8.0_102]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1678)
> [hadoop-common-2.7.3.jar:na]
> at
> org.apache.nifi.processors.hadoop.PutHDFS.onTrigger(PutHDFS.java:230)
> [nifi-hdfs-processors-1.1.1.jar:1.1.1]
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> [nifi-api-1.1.1.jar:1.1.1]
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099)
> [nifi-framework-core-1.1.1.jar:1.1.1]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
> [nifi-framework-core-1.1.1.jar:1.1.1]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> [nifi-framework-core-1.1.1.jar:1.1.1]
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
> [nifi-framework-core-1.1.1.jar:1.1.1]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_102]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [na:1.8.0_102]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_102]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [na:1.8.0_102]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_102]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_102]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]{code}
> After stopping and starting the PutHDFS processor, flowfiles were able to be
> transferred to HDFS.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
