Jeff Storck created NIFI-3472:
---------------------------------
Summary: PutHDFS Kerberos relogin not working (tgt) after ticket
expires
Key: NIFI-3472
URL: https://issues.apache.org/jira/browse/NIFI-3472
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 1.1.1
Reporter: Jeff Storck
PutHDFS is not able to relogin if the ticket expires.
NiFi, running locally as standalone, was sending files to HDFS. After
suspending the system for the weekend, when the flow attempted to continue to
processo flowfiles, the following exception occurred:
{code}2017-02-13 11:59:53,460 WARN [Timer-Driven Process Thread-10]
org.apache.hadoop.ipc.Client Exception encountered while connecting to the
server : javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)]
2017-02-13 11:59:53,463 INFO [Timer-Driven Process Thread-10]
o.a.h.io.retry.RetryInvocationHandler Exception while invoking getFileInfo of
class ClientNamenodeProtocolTranslatorPB over
hdp-cluster-2-1.novalocal/172.22.105.152:8020 after 3 fail over attempts.
Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Failed to find any Kerberos
tgt)]; Host Details : local host is: "hw13184.local/10.0.0.153"; destination
host is: "hdp-cluster-2-1.novalocal":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
~[hadoop-common-2.7.3.jar:na]
at org.apache.hadoop.ipc.Client.call(Client.java:1479)
~[hadoop-common-2.7.3.jar:na]
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
~[hadoop-common-2.7.3.jar:na]
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
~[hadoop-common-2.7.3.jar:na]
at com.sun.proxy.$Proxy136.getFileInfo(Unknown Source) ~[na:na]
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
~[hadoop-hdfs-2.7.3.jar:na]
at sun.reflect.GeneratedMethodAccessor386.invoke(Unknown Source)
~[na:na]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.8.0_102]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_102]
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
~[hadoop-common-2.7.3.jar:na]
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
~[hadoop-common-2.7.3.jar:na]
at com.sun.proxy.$Proxy137.getFileInfo(Unknown Source) [na:na]
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108)
[hadoop-hdfs-2.7.3.jar:na]
at
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
[hadoop-hdfs-2.7.3.jar:na]
at
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
[hadoop-hdfs-2.7.3.jar:na]
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
[hadoop-common-2.7.3.jar:na]
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
[hadoop-hdfs-2.7.3.jar:na]
at org.apache.nifi.processors.hadoop.PutHDFS$1.run(PutHDFS.java:262)
[nifi-hdfs-processors-1.1.1.jar:1.1.1]
at java.security.AccessController.doPrivileged(Native Method)
[na:1.8.0_102]
at javax.security.auth.Subject.doAs(Subject.java:360) [na:1.8.0_102]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1678)
[hadoop-common-2.7.3.jar:na]
at
org.apache.nifi.processors.hadoop.PutHDFS.onTrigger(PutHDFS.java:230)
[nifi-hdfs-processors-1.1.1.jar:1.1.1]
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
[nifi-api-1.1.1.jar:1.1.1]
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099)
[nifi-framework-core-1.1.1.jar:1.1.1]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
[nifi-framework-core-1.1.1.jar:1.1.1]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
[nifi-framework-core-1.1.1.jar:1.1.1]
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
[nifi-framework-core-1.1.1.jar:1.1.1]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_102]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_102]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_102]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_102]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_102]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_102]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]{code}
After stopping and starting the PutHDFS processor, flowfiles were able to be
transferred to HDFS.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
