Jeff Storck created NIFI-3473:
---------------------------------
Summary: PutHDFS generates no bulletin for a connection timeout
error
Key: NIFI-3473
URL: https://issues.apache.org/jira/browse/NIFI-3473
Project: Apache NiFi
Issue Type: Bug
Components: Extensions
Affects Versions: 1.1.1
Reporter: Jeff Storck
Priority: Minor
When connection to an HDFS cluster is lost, PutHDFS will be unable to send
files to HDFS, and the files will be put back into the incoming queue, but no
bulletin is generated when the timeout exceptions occur.
{noformat}2017-02-13 14:26:18,683 INFO [Timer-Driven Process Thread-7]
o.a.h.io.retry.RetryInvocationHandler Exception while invoking getFileInfo of
class ClientNamenodeProtocolTranslatorPB over [host/ip:port] after 8 fail over
attempts. Trying to fail over after sleeping for 22257ms.
org.apache.hadoop.net.ConnectTimeoutException: Call From [host/ip] to
[host:port] failed on socket timeout exception:
org.apache.hadoop.net.ConnectTimeoutException: 20000 millis timeout while
waiting for channel to be ready for connect. ch :
java.nio.channels.SocketChannel[connection-pending remote=host/ip:port]; For
more details see: http://wiki.apache.org/hadoop/SocketTimeout
at sun.reflect.GeneratedConstructorAccessor380.newInstance(Unknown
Source) ~[na:na]
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
~[na:1.8.0_102]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
~[na:1.8.0_102]
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
~[hadoop-common-2.7.3.jar:na]
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:751)
~[hadoop-common-2.7.3.jar:na]
at org.apache.hadoop.ipc.Client.call(Client.java:1479)
~[hadoop-common-2.7.3.jar:na]
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
~[hadoop-common-2.7.3.jar:na]
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
~[hadoop-common-2.7.3.jar:na]
at com.sun.proxy.$Proxy136.getFileInfo(Unknown Source) ~[na:na]
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
~[hadoop-hdfs-2.7.3.jar:na]
at sun.reflect.GeneratedMethodAccessor386.invoke(Unknown Source)
~[na:na]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.8.0_102]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_102]
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
~[hadoop-common-2.7.3.jar:na]
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
~[hadoop-common-2.7.3.jar:na]
at com.sun.proxy.$Proxy137.getFileInfo(Unknown Source) [na:na]
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108)
[hadoop-hdfs-2.7.3.jar:na]
at
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
[hadoop-hdfs-2.7.3.jar:na]
at
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
[hadoop-hdfs-2.7.3.jar:na]
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
[hadoop-common-2.7.3.jar:na]
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
[hadoop-hdfs-2.7.3.jar:na]
at org.apache.nifi.processors.hadoop.PutHDFS$1.run(PutHDFS.java:262)
[nifi-hdfs-processors-1.1.1.jar:1.1.1]
at java.security.AccessController.doPrivileged(Native Method)
[na:1.8.0_102]
at javax.security.auth.Subject.doAs(Subject.java:360) [na:1.8.0_102]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1678)
[hadoop-common-2.7.3.jar:na]
at
org.apache.nifi.processors.hadoop.PutHDFS.onTrigger(PutHDFS.java:230)
[nifi-hdfs-processors-1.1.1.jar:1.1.1]
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
[nifi-api-1.1.1.jar:1.1.1]
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099)
[nifi-framework-core-1.1.1.jar:1.1.1]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
[nifi-framework-core-1.1.1.jar:1.1.1]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
[nifi-framework-core-1.1.1.jar:1.1.1]
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
[nifi-framework-core-1.1.1.jar:1.1.1]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_102]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_102]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_102]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_102]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_102]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_102]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
Caused by: org.apache.hadoop.net.ConnectTimeoutException: 20000 millis timeout
while waiting for channel to be ready for connect. ch :
java.nio.channels.SocketChannel[connection-pending remote=host/ip:port]
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:534)
~[hadoop-common-2.7.3.jar:na]
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
~[hadoop-common-2.7.3.jar:na]
at
org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:614)
~[hadoop-common-2.7.3.jar:na]
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:712)
~[hadoop-common-2.7.3.jar:na]
at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375)
~[hadoop-common-2.7.3.jar:na]
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1528)
~[hadoop-common-2.7.3.jar:na]
at org.apache.hadoop.ipc.Client.call(Client.java:1451)
~[hadoop-common-2.7.3.jar:na]
... 32 common frames omitted{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
