[
https://issues.apache.org/jira/browse/FLUME-2261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843412#comment-13843412
]
Sergey commented on FLUME-2261:
-------------------------------
I didn't find any WARN/ERROR for the whole HDFS service on that time with huge
time overlap.
It just didn't recover. I don't know why.
Right now I did set these properties:
HDFS Client Configuration Safety Valve for hdfs-site.xml (related to client
i.e. HDFSSink)
{code}
<property>
<name>dfs.client.block.write.replace-datanode-on-failure.enable</name>
<value>true</value>
</property>
<property>
<name>dfs.client.block.write.replace-datanode-on-failure.policy</name>
<value>NEVER</value>
</property>
{code}
> Flume HDFS sinks can't overcome NN HA switching
> -----------------------------------------------
>
> Key: FLUME-2261
> URL: https://issues.apache.org/jira/browse/FLUME-2261
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.4.0
> Environment: CDH 4.5 1.4.0+23
> Reporter: Sergey
>
> looks like related to #FLUME-2228
> (https://issues.apache.org/jira/browse/FLUME-2228)
> We have 3-node cluster with NN HA. Sometimes nodes are switching.
> And Flume HDFS sink can't correctly handle such situation.
> Here is the log:
> {code}
> 9 Dec 2013 22:14:49,175 INFO [pool-6-thread-1]
> (org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream:171)
> - [id: 0x1b2ae35d, /176.9.1.174:37697 :> /88.198.23.238:60011] DISCONNECTED
> 09 Dec 2013 22:14:49,175 INFO [pool-6-thread-1]
> (org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream:171)
> - [id: 0x1b2ae35d, /176.9.1.174:37697 :> /88.198.23.238:60011] UNBOUND
> 09 Dec 2013 22:14:49,175 INFO [pool-6-thread-1]
> (org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream:171)
> - [id: 0x1b2ae35d, /176.9.1.174:37697 :> /88.198.23.238:60011] CLOSED
> 09 Dec 2013 22:14:49,175 INFO [pool-6-thread-1]
> (org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.channelClosed:209) -
> Connection to /176.9.1.174:37697 disconnected.
> 09 Dec 2013 22:14:49,956 ERROR
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.sink.hdfs.AbstractHDFSWriter.isUnderReplicated:96) -
> Unexpected error while checking replication factor
> java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.flume.sink.hdfs.AbstractHDFSWriter.getNumCurrentReplicas(AbstractHDFSWriter.java:162)
> at
> org.apache.flume.sink.hdfs.AbstractHDFSWriter.isUnderReplicated(AbstractHDFSWriter.java:82)
> at
> org.apache.flume.sink.hdfs.BucketWriter.shouldRotate(BucketWriter.java:452)
> at org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:387)
> at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:392)
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: Failed to add a datanode. User may turn off
> this feature by setting
> dfs.client.block.write.replace-datanode-on-failure.policy in configuration,
> where the current policy is DEFAULT. (Nodes: current=[88.198.23.238:50010,
> 176.9.1.174:50010], original=[88.198.23.238:50010, 176.9.1.174:50010])
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:817)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:877)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:983)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:780)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449)
> 09 Dec 2013 22:14:49,956 WARN
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.sink.hdfs.BucketWriter.append:424) - Caught IOException
> writing to HDFSWriter (Failed to add a datanode. User may turn off this
> feature by setting dfs.client.block.write.replace-datanode-on-failure.policy
> in configuration, where the current policy is DEFAULT. (Nodes:
> current=[88.198.23.238:50010, 176.9.1.174:50010],
> original=[88.198.23.238:50010, 176.9.1.174:50010])). Closing file
> (/staging/landing/stream/js_tracker/visit/2013/12/09/07/visit.1386560349521.bz2.tmp)
> and rethrowing exception.
> 09 Dec 2013 22:14:49,957 WARN
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.sink.hdfs.BucketWriter.append:430) - Caught IOException
> while closing file
> (/staging/landing/stream/js_tracker/visit/2013/12/09/07/visit.1386560349521.bz2.tmp).
> Exception follows.
> java.io.IOException: Failed to add a datanode. User may turn off this
> feature by setting dfs.client.block.write.replace-datanode-on-failure.policy
> in configuration, where the current policy is DEFAULT. (Nodes:
> current=[88.198.23.238:50010, 176.9.1.174:50010],
> original=[88.198.23.238:50010, 176.9.1.174:50010])
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:817)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:877)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:983)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:780)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449)
> 09 Dec 2013 22:14:49,957 WARN
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.sink.hdfs.HDFSEventSink.process:418) - HDFS IO error
> java.io.IOException: Failed to add a datanode. User may turn off this
> feature by setting dfs.client.block.write.replace-datanode-on-failure.policy
> in configuration, where the current policy is DEFAULT. (Nodes:
> current=[88.198.23.238:50010, 176.9.1.174:50010],
> original=[88.198.23.238:50010, 176.9.1.174:50010])
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:817)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:877)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:983)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:780)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449)
> 09 Dec 2013 22:14:54,957 ERROR
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.sink.hdfs.AbstractHDFSWriter.isUnderReplicated:96) -
> Unexpected error while checking replication factor
> java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.flume.sink.hdfs.AbstractHDFSWriter.getNumCurrentReplicas(AbstractHDFSWriter.java:162)
> at
> org.apache.flume.sink.hdfs.AbstractHDFSWriter.isUnderReplicated(AbstractHDFSWriter.java:82)
> at
> org.apache.flume.sink.hdfs.BucketWriter.shouldRotate(BucketWriter.java:452)
> at org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:387)
> at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:392)
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: Failed to add a datanode. User may turn off
> this feature by setting
> dfs.client.block.write.replace-datanode-on-failure.policy in configuration,
> where the current policy is DEFAULT. (Nodes: current=[88.198.23.238:50010,
> 176.9.1.174:50010], original=[88.198.23.238:50010, 176.9.1.174:50010])
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:817)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:877)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:983)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:780)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449)
> 09 Dec 2013 22:14:54,958 WARN
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.sink.hdfs.BucketWriter.append:424) - Caught IOException
> writing to HDFSWriter (Failed to add a datanode. User may turn off this
> feature by setting dfs.client.block.write.replace-datanode-on-failure.policy
> in configuration, where the current policy is DEFAULT. (Nodes:
> current=[88.198.23.238:50010, 176.9.1.174:50010],
> original=[88.198.23.238:50010, 176.9.1.174:50010])). Closing file
> (/staging/landing/stream/js_tracker/visit/2013/12/09/07/visit.1386560349521.bz2.tmp)
> and rethrowing exception.
> 09 Dec 2013 22:14:54,958 WARN
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.sink.hdfs.BucketWriter.append:430) - Caught IOException
> while closing file (/staging/landing/strea
> {code}
> Flume stops to write data to HDFS and can't close opened file. It's a
> disaster because we don't have any notification (is it possible to get any)?
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
