Sergey created FLUME-2261:
-----------------------------
Summary: Flume HDFS sinks can't overcome NN HA switching
Key: FLUME-2261
URL: https://issues.apache.org/jira/browse/FLUME-2261
Project: Flume
Issue Type: Bug
Components: Sinks+Sources
Affects Versions: v1.4.0
Environment: CDH 4.5 1.4.0+23
Reporter: Sergey
looks like related to #FLUME-2228
(https://issues.apache.org/jira/browse/FLUME-2228)
We have 3-node cluster with NN HA. Sometimes nodes are switching.
And Flume HDFS sink can't correctly handle such situation.
Here is the log:
{code}
9 Dec 2013 22:14:49,175 INFO [pool-6-thread-1]
(org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream:171) -
[id: 0x1b2ae35d, /176.9.1.174:37697 :> /88.198.23.238:60011] DISCONNECTED
09 Dec 2013 22:14:49,175 INFO [pool-6-thread-1]
(org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream:171) -
[id: 0x1b2ae35d, /176.9.1.174:37697 :> /88.198.23.238:60011] UNBOUND
09 Dec 2013 22:14:49,175 INFO [pool-6-thread-1]
(org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream:171) -
[id: 0x1b2ae35d, /176.9.1.174:37697 :> /88.198.23.238:60011] CLOSED
09 Dec 2013 22:14:49,175 INFO [pool-6-thread-1]
(org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.channelClosed:209) -
Connection to /176.9.1.174:37697 disconnected.
09 Dec 2013 22:14:49,956 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.flume.sink.hdfs.AbstractHDFSWriter.isUnderReplicated:96) -
Unexpected error while checking replication factor
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.flume.sink.hdfs.AbstractHDFSWriter.getNumCurrentReplicas(AbstractHDFSWriter.java:162)
at
org.apache.flume.sink.hdfs.AbstractHDFSWriter.isUnderReplicated(AbstractHDFSWriter.java:82)
at
org.apache.flume.sink.hdfs.BucketWriter.shouldRotate(BucketWriter.java:452)
at org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:387)
at
org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:392)
at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Failed to add a datanode. User may turn off
this feature by setting
dfs.client.block.write.replace-datanode-on-failure.policy in configuration,
where the current policy is DEFAULT. (Nodes: current=[88.198.23.238:50010,
176.9.1.174:50010], original=[88.198.23.238:50010, 176.9.1.174:50010])
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:817)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:877)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:983)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:780)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449)
09 Dec 2013 22:14:49,956 WARN [SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.flume.sink.hdfs.BucketWriter.append:424) - Caught IOException
writing to HDFSWriter (Failed to add a datanode. User may turn off this
feature by setting dfs.client.block.write.replace-datanode-on-failure.policy in
configuration, where the current policy is DEFAULT. (Nodes:
current=[88.198.23.238:50010, 176.9.1.174:50010],
original=[88.198.23.238:50010, 176.9.1.174:50010])). Closing file
(/staging/landing/stream/js_tracker/visit/2013/12/09/07/visit.1386560349521.bz2.tmp)
and rethrowing exception.
09 Dec 2013 22:14:49,957 WARN [SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.flume.sink.hdfs.BucketWriter.append:430) - Caught IOException
while closing file
(/staging/landing/stream/js_tracker/visit/2013/12/09/07/visit.1386560349521.bz2.tmp).
Exception follows.
java.io.IOException: Failed to add a datanode. User may turn off this feature
by setting dfs.client.block.write.replace-datanode-on-failure.policy in
configuration, where the current policy is DEFAULT. (Nodes:
current=[88.198.23.238:50010, 176.9.1.174:50010],
original=[88.198.23.238:50010, 176.9.1.174:50010])
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:817)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:877)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:983)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:780)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449)
09 Dec 2013 22:14:49,957 WARN [SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.flume.sink.hdfs.HDFSEventSink.process:418) - HDFS IO error
java.io.IOException: Failed to add a datanode. User may turn off this feature
by setting dfs.client.block.write.replace-datanode-on-failure.policy in
configuration, where the current policy is DEFAULT. (Nodes:
current=[88.198.23.238:50010, 176.9.1.174:50010],
original=[88.198.23.238:50010, 176.9.1.174:50010])
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:817)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:877)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:983)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:780)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449)
09 Dec 2013 22:14:54,957 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.flume.sink.hdfs.AbstractHDFSWriter.isUnderReplicated:96) -
Unexpected error while checking replication factor
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.flume.sink.hdfs.AbstractHDFSWriter.getNumCurrentReplicas(AbstractHDFSWriter.java:162)
at
org.apache.flume.sink.hdfs.AbstractHDFSWriter.isUnderReplicated(AbstractHDFSWriter.java:82)
at
org.apache.flume.sink.hdfs.BucketWriter.shouldRotate(BucketWriter.java:452)
at org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:387)
at
org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:392)
at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Failed to add a datanode. User may turn off
this feature by setting
dfs.client.block.write.replace-datanode-on-failure.policy in configuration,
where the current policy is DEFAULT. (Nodes: current=[88.198.23.238:50010,
176.9.1.174:50010], original=[88.198.23.238:50010, 176.9.1.174:50010])
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:817)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:877)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:983)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:780)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449)
09 Dec 2013 22:14:54,958 WARN [SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.flume.sink.hdfs.BucketWriter.append:424) - Caught IOException
writing to HDFSWriter (Failed to add a datanode. User may turn off this
feature by setting dfs.client.block.write.replace-datanode-on-failure.policy in
configuration, where the current policy is DEFAULT. (Nodes:
current=[88.198.23.238:50010, 176.9.1.174:50010],
original=[88.198.23.238:50010, 176.9.1.174:50010])). Closing file
(/staging/landing/stream/js_tracker/visit/2013/12/09/07/visit.1386560349521.bz2.tmp)
and rethrowing exception.
09 Dec 2013 22:14:54,958 WARN [SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.flume.sink.hdfs.BucketWriter.append:430) - Caught IOException
while closing file (/staging/landing/strea
{code}
Flume stops to write data to HDFS and can't close opened file. It's a disaster
because we don't have any notification (is it possible to get any)?
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
