I am getting the same error too. Thanks,Himanshu
Date: Wed, 26 Mar 2014 11:20:10 +0800 Subject: Re: HDFS IO Error From: [email protected] To: [email protected] we got the same problem!. On Wed, Mar 26, 2014 at 2:43 AM, Abraham Fine <[email protected]> wrote: Hello- We have Flume agents running 1.4.0 that sink to HDFS (version 2.0.0-cdh4.2.1). Exceptions start occurring at the same time across our Flume agentswhen a datanode in HDFS goes down. We did not have this issue whilerunning Flume 1.3. We noticed a similar issue posted on the mailing list herehttp://mail-archives.apache.org/mod_mbox/flume-user/201307.mbox/%3ccapzq-vkmdgptboweaf+re-1neibutq36+ehqukn5b7fum4q...@mail.gmail.com%3E <http://mail-archives.apache.org/mod_mbox/flume-user/201307.mbox/%3ccapzq-vkmdgptboweaf+re-1neibutq36+ehqukn5b7fum4q...@mail.gmail.com%3E>and on JIRAhttps://issues.apache.org/jira/browse/FLUME-2261 <https://issues.apache.org/jira/browse/FLUME-2261>but couldnot find a solution. We have noticed the following in the Flume logs: WARN [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hdfs.HDFSEventSink.process:418) - HDFS IO error java.io.IOException: Callable timed out after 20000 ms on file <FILEPATH> : at org.apache.flume.sink.hdfs.BucketWriter.callWithTimeout(BucketWriter.java:550) at org.apache.flume.sink.hdfs.BucketWriter.doFlush(BucketWriter.java:353) at org.apache.flume.sink.hdfs.BucketWriter.flush(BucketWriter.java:319) at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:405) at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.TimeoutException at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:228) at java.util.concurrent.FutureTask.get(FutureTask.java:91) at org.apache.flume.sink.hdfs.BucketWriter.callWithTimeout(BucketWriter.java:543) ... 6 more This is usually followed by: WARN [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hdfs.HDFSEventSink.process:418) - HDFS IO error java.io.IOException: This bucket writer was closed due to idling and this handle is thus no longer valid at org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:380) at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:392) at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) at java.lang.Thread.run(Thread.java:662) When these exceptions occur, the HDFS sink does not close files. Weoften end up with multi-gigabyte files in HDFS. Our sink configuration: agentX.sinks.hdfs-sinkX-1.channel = chX agentX.sinks.hdfs-sinkX-1.type = hdfs agentX.sinks.hdfs-sinkX-1.hdfs.path = <FILEPATH> agentX.sinks.hdfs-sinkX-1.hdfs.filePrefix = event agentX.sinks.hdfs-sinkX-1.hdfs.writeFormat = Text agentX.sinks.hdfs-sinkX-1.hdfs.rollInterval = 120 agentX.sinks.hdfs-sinkX-1.hdfs.idleTimeout= 180 agentX.sinks.hdfs-sinkX-1.hdfs.rollCount = 0 agentX.sinks.hdfs-sinkX-1.hdfs.rollSize = 0 agentX.sinks.hdfs-sinkX-1.hdfs.fileType = DataStream agentX.sinks.hdfs-sinkX-1.hdfs.batchSize = 24000 agentX.sinks.hdfs-sinkX-1.hdfs.txnEventSize = 24000 agentX.sinks.hdfs-sinkX-1.hdfs.callTimeout = 20000 agentX.sinks.hdfs-sinkX-1.hdfs.threadsPoolSize = 1 The file paths are unique to each sink. Thank you for your help. -- Abraham Fine | Software Engineer BrightRoll, Inc. | Smart Video Advertising |www.brightroll.com <http://www.brightroll.com/>
