Hi, Could you the full log on pastebin with us?
1) That error message is bad, setting it to 0 waits 0 time, not disabling it. https://issues.apache.org/jira/browse/FLUME-1599 2) Do you know how full the channel is during the error? Or how many events per second are going through? Brock On Fri, Sep 21, 2012 at 12:25 AM, Jagadish Bihani <[email protected]> wrote: > Hi > > In my flume agent with "Avro source-File Channel-HDFS sink"; I am getting > following exception: > > java.io.IOException: Failed to obtain lock for writing to the log. Try > increasing the log write timeout value or disabling it by setting it to 0. > [channel=fileChannel] > > when I set my file roll interval to some high value (say 600 sec). It works > fine for smaller values like (180 sec). > I have also tried reducing batch size. I have tried setting both > "keep-alive" and "write-timeout" values of the channel > to 0 once and also to high value like 100 other time. But in both the cases > same exception appeared. > > Regards, > Jagadish > > This is my configuration: > ============================================= > agent.sources = avro-collection-source > agent.channels = fileChannel > agent.sinks = hdfsSink fileSink > > # For each one of the sources, the type is defined > agent.sources.avro-collection-source.type=avro > agent.sources.avro-collection-source.bind=10.0.17.3 > agent.sources.avro-collection-source.port=10011 > agent.sources.avro-collection-source.interceptors = ts > agent.sources.avro-collection-source.interceptors.ts.type = timestamp > # The channel can be defined as follows. > agent.sources.avro-collection-source.channels = fileChannel > > # Each sink's type must be defined > agent.sinks.hdfsSink.type = hdfs > #agent.sinks.hdfsSink.hdfs.path= > hdfs://mltest2001.pubmatic.com/flume/experiments > agent.sinks.hdfsSink.hdfs.path=hdfs://mltest2001.pubmatic.com/flume/tracker3 > > #agent.sinks.hdfsSink.hdfs.fileType =DataStream > agent.sinks.hdfsSink.hdfs.fileType =CompressedStream > agent.sinks.hdfsSink.hdfs.filePrefix=adtrack_backup_sizeroll_%Y%m%d_%H%M%S_ > #agent.sinks.hdfsSink.hdfs.filePrefix=adtrack_backup_ > agent.sinks.hdfsSink.hdfs.rollSize=0 > agent.sinks.hdfsSink.hdfs.codeC=bzip2 > agent.sinks.hdfsSink.hdfs.rollCount=0 > agent.sinks.hdfsSink.hdfs.batchSize=1000 > #agent.sinks.hdfsSink.hdfs.writeFormat=Text > agent.sinks.hdfsSink.hdfs.rollInterval=600 > > > #Define file sink > agent.sinks.fileSink.type = file_roll > agent.sinks.fileSink.sink.directory = /root/flume_sink > > #Specify the channel the sink should use > agent.sinks.hdfsSink.channel= fileChannel > agent.channels.fileChannel.type=file > agent.channels.fileChannel.dataDirs=/root/flume_channel/dataDir13 > agent.channels.fileChannel.checkpointDir=/root/flume_channel/checkpointDir13 > agent.channels.fileChannel.write-timeout=30 > -- Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
