Hi
In my flume agent with "Avro source-File Channel-HDFS sink"; I am
getting following exception:
*java.io.IOException: Failed to obtain lock for writing to the log. Try
increasing the log write timeout value or disabling it by setting it to
0. [channel=fileChannel]**
*
when I set my file roll interval to some high value (say 600 sec). It
works fine for smaller values like (180 sec).
I have also tried reducing batch size. I have tried setting both
"keep-alive" and "write-timeout" values of the channel
to 0 once and also to high value like 100 other time. But in both the
cases same exception appeared.
Regards,
Jagadish
This is my configuration:
=============================================
agent.sources = avro-collection-source
agent.channels = fileChannel
agent.sinks = hdfsSink fileSink
# For each one of the sources, the type is defined
agent.sources.avro-collection-source.type=avro
agent.sources.avro-collection-source.bind=10.0.17.3
agent.sources.avro-collection-source.port=10011
agent.sources.avro-collection-source.interceptors = ts
agent.sources.avro-collection-source.interceptors.ts.type = timestamp
# The channel can be defined as follows.
agent.sources.avro-collection-source.channels = fileChannel
# Each sink's type must be defined
agent.sinks.hdfsSink.type = hdfs
#agent.sinks.hdfsSink.hdfs.path=
hdfs://mltest2001.pubmatic.com/flume/experiments
agent.sinks.hdfsSink.hdfs.path=hdfs://mltest2001.pubmatic.com/flume/tracker3
#agent.sinks.hdfsSink.hdfs.fileType =DataStream
agent.sinks.hdfsSink.hdfs.fileType =CompressedStream
agent.sinks.hdfsSink.hdfs.filePrefix=adtrack_backup_sizeroll_%Y%m%d_%H%M%S_
#agent.sinks.hdfsSink.hdfs.filePrefix=adtrack_backup_
agent.sinks.hdfsSink.hdfs.rollSize=0
agent.sinks.hdfsSink.hdfs.codeC=bzip2
agent.sinks.hdfsSink.hdfs.rollCount=0
agent.sinks.hdfsSink.hdfs.batchSize=1000
#agent.sinks.hdfsSink.hdfs.writeFormat=Text
agent.sinks.hdfsSink.hdfs.rollInterval=600
#Define file sink
agent.sinks.fileSink.type = file_roll
agent.sinks.fileSink.sink.directory = /root/flume_sink
#Specify the channel the sink should use
agent.sinks.hdfsSink.channel= fileChannel
agent.channels.fileChannel.type=file
agent.channels.fileChannel.dataDirs=/root/flume_channel/dataDir13
agent.channels.fileChannel.checkpointDir=/root/flume_channel/checkpointDir13
agent.channels.fileChannel.write-timeout=30
