Hello Team,
My development activity is stuck due to below error in the Apache agent.
org.apache.flume.ChannelException: Unable to put batch on required channel:
FileChannel file-channel-1
I have the following configurationOne Source - KafkaThree channels - File
channelFour Sinks - HDFS Sinks
Attached is the flume agent config file and below is the error message
Things tried1) Lowering or increasing the batch size for Kafka source and Hive
HDFS sink2) Lowering or increasing the TransactionCapacity and Capacity for
File channel3) With some help from the net, tried keeping File channel
TransactionCapacity to be 5x of Kafka and HDFS Batch size but in vain.
Please help advise
Error message in detail :
16/10/17 18:06:08 ERROR kafka.KafkaSource: KafkaSource EXCEPTION, {}
org.apache.flume.ChannelException: Unable to put batch on required channel:
FileChannel file-channel-1 { dataDirs:
[/home/dheeraj.rokade/flume/.flume/file-channel1/data] }
at
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
at
org.apache.flume.source.kafka.KafkaSource.process(KafkaSource.java:123)
at
org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:139)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at
org.apache.flume.channel.file.proto.ProtosFactory$FlumeEventHeader$Builder.setValue(ProtosFactory.java:7415)
at org.apache.flume.channel.file.Put.writeProtos(Put.java:85)
at
org.apache.flume.channel.file.TransactionEventRecord.toByteBuffer(TransactionEventRecord.java:174)
at org.apache.flume.channel.file.Log.put(Log.java:642)
at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:468)
at
org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
at
org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
at
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
... 3 more
16/10/17 18:06:13 ERROR kafka.KafkaSource: KafkaSource EXCEPTION, {}
org.apache.flume.ChannelException: Unable to put batch on required channel:
FileChannel file-channel-1 { dataDirs:
[/home/dheeraj.rokade/flume/.flume/file-channel1/data] }
at
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
at
org.apache.flume.source.kafka.KafkaSource.process(KafkaSource.java:123)
at
org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:139)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at
org.apache.flume.channel.file.proto.ProtosFactory$FlumeEventHeader$Builder.setValue(ProtosFactory.java:7415)
at org.apache.flume.channel.file.Put.writeProtos(Put.java:85)
at
org.apache.flume.channel.file.TransactionEventRecord.toByteBuffer(TransactionEventRecord.java:174)
at org.apache.flume.channel.file.Log.put(Log.java:642)
at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:468)
at
org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
at
org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
at
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
... 3 more
