[
https://issues.apache.org/jira/browse/FLUME-2519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183802#comment-14183802
]
Bijith Kumar commented on FLUME-2519:
-------------------------------------
Yes. I am setting TIMESTAMP and a couple of custom headers and it could be null
sometimes. I will change it to set as empty string when null. Thanks Hari.
Btw, when something like this happens, is there any way to recover it by
deleting just the invalid event?
> Avro source closes connection when unable to process an event batch
> -------------------------------------------------------------------
>
> Key: FLUME-2519
> URL: https://issues.apache.org/jira/browse/FLUME-2519
> Project: Flume
> Issue Type: Bug
> Affects Versions: v1.5.0
> Reporter: Bijith Kumar
>
> I have had this issue a few times now while sending log events from one agent
> to another (avro sink). The flume receiver fails to process an event and then
> closes the connection. The sender receives RPC call FAILED response and it
> retries the event and ends up in an infinite loop of "Failed to send events"
> error.
> Following are the logs from sender and receiver
> Sender
> ---------
> [SinkRunner-PollingRunner-LoadBalancingSinkProcessor]
> (org.apache.flume.sink.LoadBalancingSinkProcessor.process:158) - Sink failed
> to consume event. Attempting next sink if
> available.
> org.apache.flume.EventDeliveryException: Failed to send events
> at
> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:392)
> at
> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient {
> host: ec2-54-188-251-126.us-west-2.compute.amazonaws.com, port: 44445 }:
> Failed to send batch
> at
> org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:311)
> at
> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:376)
> ... 3 more
> Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient {
> host: ec2-54-188-251-126.us-west-2.compute.amazonaws.com, port: 44445 }: Avro
> RPC call returned Status: FAILED
> at
> org.apache.flume.api.NettyAvroRpcClient.waitForStatusOK(NettyAvroRpcClient.java:387)
> at
> org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:370)
> at
> org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:299)
> ... 4 more
> Receiver
> ------------
> 23 Oct 2014 12:25:22,595 ERROR [New I/O worker #30]
> (org.apache.flume.source.AvroSource.appendBatch:380) - Avro source
> avrosource: Unable to process event batch. Exception follows.
> org.apache.flume.ChannelException: Unable to put batch on required channel:
> FileChannel filechannel-invalid { dataDirs:
> [/mnt1/data/flume/filechannel-invalid/data] }
> at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
> at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:378)
> at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:786)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:458)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:439)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:311)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:553)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:84)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:471)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:332)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at
> org.apache.flume.channel.file.proto.ProtosFactory$FlumeEventHeader$Builder.setValue(ProtosFactory.java:7415)
> at org.apache.flume.channel.file.Put.writeProtos(Put.java:85)
> at
> org.apache.flume.channel.file.TransactionEventRecord.toByteBuffer(TransactionEventRecord.java:174)
> at org.apache.flume.channel.file.Log.put(Log.java:611)
> at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:458)
> at
> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> at
> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> ... 29 more
> 23 Oct 2014 12:25:22,679 INFO [New I/O worker #30]
> (org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream:171)
> - [id: 0x1b64c49f, /54.221.18.120:51345 :> /10.196.7.161:44445] DISCONNECT
> ED
> 23 Oct 2014 12:25:22,679 INFO [New I/O worker #30]
> (org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream:171)
> - [id: 0x1b64c49f, /54.221.18.120:51345 :> /10.196.7.161:44445] UNBOUND
> 23 Oct 2014 12:25:22,679 INFO [New I/O worker #30]
> (org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream:171)
> - [id: 0x1b64c49f, /54.221.18.120:51345 :> /10.196.7.161:44445] CLOSED
> 23 Oct 2014 12:25:22,679 INFO [New I/O worker #30]
> (org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.channelClosed:209) -
> Connection to /54.221.18.120:51345 disconnected.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
