Hi;
I have deplyed two flume, each on a cluster. Version is 1.5.2
One will collect tail of a log file then send to another
Flume 1 CONFIG:
a2.sources = r1
a2.sinks = k1
a2.channels = c1
a2.sources.r1.type = exec
a2.sources.r1.command = tail -F /var/game_log/realtime_online
a2.sinks.k1.type = avro
a2.sinks.k1.hostname = 192.168.1.133
a2.sinks.k1.port = 41414
# Use a channel which buffers events in memory
a2.channels.c1.type = memory
a2.channels.c1.capacity = 1000
a2.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a2.sources.r1.channels = c1
a2.sinks.k1.channel = c1
Flume 2 Config:
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = 192.168.1.133
a1.sources.r1.port = 31310
# Describe the sink
#a1.sinks.k1.type = logger
a1.sinks.k1.type = file_roll
a1.sinks.k1.sink.rollInterval = 3600
a1.sinks.k1.sink.directory = /home/noyce/flume/log
a1.sinks.k1.filePrefix = %{basename}
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
------------------------------------------------------------------------------------------------------------------------------------------------------------------
When the cluster hold Flume1 reboot, I restart flume1, then got ERROR as
below:
2015-04-21 23:38:32,930 (SinkRunner-PollingRunner-DefaultSinkProcessor)
[WARN -
org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:634)]
Using default maxIOWorkers
2015-04-21 23:38:38,199 (agent-shutdown-hook) [INFO -
org.apache.flume.lifecycle.LifecycleSupervisor.stop(LifecycleSupervisor.java:79)]
Stopping lifecycle supervisor 10
2015-04-21 23:38:38,224 (SinkRunner-PollingRunner-DefaultSinkProcessor)
[ERROR -
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)] Unable
to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to send events
at
org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:392)
at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient {
host: 192.168.1.133, port: 41414 }: Failed to send batch
at
org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:315)
at
org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:376)
... 3 more
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient {
host: 192.168.1.133, port: 41414 }: Interrupted in handshake
at
org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:362)
at
org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:303)
... 4 more
Caused by: java.lang.InterruptedException
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:400)
Everytime the cluster reboot then I have to restart all flume1 and flume2?
any solution to failover these scene???
Thanks
Best Regards
Yours
Meng
