Hi Sharma, If you want to use a failover mechanism, you should connect both sinks in your failover group to a single channel. So I suggest you remove mem-channel1.
agent.channels = mem-channel mem-channel1 becomes agent.channels = mem-channel agent.sources.tail-source.channels = mem-channel mem-channel1 becomes agent.sources.tail-source.channels = mem-channel Delete agent.sources.tail-source.selector.type = replicating agent.sinks.avro-forward-sink2.channel = mem-channel1 becomes agent.sinks.avro-forward-sink2.channel = mem-channel and you can also delete the last 3 lines that define mem-channel1. Hope this helps, Mark On Thu, 2012-08-16 at 01:23 +0000, ashutosh(오픈플랫폼개발팀) wrote: > Hi Mark, > > Below is my configuration file. Please suggest if there is any mistake. > > ## Name the components on this agent > agent.sources = tail-source > agent.sinks = avro-forward-sink1 avro-forward-sink2 > agent.channels = mem-channel mem-channel1 > agent.sinkgroups = failover-sink-group > > ## Describe the tail source > agent.sources.tail-source.type = exec > agent.sources.tail-source.command = tail -F /home/ashutosh/test.txt > agent.sources.tail-source.channels = mem-channel mem-channel1 > agent.sources.tail-source.selector.type = replicating > agent.sources.tail-source.interceptors = host t-stamp > > ## Describe Host interceptor > agent.sources.tail-source.interceptors.host.type = > org.apache.flume.interceptor.HostInterceptor$Builder > agent.sources.tail-source.interceptors.host.preserveExisting = false > agent.sources.tail-source.interceptors.host.hostHeader = hostname > agent.sources.tail-source.interceptors.host.useIP = false > > ## Describe Timestamp interceptor > agent.sources.tail-source.interceptors.t-stamp.type = > org.apache.flume.interceptor.TimestampInterceptor$Builder > > ## Describe the avro sink to forward the events to down the line > agent1(consumer2) > agent.sinks.avro-forward-sink1.type = avro > agent.sinks.avro-forward-sink1.channel = mem-channel > agent.sinks.avro-forward-sink1.hostname = 192.168.32.27 > agent.sinks.avro-forward-sink1.port = 61001 > agent.sinks.avro-forward-sink1.batch-size = 20 > > ## Describe the avro sink to forward the events to down the line > agent2(consumer1) > agent.sinks.avro-forward-sink2.type = avro > agent.sinks.avro-forward-sink2.channel = mem-channel1 > agent.sinks.avro-forward-sink2.hostname = 192.168.32.26 > agent.sinks.avro-forward-sink2.port = 61001 > agent.sinks.avro-forward-sink2.batch-size = 20 > > ## Describe the sink processor for failover. Consumer1 takes charges if > consumer2 fails. > agent.sinkgroups.failover-sink-group.processor.type = failover > agent.sinkgroups.failover-sink-group.sinks = avro-forward-sink1 > avro-forward-sink2 > agent.sinkgroups.failover-sink-group.processor.priority.avro-forward-sink1 = > 10 > agent.sinkgroups.failover-sink-group.processor.priority.avro-forward-sink2 = 5 > > ## Define the memory channel > agent.channels.mem-channel.type = memory > agent.channels.mem-channel.capacity = 1000 > agent.channels.mem-channel.transactionCapacity = 20 > > ## Define the file channel > agent.channels.mem-channel1.type = memory > agent.channels.mem-channel1.capacity = 1000 > agent.channels.mem-channel1.transactionCapacity = 20 > > Thanks & Regards, > Ashutosh Sharma > > > > -----Original Message----- > From: Stern, Mark [mailto:[email protected]] > Sent: Tuesday, August 14, 2012 8:45 PM > To: [email protected] > Subject: RE: Sink Processor > > This sounds like a configuration error. Can you provide your configuration > file? > ________________________________________ > From: ashutosh(오픈플랫폼개발팀) [[email protected]] > Sent: Tuesday, August 14, 2012 12:07 PM > To: [email protected] > Subject: Sink Processor > > Hi All, > > My query is related to the flume’s sink processor for the failover. I > implemented the failover data flow to transfer the event to downstream in > case of agent failure. > > I have configured the flume’s sinkgroups for failover of the sinks. In case > of the failure of sink1, sink2 take charge of storing data into hdfs. > I tested the functionality of the sinkgroups’s failover successfully. When > sink1 fails, sink2 successfully takes the charge and transfer the data to the > destination. > > Here I am transferring the events by tailing a file contains 100 records. I > shutdown the sink1 when it has processed 40 records.. After that all data > start transferred by sink2. But, the file saved in hdfs contains 140 records. > It means both downstream agents hold the events, if one fails another start > transfer the events from the start as it hold all data in its defined channel. > > Is this the intended behavior of the flume’s sinkgroup in case of failover? > How can I avoid the duplicate events? Is there any solution as part of > flume’s sink processor to deal with the duplicates? > > Thanks & Regards, > Ashutosh > > > > 이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에 포함된 > 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된 > 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다. > This E-mail may contain confidential information and/or copyright material. > This email is intended for the use of the addressee only. If you receive this > email by mistake, please either delete it without reproducing, distributing > or retaining copies thereof or notify the sender immediately. > > > > 이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에 포함된 > 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된 > 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다. > This E-mail may contain confidential information and/or copyright material. > This email is intended for the use of the addressee only. If you receive this > email by mistake, please either delete it without reproducing, distributing > or retaining copies thereof or notify the sender immediately.
