Thank you very much Mark!! It works perfectly. Thanks & Regards, Ashutosh Sharma
-----Original Message----- From: Stern, Mark [mailto:[email protected]] Sent: Thursday, August 16, 2012 3:15 PM To: ashutosh(오픈플랫폼개발팀) Cc: [email protected] Subject: RE: Sink Processor Hi Sharma, If you want to use a failover mechanism, you should connect both sinks in your failover group to a single channel. So I suggest you remove mem-channel1. agent.channels = mem-channel mem-channel1 becomes agent.channels = mem-channel agent.sources.tail-source.channels = mem-channel mem-channel1 becomes agent.sources.tail-source.channels = mem-channel Delete agent.sources.tail-source.selector.type = replicating agent.sinks.avro-forward-sink2.channel = mem-channel1 becomes agent.sinks.avro-forward-sink2.channel = mem-channel and you can also delete the last 3 lines that define mem-channel1. Hope this helps, Mark On Thu, 2012-08-16 at 01:23 +0000, ashutosh(오픈플랫폼개발팀) wrote: > Hi Mark, > > Below is my configuration file. Please suggest if there is any mistake. > > ## Name the components on this agent > agent.sources = tail-source > agent.sinks = avro-forward-sink1 avro-forward-sink2 agent.channels = > mem-channel mem-channel1 agent.sinkgroups = failover-sink-group > > ## Describe the tail source > agent.sources.tail-source.type = exec > agent.sources.tail-source.command = tail -F /home/ashutosh/test.txt > agent.sources.tail-source.channels = mem-channel mem-channel1 > agent.sources.tail-source.selector.type = replicating > agent.sources.tail-source.interceptors = host t-stamp > > ## Describe Host interceptor > agent.sources.tail-source.interceptors.host.type = > org.apache.flume.interceptor.HostInterceptor$Builder > agent.sources.tail-source.interceptors.host.preserveExisting = false > agent.sources.tail-source.interceptors.host.hostHeader = hostname > agent.sources.tail-source.interceptors.host.useIP = false > > ## Describe Timestamp interceptor > agent.sources.tail-source.interceptors.t-stamp.type = > org.apache.flume.interceptor.TimestampInterceptor$Builder > > ## Describe the avro sink to forward the events to down the line > agent1(consumer2) agent.sinks.avro-forward-sink1.type = avro > agent.sinks.avro-forward-sink1.channel = mem-channel > agent.sinks.avro-forward-sink1.hostname = 192.168.32.27 > agent.sinks.avro-forward-sink1.port = 61001 > agent.sinks.avro-forward-sink1.batch-size = 20 > > ## Describe the avro sink to forward the events to down the line > agent2(consumer1) agent.sinks.avro-forward-sink2.type = avro > agent.sinks.avro-forward-sink2.channel = mem-channel1 > agent.sinks.avro-forward-sink2.hostname = 192.168.32.26 > agent.sinks.avro-forward-sink2.port = 61001 > agent.sinks.avro-forward-sink2.batch-size = 20 > > ## Describe the sink processor for failover. Consumer1 takes charges if > consumer2 fails. > agent.sinkgroups.failover-sink-group.processor.type = failover > agent.sinkgroups.failover-sink-group.sinks = avro-forward-sink1 > avro-forward-sink2 > agent.sinkgroups.failover-sink-group.processor.priority.avro-forward-s > ink1 = 10 > agent.sinkgroups.failover-sink-group.processor.priority.avro-forward-s > ink2 = 5 > > ## Define the memory channel > agent.channels.mem-channel.type = memory > agent.channels.mem-channel.capacity = 1000 > agent.channels.mem-channel.transactionCapacity = 20 > > ## Define the file channel > agent.channels.mem-channel1.type = memory > agent.channels.mem-channel1.capacity = 1000 > agent.channels.mem-channel1.transactionCapacity = 20 > > Thanks & Regards, > Ashutosh Sharma > > > > -----Original Message----- > From: Stern, Mark [mailto:[email protected]] > Sent: Tuesday, August 14, 2012 8:45 PM > To: [email protected] > Subject: RE: Sink Processor > > This sounds like a configuration error. Can you provide your configuration > file? > ________________________________________ > From: ashutosh(오픈플랫폼개발팀) [[email protected]] > Sent: Tuesday, August 14, 2012 12:07 PM > To: [email protected] > Subject: Sink Processor > > Hi All, > > My query is related to the flume’s sink processor for the failover. I > implemented the failover data flow to transfer the event to downstream in > case of agent failure. > > I have configured the flume’s sinkgroups for failover of the sinks. In case > of the failure of sink1, sink2 take charge of storing data into hdfs. > I tested the functionality of the sinkgroups’s failover successfully. When > sink1 fails, sink2 successfully takes the charge and transfer the data to the > destination. > > Here I am transferring the events by tailing a file contains 100 records. I > shutdown the sink1 when it has processed 40 records.. After that all data > start transferred by sink2. But, the file saved in hdfs contains 140 records. > It means both downstream agents hold the events, if one fails another start > transfer the events from the start as it hold all data in its defined channel. > > Is this the intended behavior of the flume’s sinkgroup in case of failover? > How can I avoid the duplicate events? Is there any solution as part of > flume’s sink processor to deal with the duplicates? > > Thanks & Regards, > Ashutosh > > > > 이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에 포함된 > 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된 > 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다. > This E-mail may contain confidential information and/or copyright material. > This email is intended for the use of the addressee only. If you receive this > email by mistake, please either delete it without reproducing, distributing > or retaining copies thereof or notify the sender immediately. > > > > 이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에 포함된 > 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된 > 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다. > This E-mail may contain confidential information and/or copyright material. > This email is intended for the use of the addressee only. If you receive this > email by mistake, please either delete it without reproducing, distributing > or retaining copies thereof or notify the sender immediately. 이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에 포함된 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다. This E-mail may contain confidential information and/or copyright material. This email is intended for the use of the addressee only. If you receive this email by mistake, please either delete it without reproducing, distributing or retaining copies thereof or notify the sender immediately.
