Hi Mark, Below is my configuration file. Please suggest if there is any mistake.
## Name the components on this agent agent.sources = tail-source agent.sinks = avro-forward-sink1 avro-forward-sink2 agent.channels = mem-channel mem-channel1 agent.sinkgroups = failover-sink-group ## Describe the tail source agent.sources.tail-source.type = exec agent.sources.tail-source.command = tail -F /home/ashutosh/test.txt agent.sources.tail-source.channels = mem-channel mem-channel1 agent.sources.tail-source.selector.type = replicating agent.sources.tail-source.interceptors = host t-stamp ## Describe Host interceptor agent.sources.tail-source.interceptors.host.type = org.apache.flume.interceptor.HostInterceptor$Builder agent.sources.tail-source.interceptors.host.preserveExisting = false agent.sources.tail-source.interceptors.host.hostHeader = hostname agent.sources.tail-source.interceptors.host.useIP = false ## Describe Timestamp interceptor agent.sources.tail-source.interceptors.t-stamp.type = org.apache.flume.interceptor.TimestampInterceptor$Builder ## Describe the avro sink to forward the events to down the line agent1(consumer2) agent.sinks.avro-forward-sink1.type = avro agent.sinks.avro-forward-sink1.channel = mem-channel agent.sinks.avro-forward-sink1.hostname = 192.168.32.27 agent.sinks.avro-forward-sink1.port = 61001 agent.sinks.avro-forward-sink1.batch-size = 20 ## Describe the avro sink to forward the events to down the line agent2(consumer1) agent.sinks.avro-forward-sink2.type = avro agent.sinks.avro-forward-sink2.channel = mem-channel1 agent.sinks.avro-forward-sink2.hostname = 192.168.32.26 agent.sinks.avro-forward-sink2.port = 61001 agent.sinks.avro-forward-sink2.batch-size = 20 ## Describe the sink processor for failover. Consumer1 takes charges if consumer2 fails. agent.sinkgroups.failover-sink-group.processor.type = failover agent.sinkgroups.failover-sink-group.sinks = avro-forward-sink1 avro-forward-sink2 agent.sinkgroups.failover-sink-group.processor.priority.avro-forward-sink1 = 10 agent.sinkgroups.failover-sink-group.processor.priority.avro-forward-sink2 = 5 ## Define the memory channel agent.channels.mem-channel.type = memory agent.channels.mem-channel.capacity = 1000 agent.channels.mem-channel.transactionCapacity = 20 ## Define the file channel agent.channels.mem-channel1.type = memory agent.channels.mem-channel1.capacity = 1000 agent.channels.mem-channel1.transactionCapacity = 20 Thanks & Regards, Ashutosh Sharma -----Original Message----- From: Stern, Mark [mailto:[email protected]] Sent: Tuesday, August 14, 2012 8:45 PM To: [email protected] Subject: RE: Sink Processor This sounds like a configuration error. Can you provide your configuration file? ________________________________________ From: ashutosh(오픈플랫폼개발팀) [[email protected]] Sent: Tuesday, August 14, 2012 12:07 PM To: [email protected] Subject: Sink Processor Hi All, My query is related to the flume’s sink processor for the failover. I implemented the failover data flow to transfer the event to downstream in case of agent failure. I have configured the flume’s sinkgroups for failover of the sinks. In case of the failure of sink1, sink2 take charge of storing data into hdfs. I tested the functionality of the sinkgroups’s failover successfully. When sink1 fails, sink2 successfully takes the charge and transfer the data to the destination. Here I am transferring the events by tailing a file contains 100 records. I shutdown the sink1 when it has processed 40 records.. After that all data start transferred by sink2. But, the file saved in hdfs contains 140 records. It means both downstream agents hold the events, if one fails another start transfer the events from the start as it hold all data in its defined channel. Is this the intended behavior of the flume’s sinkgroup in case of failover? How can I avoid the duplicate events? Is there any solution as part of flume’s sink processor to deal with the duplicates? Thanks & Regards, Ashutosh 이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에 포함된 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다. This E-mail may contain confidential information and/or copyright material. This email is intended for the use of the addressee only. If you receive this email by mistake, please either delete it without reproducing, distributing or retaining copies thereof or notify the sender immediately. 이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에 포함된 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다. This E-mail may contain confidential information and/or copyright material. This email is intended for the use of the addressee only. If you receive this email by mistake, please either delete it without reproducing, distributing or retaining copies thereof or notify the sender immediately.
