Re: how to make KafkaSource consume the existing messages

午 06:32:02---Hi, One more thing. If you switch to the new group.id and would like toFrom: Attila Simon <s...@cloudera.com>To: user@flume.apache.org, d...@flume.apache.orgDate: 2016-10-13 下午 06:32Subject: Re: how to make KafkaSource consume the existing messages Hi,One more thing. If you

Re: how to make KafkaSource consume the existing messages

Hi, Which version of Kafka are you using? Off the top of my head it should be: tier2.sources.source1.kafka.auto.offset.reset = earliest Of course changing the group ID or if it's an older version of Kafka removing the corresponding offset znode from zookeeper ought to do the trick -- Chris

Re: how to make KafkaSource consume the existing messages

Hi, One more thing. If you switch to the new group.id and would like to maintain the read from beginning behaviour every time flume restart then you might try setting enable.auto.commit to false. Again Kafka normally won't store the events indefinitely. Cheers, Attila On Thu, Oct 13, 2016 at

Re: how to make KafkaSource consume the existing messages

for the records cc dev@ On Thu, Oct 13, 2016 at 11:43 AM, Attila Simon wrote: > Hi, > > auto.offset.reset aim to handle failure scenarios when Flume lost the > track of offsets. When Flume is able to successfully consume the > messages it also commits the last processed

Re: how to make KafkaSource consume the existing messages

Hi, auto.offset.reset aim to handle failure scenarios when Flume lost the track of offsets. When Flume is able to successfully consume the messages it also commits the last processed offset. When failure happens and was set resetting offset would use the last committed value. I don't think that