for the records cc dev@

On Thu, Oct 13, 2016 at 11:43 AM, Attila Simon <> wrote:
> Hi,
> auto.offset.reset aim to handle failure scenarios when Flume lost the
> track of offsets. When Flume is able to successfully consume the
> messages it also commits the last processed offset. When failure
> happens and <earliest> was set resetting offset would use the last
> committed value.
> I don't think that always starting from "zero" offset would be
> valuable (would result a lot of duplicates). So I assume you would
> like to have a recovery scenario. What you can do is setting the
> consumer to something new so if kafka still has the messages
> - you can check that with command line kafka consumer setting the
> --from-beginning argument as kafka by default purges them periodically
> - then flume would reset the offset to the effective beginning since
> offsets are stored per
> Quoted from Kafka docs
> (
> auto.offset.reset - What to do when there is no initial offset in
> Kafka or if the current offset does not exist any more on the server
> (e.g. because that data has been deleted):
> earliest: automatically reset the offset to the earliest offset
> latest: automatically reset the offset to the latest offset
> none: throw exception to the consumer if no previous offset is found
> for the consumer's group
> anything else: throw exception to the consumer.
> Cheers,
> Attila
> On Thu, Oct 13, 2016 at 10:00 AM, Ping PW Wang <> wrote:
>> Hi,
>> I used KafkaSource to consume the messages from Kafka. I found only new
>> messages were received while the old existing message not. I tried to use a
>> new consumer group and update the parameter "auto.offset.reset = latest" to
>> "earliest", but this does not work.
>> = test-consumer-group-new
>> = earliest
>> Anyone knows how to make KafkaSource consume the existing messages?
>> Thanks a lot for any advice!
>> Best Regards,
>> Wang Ping (王苹)
>> InfoSphere BigInsights, CDL
>> Email: Phone: (8610)82453448 Mobile: (86)17090815725
>> Address: Ring Bldg.No.28 Building,ZhongGuanCun Software Park,No.8 Dong Bei
>> Wang West Road, Haidian District Beijing P.R.China 100193
>> 地址:北京市海淀区东北旺西路8号,中关村软件园28号楼 邮编:100193

Reply via email to