Re: How to consume kafka from the last offset?

2020-03-26 Thread Jim Chen
Hi,
I am so sorry. It's not auto.offset.reset. Correctly, it is
*enable.auto.commit=false*

Best Wishs!

Dominik Wosiński  于2020年3月26日周四 下午4:20写道:

> Hey,
> Are You completely sure you mean *auto.offset.reset ??  *False is not
> valid setting for that AFAIK.
>
> Best,
> Dom.
>
> czw., 26 mar 2020 o 08:38 Jim Chen 
> napisał(a):
>
>> Thanks!
>>
>> I made a mistake. I forget to set the auto.offset.reset=false. It's my
>> fault.
>>
>> Dominik Wosiński  于2020年3月25日周三 下午6:49写道:
>>
>>> Hi Jim,
>>> Well, *auto.offset.reset *is only used when there is no offset saved
>>> for this *group.id * in Kafka. So, if You want to read
>>> the data from the latest record (and by latest I mean the newest here) You
>>> should assign the *group.id * that was not previously
>>> used and then FlinkKafkaConsumer should automatically fetch the last offset
>>> and start reading from that place.
>>>
>>>
>>> Best Regards,
>>> Dom.
>>>
>>> śr., 25 mar 2020 o 11:19 Jim Chen 
>>> napisał(a):
>>>
 Hi, All
   I use flink-connector-kafka-0.11 consume the Kafka0.11. In
 KafkaConsumer params, i set the group.id and auto.offset.reset. In the
 Flink1.10, set the kafkaConsumer.setStartFromGroupOffsets();
   Then, i restart the application, found the offset is not from the
 last position. Any one know where is wrong? HELP!

>>>


Re: How to consume kafka from the last offset?

2020-03-26 Thread Dominik Wosiński
Hey,
Are You completely sure you mean *auto.offset.reset ??  *False is not valid
setting for that AFAIK.

Best,
Dom.

czw., 26 mar 2020 o 08:38 Jim Chen  napisał(a):

> Thanks!
>
> I made a mistake. I forget to set the auto.offset.reset=false. It's my
> fault.
>
> Dominik Wosiński  于2020年3月25日周三 下午6:49写道:
>
>> Hi Jim,
>> Well, *auto.offset.reset *is only used when there is no offset saved for
>> this *group.id * in Kafka. So, if You want to read the
>> data from the latest record (and by latest I mean the newest here) You
>> should assign the *group.id * that was not previously
>> used and then FlinkKafkaConsumer should automatically fetch the last offset
>> and start reading from that place.
>>
>>
>> Best Regards,
>> Dom.
>>
>> śr., 25 mar 2020 o 11:19 Jim Chen 
>> napisał(a):
>>
>>> Hi, All
>>>   I use flink-connector-kafka-0.11 consume the Kafka0.11. In
>>> KafkaConsumer params, i set the group.id and auto.offset.reset. In the
>>> Flink1.10, set the kafkaConsumer.setStartFromGroupOffsets();
>>>   Then, i restart the application, found the offset is not from the last
>>> position. Any one know where is wrong? HELP!
>>>
>>


Re: How to consume kafka from the last offset?

2020-03-26 Thread Jim Chen
Thanks!

I made a mistake. I forget to set the auto.offset.reset=false. It's my
fault.

Dominik Wosiński  于2020年3月25日周三 下午6:49写道:

> Hi Jim,
> Well, *auto.offset.reset *is only used when there is no offset saved for
> this *group.id * in Kafka. So, if You want to read the
> data from the latest record (and by latest I mean the newest here) You
> should assign the *group.id * that was not previously
> used and then FlinkKafkaConsumer should automatically fetch the last offset
> and start reading from that place.
>
>
> Best Regards,
> Dom.
>
> śr., 25 mar 2020 o 11:19 Jim Chen 
> napisał(a):
>
>> Hi, All
>>   I use flink-connector-kafka-0.11 consume the Kafka0.11. In
>> KafkaConsumer params, i set the group.id and auto.offset.reset. In the
>> Flink1.10, set the kafkaConsumer.setStartFromGroupOffsets();
>>   Then, i restart the application, found the offset is not from the last
>> position. Any one know where is wrong? HELP!
>>
>


Re: How to consume kafka from the last offset?

2020-03-25 Thread Dominik Wosiński
Hi Jim,
Well, *auto.offset.reset *is only used when there is no offset saved for
this *group.id * in Kafka. So, if You want to read the
data from the latest record (and by latest I mean the newest here) You
should assign the *group.id * that was not previously used
and then FlinkKafkaConsumer should automatically fetch the last offset and
start reading from that place.


Best Regards,
Dom.

śr., 25 mar 2020 o 11:19 Jim Chen  napisał(a):

> Hi, All
>   I use flink-connector-kafka-0.11 consume the Kafka0.11. In KafkaConsumer
> params, i set the group.id and auto.offset.reset. In the Flink1.10, set
> the kafkaConsumer.setStartFromGroupOffsets();
>   Then, i restart the application, found the offset is not from the last
> position. Any one know where is wrong? HELP!
>


How to consume kafka from the last offset?

2020-03-25 Thread Jim Chen
Hi, All
  I use flink-connector-kafka-0.11 consume the Kafka0.11. In KafkaConsumer
params, i set the group.id and auto.offset.reset. In the Flink1.10, set
the kafkaConsumer.setStartFromGroupOffsets();
  Then, i restart the application, found the offset is not from the last
position. Any one know where is wrong? HELP!