I can see that offsets are stored in zookeeper and are not returned when i
query through kafka offset checker.
Can you please tell me how to monitor kafka flink consumer lag for 0.8
flink kafka consumer.
On Wed, Aug 3, 2016 at 3:29 PM, Stephan Ewen <se...@apache.org> wrote:
> Hi!
>
> Just check in the code. The 0.8 FlinkKafkaConsumer should always commit
> offsets, regardless of whether checkpointing is enables. The 0.9
> FlinkKafkaConsumer actually does not do any periodic offset committing when
> checkpointing is disabled.
>
> Greetings,
> Stephan
>
> On Wed, Aug 3, 2016 at 7:36 AM, Janardhan Reddy <
> janardhan.re...@olacabs.com> wrote:
>
>> Checkpointing wasn't enabled in the streaming job, but the offsets should
>> have been committed to zookeeper.
>>
>> But we don't see the offsets being written to zookeeper.
>>
>> On Tue, Aug 2, 2016 at 7:41 PM, Till Rohrmann <trohrm...@apache.org>
>> wrote:
>>
>>> Hi Janardhan,
>>>
>>> Flink should commit the current offsets to Zookeeper whenever a
>>> checkpoint has been completed. In case that you disabled checkpointing,
>>> then the offsets will be periodically committed to ZooKeeper. The default
>>> value is 60s.
>>>
>>> Could it be that there wasn't yet a completed checkpoint? Which version
>>> of Flink are you using?
>>>
>>> Cheers,
>>> Till
>>>
>>> On Tue, Aug 2, 2016 at 7:26 PM, Janardhan Reddy <
>>> janardhan.re...@olacabs.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> When the run the following command i am getting that no topic is
>>>> available for that consumer group. i am suing
>>>> flink-connector-kafka-0.8_${scala.version}(2.11).
>>>>
>>>> ./bin/kafka-consumer-groups.sh --zookeeper <> --group <> --describe
>>>>
>>>> No topic available for consumer group provided
>>>>
>>>>
>>>> Does the kafka consumer commit offset to the broker always ? Do we need
>>>> to enable checkpointing for the offsets to be committed ?
>>>>
>>>>
>>>> Thanks
>>>>
>>>
>>>
>>
>
