Re: How does Flink manage the kafka offset

2020-02-20 文章 Jin Yi
Hi Benchao, Thanks a lot! Eleanore On Thu, Feb 20, 2020 at 4:30 PM Benchao Li wrote: > Hi Jin, > > See below inline replies: > > My understanding is, upon startup, Flink Job Manager will contact kafka to >> get the offset for each partition for this consume group, and distribute >> the task to

Re: How does Flink manage the kafka offset

2020-02-20 文章 Benchao Li
Hi Jin, See below inline replies: My understanding is, upon startup, Flink Job Manager will contact kafka to > get the offset for each partition for this consume group, and distribute > the task to task managers, and it does not use kafka to manage the consumer > group. Generally, yes. If you a

How does Flink manage the kafka offset

2020-02-20 文章 Jin Yi
Hi there, We are running apache beam application with flink being the runner. We use the KafkaIO connector to read from topics: https://beam.apache.org/releases/javadoc/2.19.0/ and we have the following configuration, which enables auto commit of offset, no checkpointing is enabled, and it is pe