Producers were publishing data for the topic. And consumers were also connected, sending heartbeat pings every 100 ms.
On Thu, 12 Jan 2017 at 17:15 Michael Freeman <mikfree...@gmail.com> wrote: > If the topic has not seen traffic for a while then Kafka will remove the > stored offset. When your consumer reconnects Kafka no longer has the offset > so it will reprocess from earliest. > > Michael > > > On 12 Jan 2017, at 11:13, Mahendra Kariya <mahendra.kar...@go-jek.com> > wrote: > > > > Hey All, > > > > We have a Kafka cluster hosted on Google Cloud. There was some network > > issue on the cloud and suddenly, the offset for a particular consumer > group > > got reset to earliest and all of a sudden the lag was in millions. We > > aren't able to figure out what went wrong. Has anybody faced the > > same/similar issue? Does anybody have any debugging tips? > > > > Some relevant info: > > > > - The auto.offset.reset config for the consumer is set to earliest > > - The offsets are stored on Kafka > > - Total nodes on cluster: 4 > > - Replication factor: 3 > > - Partitions: 50 >
