Hi Lorenzo, In theory, it should commit every 5 secs IF you keep polling the server. But I saw you "stopped" the consumer for some hours, which means the commit won't happen during this period. So, if it exceeds the retention period, it'll get deleted. That's my assumption. You need to check the logs (both client and server) to find the clues.
Luke On Mon, Oct 10, 2022 at 5:53 PM Lorenzo Rovere <l.rov...@reply.it> wrote: > Hi, thanks for your response. > > Is there any chance the offset is never committed on the > "__consumer_offsets” topic, although auto commit is enabled every 5 seconds? > We are checking daily and the offset is always set to NULL. > > > > Lorenzo Rovere > > Technology Reply > Via Avogadri, 2 > 31057 - Silea (TV) - ITALY > phone: +39 0422 1836521 > l.rov...@reply.it > www.reply.it > -----Original Message----- > From: Luke Chen <show...@gmail.com> > Sent: 7 October, 2022 2:15 PM > To: users@kafka.apache.org > Subject: Re: consumer loses offset > > Hi Lorenzo, > > Sounds like it is caused by this bug: > https://issues.apache.org/jira/browse/KAFKA-13636 > If you're not in the versions of fixed version list or newer, please try > to upgrade it. > > Thanks. > Luke > > On Fri, Oct 7, 2022 at 5:36 PM Lorenzo Rovere <l.rov...@reply.it> wrote: > > > Hi everyone, I have a simple question about Kafka offsets. > > > > We have 1 producer and 1 consumer. > > > > Imagine the consumer reading till offset 5 (for example) and then > > suddenly stops for some hours. In the meantime the producer keeps > > writing messages and the offset of the last message is 10 (always for > > example). When we restart the microservice that contains the consumer, > > where does it start to read from, if we have auto.offset.reset=latest? > > > > > > > > I ask this because one of our costumers complains about starting > > reading from 10, thus losing messages between 5 and 10. Is that correct? > > > > > > > > Other configs: > > enable.auto.commit = true > > > > auto.commit.interval.ms = 5000 > > > > log.retention.hours=168 > > > > offsets.retention.minutes (7d) > > > > > > > > We also noticed that on the "__consumer_offsets” topic offset is > > always set to NULL > > > > [consumer_group, topic_name,partition]::NULL > > > > > > > > Can you help me understanding what’s happening? Thanks a lot > > > > > > Lorenzo Rovere > > > > Technology Reply > > Via Avogadri, 2 > > 31057 - Silea (TV) - ITALY > > phone: +39 0422 1836521 > > l.rov...@reply.it > > www.reply.it > > > > [image: Technology Reply] > > >
