Hello, Kafka users!
I've been trying to build Apache Kafka into our system, and for one part of
our system, we have a use-case where we have to leave auto-commit enabled
but would like to reset the offset numbers to an earlier offset if a
failure happens in our code. We are using auto-commit because the 0.8.x
branch (0.8.2.1 for us, atm) has a bug in the commitOffsets logic and
doesn't retry like it should; although, this has been fixed in the
unreleased master branch.
Anyway, this is our issue:
1. Streaming data from kafka using a high level consumer
2. Failure happens on our end:
1. Initiate shutdown of the high level consumer/reader.
2. Send a OffsetCommitRequest using a SimpleConsumer connected to the
offset coordinator.for the very same consumer group that the high level
consumer/reader was using.
3. Offsets are then reset.
4. A few seconds later, offsets get changed back to the latest
automatically.
I suspect #4 happens due to autocommit logic, but why? As far as I can
tell, we shut down our consumer that had known about the higher offsets.
Plus, I thought once I sent an OffsetCommitRequest for a given consumer
group, that information should be flowed to the rest of the consumers in
the consumer group.
Is this another bug, or am I misunderstanding something about Kafka?
--
*William GrimSr. Software Engineerm: 914 418 4115
<914%20418%204115>e: wg...@signal.co <wg...@signal.co>signal.co
<http://www.google.com/url?q=http%3A%2F%2Fsignal.co%2F&sa=D&sntz=1&usg=AFrqEzf9BbOBbhu7G5O2liTp3wXdU2t6FA>________________________Cut
Through the NoiseThis e-mail and any files transmitted with it are for the
sole use of the intended recipient(s) and may contain confidential and
privileged information. Any unauthorized use of this email is strictly
prohibited. ©2015 Signal. All rights reserved.*
