[
https://issues.apache.org/jira/browse/KAFKA-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114320#comment-14114320
]
Gian Merlino commented on KAFKA-1612:
-------------------------------------
I am currently working around this by setting auto.commit.enable = false and
calling commitOffsets() manually from time to time, at certain points when I
know that all consumed data has been fully processed. But I was hoping that
auto.commit.enable could be made to offer guaranteed delivery for people that
fully process each message before reading the next one. I think it doesn't
currently, since the kafka-consumer-autocommit thread can commit the offset for
a message at any time after kafkaStream.next() returns that message, even if it
hasn't been fully processed yet.
Maybe instead it could commit offsets only for the previous message returned,
instead of the current message. People that do batch processing of messages
would still need to set auto.commit.enable = false and commitOffsets() manually
to get guaranteed delivery, but I think that's okay, since they're doing
something more complicated than the iterator is really able to help them with.
> Consumer offsets auto-commit before processing finishes
> -------------------------------------------------------
>
> Key: KAFKA-1612
> URL: https://issues.apache.org/jira/browse/KAFKA-1612
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Affects Versions: 0.8.1.1
> Reporter: Gian Merlino
> Assignee: Neha Narkhede
>
> In a loop like this,
> for (message <- kafkaStream) {
> process(message)
> }
> The consumer can commit offsets for the next message while "process" is
> running. If the program crashes during "process", the next run will pick up
> from the *next* message. The message in flight at the time of the crash will
> never actually finish processing. Instead, I would have expected the high
> level consumer to deliver messages at least once.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
