[
https://issues.apache.org/jira/browse/KAFKA-1144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13832019#comment-13832019
]
Imran Rashid commented on KAFKA-1144:
-------------------------------------
yes, that addition to the commit api would do the trick. (that is the main
point of this patch, I just happened to choose a different signature.) With
rebalance notifications, that removes the need for conditional updates.
glad to hear this will all be in 0.9 -- but does that mean this patch is off
the table for 0.8.*? That's too bad, this would be a big help now. I could
change the signature to match what it will be in 0.9. If not, I suppose I can
always just make the zk updates myself directly, under the covers of the
consumer api wrapper I'm writing.
> commitOffsets can be passed the offsets to commit
> -------------------------------------------------
>
> Key: KAFKA-1144
> URL: https://issues.apache.org/jira/browse/KAFKA-1144
> Project: Kafka
> Issue Type: Improvement
> Components: consumer
> Affects Versions: 0.8
> Reporter: Imran Rashid
> Assignee: Neha Narkhede
> Attachments:
> 0001-allow-committing-of-arbitrary-offsets-to-facilitate-.patch,
> 0002-add-protection-against-backward-commits.patch
>
>
> This adds another version of commitOffsets that takes the offsets to commit
> as a parameter.
> Without this change, getting correct user code is very hard. Despite kafka's
> at-least-once guarantees, most user code doesn't actually have that
> guarantee, and is almost certainly wrong if doing batch processing. Getting
> it right requires some very careful synchronization between all consumer
> threads, which is both:
> 1) painful to get right
> 2) slow b/c of the need to stop all workers during a commit.
> This small change simplifies a lot of this. This was discussed extensively on
> the user mailing list, on the thread "are kafka consumer apps guaranteed to
> see msgs at least once?"
> You can also see an example implementation of a user api which makes use of
> this, to get proper at-least-once guarantees by user code, even for batches:
> https://github.com/quantifind/kafka-utils/pull/1
> I'm open to any suggestions on how to add unit tests for this.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
