[
https://issues.apache.org/jira/browse/KAFKA-2123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524491#comment-14524491
]
Ewen Cheslack-Postava commented on KAFKA-2123:
----------------------------------------------
Updated to add backoff back in upon retries, which was removed when I unified
the sync and async processing. I ended up generalizing this a bit to provide
for scheduling tasks that need to be delayed for some time. With the
generalization, I could also move the automatic background offset commits to
this system as well. This also fixed a bug in the existing implementation where
a long poll might cause automatic offset commit to not run since we only
checked before the poll if a commit was needed but didn't account for the need
to stop polling and commit offsets if the commit should happen sometime during
the poll() period.
[~jkreps] and [~junrao] might want to take a look -- the producer code has
gotten complicated because we have so many timeouts to coordinate, so computing
the right poll timeouts has to take into account a bunch of variables. I think
a centralized scheduler might help keep this complexity in check.
> Make new consumer offset commit API use callback + future
> ---------------------------------------------------------
>
> Key: KAFKA-2123
> URL: https://issues.apache.org/jira/browse/KAFKA-2123
> Project: Kafka
> Issue Type: Improvement
> Components: clients, consumer
> Reporter: Ewen Cheslack-Postava
> Assignee: Ewen Cheslack-Postava
> Fix For: 0.8.3
>
> Attachments: KAFKA-2123.patch, KAFKA-2123_2015-04-30_11:23:05.patch,
> KAFKA-2123_2015-05-01_19:33:19.patch
>
>
> The current version of the offset commit API in the new consumer is
> void commit(offsets, commit type)
> where the commit type is either sync or async. This means you need to use
> sync if you ever want confirmation that the commit succeeded. Some
> applications will want to use asynchronous offset commit, but be able to tell
> when the commit completes.
> This is basically the same problem that had to be fixed going from old
> consumer -> new consumer and I'd suggest the same fix using a callback +
> future combination. The new API would be
> Future<Void> commit(Map<TopicPartition, Long> offsets, ConsumerCommitCallback
> callback);
> where ConsumerCommitCallback contains a single method:
> public void onCompletion(Exception exception);
> We can provide shorthand variants of commit() for eliding the different
> arguments.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
