[
https://issues.apache.org/jira/browse/KAFKA-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234557#comment-14234557
]
Jiangjie Qin commented on KAFKA-1650:
-------------------------------------
[~junrao] Could you also help review the patch as Joel thought it is kind of a
big patch of new design for mirror maker.
Some major changes are listed below to expedite the review:
1. ZookeeperConsumerConnector auto commit is turned off.
2. A Map<SourceTopicPartition, Map<TargetTopicPartition, Offset>> is used to
keep track of the offsets of messages successfully sent to target cluster. The
map is updated on new producer's callback.
3. ZookeeperConsumerConnector has been modified to take a rebalance listener
and take external offset map to commit offset.
4. Mirror maker wires in a consumer rebalance listener to avoid duplicates on
consumer rebalance. On consumer rebalance, before releasing the partition
ownership, it will first clean the messages in data channel, wait until the
messages taken by producer receives callback, commit offset and continue the
rebalance.
5. An offset commit thread will be responsible for committing offset
periodically.
> Mirror Maker could lose data on unclean shutdown.
> -------------------------------------------------
>
> Key: KAFKA-1650
> URL: https://issues.apache.org/jira/browse/KAFKA-1650
> Project: Kafka
> Issue Type: Improvement
> Reporter: Jiangjie Qin
> Assignee: Jiangjie Qin
> Attachments: KAFKA-1650.patch, KAFKA-1650_2014-10-06_10:17:46.patch,
> KAFKA-1650_2014-11-12_09:51:30.patch, KAFKA-1650_2014-11-17_18:44:37.patch,
> KAFKA-1650_2014-11-20_12:00:16.patch, KAFKA-1650_2014-11-24_08:15:17.patch,
> KAFKA-1650_2014-12-03_15:02:31.patch, KAFKA-1650_2014-12-03_19:02:13.patch,
> KAFKA-1650_2014-12-04_11:59:07.patch
>
>
> Currently if mirror maker got shutdown uncleanly, the data in the data
> channel and buffer could potentially be lost. With the new producer's
> callback, this issue could be solved.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
