[
https://issues.apache.org/jira/browse/KAFKA-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14956273#comment-14956273
]
Onur Karaman commented on KAFKA-2017:
-------------------------------------
Hey everyone. So yeah there are some nice benefits to persisting to kafka
(maybe write group changes to a __group_states topic or something to that
effect):
1. you can reduce zk load - especially writes.
2. I think it's conceptually cleaner to have consumer state and broker state
cleanly separated: consumer state like offsets and groups in one or several
kafka topics; broker state in zookeeper.
3. It is possible to merge offset information with this group state into one
new topic (we can get rid of the __consumer_offsets topic) so you'd be able to
see a complete history of a group including its commits and group membership
changes. Admittedly, this kind of group history can be achieved to a small
extent in the coordinator's log4j logs. Also it's unclear what the right
cleanup policy would be. Compaction would eventually only keep the latest
state, while regular deletion would get rid of some latest but old group
history.
> Persist Coordinator State for Coordinator Failover
> --------------------------------------------------
>
> Key: KAFKA-2017
> URL: https://issues.apache.org/jira/browse/KAFKA-2017
> Project: Kafka
> Issue Type: Sub-task
> Components: consumer
> Affects Versions: 0.9.0.0
> Reporter: Onur Karaman
> Assignee: Guozhang Wang
> Fix For: 0.9.0.0
>
> Attachments: KAFKA-2017.patch, KAFKA-2017_2015-05-20_09:13:39.patch,
> KAFKA-2017_2015-05-21_19:02:47.patch
>
>
> When a coordinator fails, the group membership protocol tries to failover to
> a new coordinator without forcing all the consumers rejoin their groups. This
> is possible if the coordinator persists its state so that the state can be
> transferred during coordinator failover. This state consists of most of the
> information in GroupRegistry and ConsumerRegistry.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
