Awesome summary, Dana. I'd like to fit this into our docs, but I'm not sure where does step-by-step-description of the protocol fits. Maybe in "Design" section?
Just one more thing: 8) At any time, the broker can respond to a fetch request with "Rebalancing" error code, at which point the assignment dance begins from scratch: consumers needs to send the sync request, leaders need to create an assignment, etc. Gwen On Tue, Mar 29, 2016 at 9:05 AM, Dana Powers <dana.pow...@gmail.com> wrote: > I also found the documentation difficult to parse when it came time to > implement group APIs. I ended up just reading the client source code and > trying api calls until it made sense. > > My general description from off the top of my head: > > (1) have all consumers submit a shared protocol_name string* and > bytes-serialized protocol_metadata, which includes the topic(s) for > subscription. > (2) All consumers should be prepared to parse a JoinResponse and identify > whether they have been selected as the group leader. > (3) The non-leaders go straight to SyncGroupRequest, with group_assignment > empty, and will wait get their partition assignments via the > SyncGroupResponse. > (4) Unlike the others, the consumer-leader will get the agreed > protocol_name and all member's metadata in its JoinGroupResponse. > (5) The consumer-leader must apply the protocol_name assignment strategy to > the member list + metadata (plus the cluster metadata), generating a set of > member assignments (member_id -> topic partitions). > (6)The consumer-leader then submits that member_assignment data in its > SyncGroupRequest. > (7) The broker will take the group member_assignment data and split it out > into each members SyncGroupResponse (including the consumer-leader). > > At this point all members have their assignments and can start chugging > along. > > I've tried to encapsulate the group request / response protocol info > cleanly into kafka-python, if that helps you: > > https://github.com/dpkp/kafka-python/blob/1.0.2/kafka/protocol/group.py > > I'd be willing to help improve the docs if someone points me at the "source > of truth". Or feel free to ping on irc #apache-kafka > > -Dana > > *for java client these are 'range' or 'roundrobin', but are opaque and can > be anything if you aren't interested in joining groups that have > heterogeneous clients. related, the protocol_metadata and member_metadata > are also opaque and technically can be anything you like. But sticking with > the documented metadata specs should in theory allow heterogenous clients > to cooperate. > > On Tue, Mar 29, 2016 at 8:21 AM, Heath Ivie <hi...@autoanything.com> > wrote: > > > Does anyone have better documentation around the group membership APIs? > > > > The information about the APIs are great at the beginning but get > > progressively sparse towards then end. > > > > I am not finding enough information about the values of the request > fields > > to join / sync the group. > > > > Can anyone help me or send me the some additional documentation? > > > > Heath Ivie > > Solutions Architect > > > > > > Warning: This e-mail may contain information proprietary to AutoAnything > > Inc. and is intended only for the use of the intended recipient(s). If > the > > reader of this message is not the intended recipient(s), you have > received > > this message in error and any review, dissemination, distribution or > > copying of this message is strictly prohibited. If you have received this > > message in error, please notify the sender immediately and delete all > > copies. > > >
