Hey James,
the algorithm is relying on client side to provide unique consumer member id. It will break unless we enforce some sort of validation (host + port) on the server side. To simplify the first version, we do not plan to enforce validation. A good comparison would be the EOS producer which is in charge of generating unique transaction id sequence. IMO for broker logic, the tolerance of client side error is not unlimited. Thank you! ________________________________ From: James Cheng <wushuja...@gmail.com> Sent: Saturday, July 28, 2018 1:26 AM To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id > On Jul 26, 2018, at 11:09 PM, Guozhang Wang <wangg...@gmail.com> wrote: > > Hi Boyang, > > Thanks for the proposed KIP. I made a pass over the wiki and here are some > comments / questions: > > 1. In order to preserve broker compatibility, we need to make sure the > broker version discovery logic can be integrated with this new logic. I.e. > if a newer versioned consumer is talking to an older versioned broker who > does not recognize V4, the client needs to downgrade its JoinGroupRequest > version to V3 and not setting the member-id specifically. You can take a > look at the ApiVersionsRequest and see how to work with it. > > 2. There may exist some manners to validate that two different clients do > not send with the same member id, for example if we pass along the > host:port information from KafkaApis to the GroupCoordinator interface. But > I think this is overly complicated the logic and may not worthwhile than > relying on users to specify unique member ids. Boyang, Thanks for the KIP! How will the algorithm behave if multiple consumers provide the same member id? -James > 3. Minor: you would need to bumping up the version of JoinGroupResponse to > V4 as well. > > 4. Minor: in the wiki page, you need to specify the actual string value for > `MEMBER_ID`, for example "member.id". > > 5. When this additional config it specified by users, we should consider > setting the default of internal `LEAVE_GROUP_ON_CLOSE_CONFIG` to false, > since otherwise its effectiveness would be less. > > > Guozhang > > > >> On Thu, Jul 26, 2018 at 9:20 PM, Boyang Chen <bche...@outlook.com> wrote: >> >> Hey friends, >> >> >> I would like to open a discussion thread on KIP-345: >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A >> +Reduce+multiple+consumer+rebalances+by+specifying+member+id >> >> >> This KIP is trying to resolve multiple rebalances by maintaining the >> consumer member id across rebalance generations. I have verified the theory >> on our internal Stream application, and it could reduce rebalance time to a >> few seconds when service is rolling restart. >> >> >> Let me know your thoughts, thank you! >> >> >> Best, >> >> Boyang >> > > > > -- > -- Guozhang
