Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

Hey Mike,

thanks for asking! We have shipped 85% of the features in 345 already, the 
remaining works are just tooling supports: 
https://issues.apache.org/jira/browse/KAFKA-7018

you should be able to use it in 2.3 

Boyang


From: Mike Freyberger 
Sent: Friday, June 14, 2019 8:42 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hi Boyang,

Thanks for the update. To what extent will KIP-345 be available in 2.3.0?

Mike

On 6/13/19, 5:36 PM, "Boyang Chen"  wrote:

Hey all,

we decided to push 2 minor changes for better usability for static 
membership.

  1.  Add `group.instance.id` field to the `DescribeGroupResponse` API. 
This gives visibility for client user to check which static members are 
registered on the current group.
  2.  Add getGroupInstanceId() function to `Subscription` class. This helps 
utilizes static member information to generate more stable resource assignment 
when the group membership remains the same during rolling bounce.

The KIP is updated with these changes. No compatibility issue is 
anticipated. Let me know if you have any concerns.

Best,
Boyang


From: Boyang Chen 
Sent: Friday, April 26, 2019 11:16 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hey all,

there is a minor change to the stream side logic for static 
membership<https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances>.
 Originally we chose to piggy-back on user to supply a unique `client.id` 
config that we could use to construct per thread level consumer 
`group.instance.id`. This approach has several drawbacks:

  1.  We already have functionalities relying on `client.id`, and it is not 
always the case where user wants to configure it differently for individual 
instances. For example, currently user could throttle requests under same 
client.id, which is a solid use case where the `client.id` should duplicate.
  2.  Existing stream users may unconsciously trigger static membership if 
they already set `client.id` in their Stream apps. This includes unexpected 
fatal errors due to `group.instance.id` fencing we are going to introduce.

In conclusion, it is not good practice to overload existing config that 
users rely on unless there is no side effect. To make more fault tolerant 
upgrade path, we decide to let stream users choose to set `group.instance.id` 
if they want to enable static membership.

Thank you Guozhang and Matthias for the great discussions and enhancements 
for the KIP!

Best,
Boyang



From: Boyang Chen 
Sent: Saturday, March 9, 2019 2:28 PM
To: dev@kafka.apache.org
    Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hi Mike,

Yes that's the plan!


From: Mike Freyberger 
Sent: Saturday, March 9, 2019 10:04 AM
To: dev@kafka.apache.org
    Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hi Boyang,

Is this work targeted for Kafka 2.3? I am eager to use this new feature.

Thanks,

Mike Freyberger

On 12/21/18, 1:21 PM, "Mayuresh Gharat"  wrote:

Hi Boyang,

Regarding "However, we shall still attempt to remove the member static 
info
if the given `member.id` points to an existing `group.instance.id` upon
LeaveGroupRequest, because I could think of the possibility that in long
term we could want to add static membership leave group logic for more
fine-grained use cases."

> I think, there is some confusion here. I am probably not putting it
> right.
>
I agree, If a static member sends LeaveGroupRequest, it should be 
removed
> from the group.
>
Now getting back to downgrade of static membership to Dynamic 
membership,
> with the example described earlier  (copying it again for ease of 
reading)
> :
>

>>1. Lets say we have 4 consumers :  c1, c2, c3, c4 in the static 
group.
>>2. The group.instance.id for each of there are as follows :
>>   - c1 -> gc1, c2 -> gc2, c3 -> gc3, c4 -> gc4
>>3. The mapping on the GroupCordinator would be :
>>   - gc1 -> mc1, gc2 -> mc2, gc3 -> mc3, gc4 -> mc4, where mc1, 
mc2,
>>   mc3, mc4 are the randomly generated memberIds for c1, c2, c3, 
c4
>>   respectively, by the GroupCoordinator.
>>4. Now we do a restart to move the group to

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Mike Freyberger]

Hi Boyang,

Thanks for the update. To what extent will KIP-345 be available in 2.3.0?

Mike

On 6/13/19, 5:36 PM, "Boyang Chen"  wrote:

Hey all,

we decided to push 2 minor changes for better usability for static 
membership.

  1.  Add `group.instance.id` field to the `DescribeGroupResponse` API. 
This gives visibility for client user to check which static members are 
registered on the current group.
  2.  Add getGroupInstanceId() function to `Subscription` class. This helps 
utilizes static member information to generate more stable resource assignment 
when the group membership remains the same during rolling bounce.

The KIP is updated with these changes. No compatibility issue is 
anticipated. Let me know if you have any concerns.

Best,
Boyang


From: Boyang Chen 
Sent: Friday, April 26, 2019 11:16 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hey all,

there is a minor change to the stream side logic for static 
membership<https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances>.
 Originally we chose to piggy-back on user to supply a unique `client.id` 
config that we could use to construct per thread level consumer 
`group.instance.id`. This approach has several drawbacks:

  1.  We already have functionalities relying on `client.id`, and it is not 
always the case where user wants to configure it differently for individual 
instances. For example, currently user could throttle requests under same 
client.id, which is a solid use case where the `client.id` should duplicate.
  2.  Existing stream users may unconsciously trigger static membership if 
they already set `client.id` in their Stream apps. This includes unexpected 
fatal errors due to `group.instance.id` fencing we are going to introduce.

In conclusion, it is not good practice to overload existing config that 
users rely on unless there is no side effect. To make more fault tolerant 
upgrade path, we decide to let stream users choose to set `group.instance.id` 
if they want to enable static membership.

Thank you Guozhang and Matthias for the great discussions and enhancements 
for the KIP!

Best,
Boyang



From: Boyang Chen 
Sent: Saturday, March 9, 2019 2:28 PM
To: dev@kafka.apache.org
    Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hi Mike,

Yes that's the plan!


From: Mike Freyberger 
Sent: Saturday, March 9, 2019 10:04 AM
To: dev@kafka.apache.org
    Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hi Boyang,

Is this work targeted for Kafka 2.3? I am eager to use this new feature.

Thanks,

Mike Freyberger

On 12/21/18, 1:21 PM, "Mayuresh Gharat"  wrote:

Hi Boyang,

Regarding "However, we shall still attempt to remove the member static 
info
if the given `member.id` points to an existing `group.instance.id` upon
LeaveGroupRequest, because I could think of the possibility that in long
term we could want to add static membership leave group logic for more
fine-grained use cases."

> I think, there is some confusion here. I am probably not putting it
> right.
>
I agree, If a static member sends LeaveGroupRequest, it should be 
removed
> from the group.
>
Now getting back to downgrade of static membership to Dynamic 
membership,
> with the example described earlier  (copying it again for ease of 
reading)
> :
>

>>1. Lets say we have 4 consumers :  c1, c2, c3, c4 in the static 
group.
>>2. The group.instance.id for each of there are as follows :
>>   - c1 -> gc1, c2 -> gc2, c3 -> gc3, c4 -> gc4
>>3. The mapping on the GroupCordinator would be :
>>   - gc1 -> mc1, gc2 -> mc2, gc3 -> mc3, gc4 -> mc4, where mc1, 
mc2,
>>   mc3, mc4 are the randomly generated memberIds for c1, c2, c3, 
c4
>>   respectively, by the GroupCoordinator.
>>4. Now we do a restart to move the group to dynamic membership.
>>5. We bounce c1 first and it rejoins with UNKNOWN_MEMBERID (since 
we
>>don't persist the previously assigned memberId mc1 anywhere on 
the c1).
>>
> - We agree that there is no way to recognize that c1 was a part of the
> group, *earlier*.  If

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

Hey all,

we decided to push 2 minor changes for better usability for static membership.

  1.  Add `group.instance.id` field to the `DescribeGroupResponse` API. This 
gives visibility for client user to check which static members are registered 
on the current group.
  2.  Add getGroupInstanceId() function to `Subscription` class. This helps 
utilizes static member information to generate more stable resource assignment 
when the group membership remains the same during rolling bounce.

The KIP is updated with these changes. No compatibility issue is anticipated. 
Let me know if you have any concerns.

Best,
Boyang


From: Boyang Chen 
Sent: Friday, April 26, 2019 11:16 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hey all,

there is a minor change to the stream side logic for static 
membership<https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances>.
 Originally we chose to piggy-back on user to supply a unique `client.id` 
config that we could use to construct per thread level consumer 
`group.instance.id`. This approach has several drawbacks:

  1.  We already have functionalities relying on `client.id`, and it is not 
always the case where user wants to configure it differently for individual 
instances. For example, currently user could throttle requests under same 
client.id, which is a solid use case where the `client.id` should duplicate.
  2.  Existing stream users may unconsciously trigger static membership if they 
already set `client.id` in their Stream apps. This includes unexpected fatal 
errors due to `group.instance.id` fencing we are going to introduce.

In conclusion, it is not good practice to overload existing config that users 
rely on unless there is no side effect. To make more fault tolerant upgrade 
path, we decide to let stream users choose to set `group.instance.id` if they 
want to enable static membership.

Thank you Guozhang and Matthias for the great discussions and enhancements for 
the KIP!

Best,
Boyang



From: Boyang Chen 
Sent: Saturday, March 9, 2019 2:28 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hi Mike,

Yes that's the plan!


From: Mike Freyberger 
Sent: Saturday, March 9, 2019 10:04 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hi Boyang,

Is this work targeted for Kafka 2.3? I am eager to use this new feature.

Thanks,

Mike Freyberger

On 12/21/18, 1:21 PM, "Mayuresh Gharat"  wrote:

Hi Boyang,

Regarding "However, we shall still attempt to remove the member static info
if the given `member.id` points to an existing `group.instance.id` upon
LeaveGroupRequest, because I could think of the possibility that in long
term we could want to add static membership leave group logic for more
fine-grained use cases."

> I think, there is some confusion here. I am probably not putting it
> right.
>
I agree, If a static member sends LeaveGroupRequest, it should be removed
> from the group.
>
Now getting back to downgrade of static membership to Dynamic membership,
> with the example described earlier  (copying it again for ease of reading)
> :
>

>>1. Lets say we have 4 consumers :  c1, c2, c3, c4 in the static group.
>>2. The group.instance.id for each of there are as follows :
>>   - c1 -> gc1, c2 -> gc2, c3 -> gc3, c4 -> gc4
>>3. The mapping on the GroupCordinator would be :
>>   - gc1 -> mc1, gc2 -> mc2, gc3 -> mc3, gc4 -> mc4, where mc1, mc2,
>>   mc3, mc4 are the randomly generated memberIds for c1, c2, c3, c4
>>   respectively, by the GroupCoordinator.
>>4. Now we do a restart to move the group to dynamic membership.
>>5. We bounce c1 first and it rejoins with UNKNOWN_MEMBERID (since we
>>don't persist the previously assigned memberId mc1 anywhere on the 
c1).
>>
> - We agree that there is no way to recognize that c1 was a part of the
> group, *earlier*.  If yes, the statement : "The dynamic member rejoins
> the group without `group.instance.id`. It will be accepted since it is a
> known member." is not necessarily true, right?
>


> - Now I *agree* with "However, we shall still attempt to remove the
> member static info if the given `member.id` points to an existing `
> group.instance.id` upon LeaveGroupRequest, because I could think of the
> possibility that in long term we could want to add static membership le

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

Hey friends,

we decide to introduce another protocol change to include `group.instance.id` 
into Sync/Heartbeat/OffsetCommit protocols. The reason is because we found that 
to be the most safe and consistent approach to fence duplicate consumer 
clients. More details in updated KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances

Let me know if you have any concerns.

Best,
Boyang


From: Boyang Chen 
Sent: Friday, April 26, 2019 11:16 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hey all,

there is a minor change to the stream side logic for static 
membership<https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances>.
 Originally we chose to piggy-back on user to supply a unique `client.id` 
config that we could use to construct per thread level consumer 
`group.instance.id`. This approach has several drawbacks:

  1.  We already have functionalities relying on `client.id`, and it is not 
always the case where user wants to configure it differently for individual 
instances. For example, currently user could throttle requests under same 
client.id, which is a solid use case where the `client.id` should duplicate.
  2.  Existing stream users may unconsciously trigger static membership if they 
already set `client.id` in their Stream apps. This includes unexpected fatal 
errors due to `group.instance.id` fencing we are going to introduce.

In conclusion, it is not good practice to overload existing config that users 
rely on unless there is no side effect. To make more fault tolerant upgrade 
path, we decide to let stream users choose to set `group.instance.id` if they 
want to enable static membership.

Thank you Guozhang and Matthias for the great discussions and enhancements for 
the KIP!

Best,
Boyang



From: Boyang Chen 
Sent: Saturday, March 9, 2019 2:28 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hi Mike,

Yes that's the plan!


From: Mike Freyberger 
Sent: Saturday, March 9, 2019 10:04 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hi Boyang,

Is this work targeted for Kafka 2.3? I am eager to use this new feature.

Thanks,

Mike Freyberger

On 12/21/18, 1:21 PM, "Mayuresh Gharat"  wrote:

Hi Boyang,

Regarding "However, we shall still attempt to remove the member static info
if the given `member.id` points to an existing `group.instance.id` upon
LeaveGroupRequest, because I could think of the possibility that in long
term we could want to add static membership leave group logic for more
fine-grained use cases."

> I think, there is some confusion here. I am probably not putting it
> right.
>
I agree, If a static member sends LeaveGroupRequest, it should be removed
> from the group.
>
Now getting back to downgrade of static membership to Dynamic membership,
> with the example described earlier  (copying it again for ease of reading)
> :
>

>>1. Lets say we have 4 consumers :  c1, c2, c3, c4 in the static group.
>>2. The group.instance.id for each of there are as follows :
>>   - c1 -> gc1, c2 -> gc2, c3 -> gc3, c4 -> gc4
>>3. The mapping on the GroupCordinator would be :
>>   - gc1 -> mc1, gc2 -> mc2, gc3 -> mc3, gc4 -> mc4, where mc1, mc2,
>>   mc3, mc4 are the randomly generated memberIds for c1, c2, c3, c4
>>   respectively, by the GroupCoordinator.
>>4. Now we do a restart to move the group to dynamic membership.
>>5. We bounce c1 first and it rejoins with UNKNOWN_MEMBERID (since we
>>don't persist the previously assigned memberId mc1 anywhere on the 
c1).
>>
> - We agree that there is no way to recognize that c1 was a part of the
> group, *earlier*.  If yes, the statement : "The dynamic member rejoins
> the group without `group.instance.id`. It will be accepted since it is a
> known member." is not necessarily true, right?
>


> - Now I *agree* with "However, we shall still attempt to remove the
> member static info if the given `member.id` points to an existing `
> group.instance.id` upon LeaveGroupRequest, because I could think of the
> possibility that in long term we could want to add static membership leave
> group logic for more fine-grained use cases."
>
But that would only happen if the GroupCoordinator all

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

Hey all,

there is a minor change to the stream side logic for static 
membership<https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances>.
 Originally we chose to piggy-back on user to supply a unique `client.id` 
config that we could use to construct per thread level consumer 
`group.instance.id`. This approach has several drawbacks:

  1.  We already have functionalities relying on `client.id`, and it is not 
always the case where user wants to configure it differently for individual 
instances. For example, currently user could throttle requests under same 
client.id, which is a solid use case where the `client.id` should duplicate.
  2.  Existing stream users may unconsciously trigger static membership if they 
already set `client.id` in their Stream apps. This includes unexpected fatal 
errors due to `group.instance.id` fencing we are going to introduce.

In conclusion, it is not good practice to overload existing config that users 
rely on unless there is no side effect. To make more fault tolerant upgrade 
path, we decide to let stream users choose to set `group.instance.id` if they 
want to enable static membership.

Thank you Guozhang and Matthias for the great discussions and enhancements for 
the KIP!

Best,
Boyang



From: Boyang Chen 
Sent: Saturday, March 9, 2019 2:28 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hi Mike,

Yes that's the plan!


From: Mike Freyberger 
Sent: Saturday, March 9, 2019 10:04 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hi Boyang,

Is this work targeted for Kafka 2.3? I am eager to use this new feature.

Thanks,

Mike Freyberger

On 12/21/18, 1:21 PM, "Mayuresh Gharat"  wrote:

Hi Boyang,

Regarding "However, we shall still attempt to remove the member static info
if the given `member.id` points to an existing `group.instance.id` upon
LeaveGroupRequest, because I could think of the possibility that in long
term we could want to add static membership leave group logic for more
fine-grained use cases."

> I think, there is some confusion here. I am probably not putting it
> right.
>
I agree, If a static member sends LeaveGroupRequest, it should be removed
> from the group.
>
Now getting back to downgrade of static membership to Dynamic membership,
> with the example described earlier  (copying it again for ease of reading)
> :
>

>>1. Lets say we have 4 consumers :  c1, c2, c3, c4 in the static group.
>>2. The group.instance.id for each of there are as follows :
>>   - c1 -> gc1, c2 -> gc2, c3 -> gc3, c4 -> gc4
>>3. The mapping on the GroupCordinator would be :
>>   - gc1 -> mc1, gc2 -> mc2, gc3 -> mc3, gc4 -> mc4, where mc1, mc2,
>>   mc3, mc4 are the randomly generated memberIds for c1, c2, c3, c4
>>   respectively, by the GroupCoordinator.
>>4. Now we do a restart to move the group to dynamic membership.
>>5. We bounce c1 first and it rejoins with UNKNOWN_MEMBERID (since we
>>don't persist the previously assigned memberId mc1 anywhere on the 
c1).
>>
> - We agree that there is no way to recognize that c1 was a part of the
> group, *earlier*.  If yes, the statement : "The dynamic member rejoins
> the group without `group.instance.id`. It will be accepted since it is a
> known member." is not necessarily true, right?
>


> - Now I *agree* with "However, we shall still attempt to remove the
> member static info if the given `member.id` points to an existing `
> group.instance.id` upon LeaveGroupRequest, because I could think of the
> possibility that in long term we could want to add static membership leave
> group logic for more fine-grained use cases."
>
But that would only happen if the GroupCoordinator allocates the same
> member.id (mc1) to the consumer c1, when it rejoins the group in step 5
> above as a dynamic member, which is very rare as it is randomly generated,
> but possible.
>


> - This raises another question, if the GroupCoordinator assigns a
> member.id (mc1~) to consumer c1 after step 5. It will join the group and
> rebalance and the group will become stable, eventually. Now the
> GroupCoordinator still maintains a mapping of  "group.instance.id ->
> member.id" (c1 -> gc1, c2 -> gc2, c3 -> gc3, c4 -> gc4) internally and
> after some time, it realizes that it has not received hear

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

Hi Mike,

Yes that's the plan!


From: Mike Freyberger 
Sent: Saturday, March 9, 2019 10:04 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hi Boyang,

Is this work targeted for Kafka 2.3? I am eager to use this new feature.

Thanks,

Mike Freyberger

On 12/21/18, 1:21 PM, "Mayuresh Gharat"  wrote:

Hi Boyang,

Regarding "However, we shall still attempt to remove the member static info
if the given `member.id` points to an existing `group.instance.id` upon
LeaveGroupRequest, because I could think of the possibility that in long
term we could want to add static membership leave group logic for more
fine-grained use cases."

> I think, there is some confusion here. I am probably not putting it
> right.
>
I agree, If a static member sends LeaveGroupRequest, it should be removed
> from the group.
>
Now getting back to downgrade of static membership to Dynamic membership,
> with the example described earlier  (copying it again for ease of reading)
> :
>

>>1. Lets say we have 4 consumers :  c1, c2, c3, c4 in the static group.
>>2. The group.instance.id for each of there are as follows :
>>   - c1 -> gc1, c2 -> gc2, c3 -> gc3, c4 -> gc4
>>3. The mapping on the GroupCordinator would be :
>>   - gc1 -> mc1, gc2 -> mc2, gc3 -> mc3, gc4 -> mc4, where mc1, mc2,
>>   mc3, mc4 are the randomly generated memberIds for c1, c2, c3, c4
>>   respectively, by the GroupCoordinator.
>>4. Now we do a restart to move the group to dynamic membership.
>>5. We bounce c1 first and it rejoins with UNKNOWN_MEMBERID (since we
>>don't persist the previously assigned memberId mc1 anywhere on the 
c1).
>>
> - We agree that there is no way to recognize that c1 was a part of the
> group, *earlier*.  If yes, the statement : "The dynamic member rejoins
> the group without `group.instance.id`. It will be accepted since it is a
> known member." is not necessarily true, right?
>


> - Now I *agree* with "However, we shall still attempt to remove the
> member static info if the given `member.id` points to an existing `
> group.instance.id` upon LeaveGroupRequest, because I could think of the
> possibility that in long term we could want to add static membership leave
> group logic for more fine-grained use cases."
>
But that would only happen if the GroupCoordinator allocates the same
> member.id (mc1) to the consumer c1, when it rejoins the group in step 5
> above as a dynamic member, which is very rare as it is randomly generated,
> but possible.
>


> - This raises another question, if the GroupCoordinator assigns a
> member.id (mc1~) to consumer c1 after step 5. It will join the group and
> rebalance and the group will become stable, eventually. Now the
> GroupCoordinator still maintains a mapping of  "group.instance.id ->
> member.id" (c1 -> gc1, c2 -> gc2, c3 -> gc3, c4 -> gc4) internally and
> after some time, it realizes that it has not received heartbeat from the
> consumer with "group.instance.id" = gc1. In that case, it will trigger
> another rebalance assuming that a static member has left the group (when
> actually it (c1) has not left the group but moved to dynamic membership).
> This can result in multiple rebalances as the same will happen for c2, c3,
> c4.
>

Thoughts ???
One thing, I can think of right now is to run :
removeMemberFromGroup(String groupId, list
groupInstanceIdsToRemove, RemoveMemberFromGroupOptions options)
with groupInstanceIdsToRemove =  once we have bounced
all the members in the group. This assumes that we will be able to complete
the bounces before the GroupCoordinator realizes that it has not received a
heartbeat for any of . This is tricky and error prone.
Will have to think more on this.

Thanks,

Mayuresh

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Mike Freyberger]

Hi Boyang,

Is this work targeted for Kafka 2.3? I am eager to use this new feature.

Thanks,

Mike Freyberger

On 12/21/18, 1:21 PM, "Mayuresh Gharat"  wrote:

Hi Boyang,

Regarding "However, we shall still attempt to remove the member static info
if the given `member.id` points to an existing `group.instance.id` upon
LeaveGroupRequest, because I could think of the possibility that in long
term we could want to add static membership leave group logic for more
fine-grained use cases."

> I think, there is some confusion here. I am probably not putting it
> right.
>
I agree, If a static member sends LeaveGroupRequest, it should be removed
> from the group.
>
Now getting back to downgrade of static membership to Dynamic membership,
> with the example described earlier  (copying it again for ease of reading)
> :
>

>>1. Lets say we have 4 consumers :  c1, c2, c3, c4 in the static group.
>>2. The group.instance.id for each of there are as follows :
>>   - c1 -> gc1, c2 -> gc2, c3 -> gc3, c4 -> gc4
>>3. The mapping on the GroupCordinator would be :
>>   - gc1 -> mc1, gc2 -> mc2, gc3 -> mc3, gc4 -> mc4, where mc1, mc2,
>>   mc3, mc4 are the randomly generated memberIds for c1, c2, c3, c4
>>   respectively, by the GroupCoordinator.
>>4. Now we do a restart to move the group to dynamic membership.
>>5. We bounce c1 first and it rejoins with UNKNOWN_MEMBERID (since we
>>don't persist the previously assigned memberId mc1 anywhere on the 
c1).
>>
> - We agree that there is no way to recognize that c1 was a part of the
> group, *earlier*.  If yes, the statement : "The dynamic member rejoins
> the group without `group.instance.id`. It will be accepted since it is a
> known member." is not necessarily true, right?
>


> - Now I *agree* with "However, we shall still attempt to remove the
> member static info if the given `member.id` points to an existing `
> group.instance.id` upon LeaveGroupRequest, because I could think of the
> possibility that in long term we could want to add static membership leave
> group logic for more fine-grained use cases."
>
But that would only happen if the GroupCoordinator allocates the same
> member.id (mc1) to the consumer c1, when it rejoins the group in step 5
> above as a dynamic member, which is very rare as it is randomly generated,
> but possible.
>


> - This raises another question, if the GroupCoordinator assigns a
> member.id (mc1~) to consumer c1 after step 5. It will join the group and
> rebalance and the group will become stable, eventually. Now the
> GroupCoordinator still maintains a mapping of  "group.instance.id ->
> member.id" (c1 -> gc1, c2 -> gc2, c3 -> gc3, c4 -> gc4) internally and
> after some time, it realizes that it has not received heartbeat from the
> consumer with "group.instance.id" = gc1. In that case, it will trigger
> another rebalance assuming that a static member has left the group (when
> actually it (c1) has not left the group but moved to dynamic membership).
> This can result in multiple rebalances as the same will happen for c2, c3,
> c4.
>

Thoughts ???
One thing, I can think of right now is to run :
removeMemberFromGroup(String groupId, list
groupInstanceIdsToRemove, RemoveMemberFromGroupOptions options)
with groupInstanceIdsToRemove =  once we have bounced
all the members in the group. This assumes that we will be able to complete
the bounces before the GroupCoordinator realizes that it has not received a
heartbeat for any of . This is tricky and error prone.
Will have to think more on this.

Thanks,

Mayuresh

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

Yep, that is correct!

Get Outlook for iOS<https://aka.ms/o0ukef>


From: Mayuresh Gharat 
Sent: Wednesday, January 2, 2019 8:30 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hi Boyang,

Thanks a lot for the reply. I think, I get what you are saying.
IIUC, once the group starts moving to dynamic membership from static
membership, the group will have a list of new member.id's and the earlier
mapping of group.instance.id -> member.id will be outdated and become
inactive from pov of triggering rebalances. This is assuming that the time
interval to move from static to dynamic membership is small.

Thanks,

Mayuresh

On Fri, Dec 28, 2018 at 11:37 PM Boyang Chen  wrote:

> Hey Mayuresh,
>
> thanks for giving another thorough thought on this matter! The session
> timeout is controlled on the member level (or more specifically member.id
> level), which has nothing to do with the static member mapping. Putting
> aside the low possibility of having a duplicate member id assignment
> throughout the restart, if this indeed happens when we switch a member c1
> to dynamic member, the static membership map will only contain [gc1 -> mc1]
> mapping.
> This is because c2~c4 will be evicted during rebalance as their
> corresponding mc2~mc4 are already timeout through heartbeatpugatory. Next
> time the dynamic member restarts while group is in stable, the static map
> will be cleaned up and rebalance will be triggered as expected (we are
> already in dynamic mode now).
>
> I hope this answers your question, thanks!
>
> Boyang
>
> 
> From: Mayuresh Gharat 
> Sent: Saturday, December 22, 2018 2:21 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> Hi Boyang,
>
> Regarding "However, we shall still attempt to remove the member static info
> if the given `member.id` points to an existing `group.instance.id` upon
> LeaveGroupRequest, because I could think of the possibility that in long
> term we could want to add static membership leave group logic for more
> fine-grained use cases."
>
> > I think, there is some confusion here. I am probably not putting it
> > right.
> >
> I agree, If a static member sends LeaveGroupRequest, it should be removed
> > from the group.
> >
> Now getting back to downgrade of static membership to Dynamic membership,
> > with the example described earlier (copying it again for ease of
> reading)
> > :
> >
>
> >> 1. Lets say we have 4 consumers : c1, c2, c3, c4 in the static
> group.
> >> 2. The group.instance.id for each of there are as follows :
> >> - c1 -> gc1, c2 -> gc2, c3 -> gc3, c4 -> gc4
> >> 3. The mapping on the GroupCordinator would be :
> >> - gc1 -> mc1, gc2 -> mc2, gc3 -> mc3, gc4 -> mc4, where mc1, mc2,
> >> mc3, mc4 are the randomly generated memberIds for c1, c2, c3, c4
> >> respectively, by the GroupCoordinator.
> >> 4. Now we do a restart to move the group to dynamic membership.
> >> 5. We bounce c1 first and it rejoins with UNKNOWN_MEMBERID (since we
> >> don't persist the previously assigned memberId mc1 anywhere on the
> c1).
> >>
> > - We agree that there is no way to recognize that c1 was a part of the
> > group, *earlier*. If yes, the statement : "The dynamic member rejoins
> > the group without `group.instance.id`. It will be accepted since it is a
> > known member." is not necessarily true, right?
> >
>
>
> > - Now I *agree* with "However, we shall still attempt to remove the
> > member static info if the given `member.id` points to an existing `
> > group.instance.id` upon LeaveGroupRequest, because I could think of the
> > possibility that in long term we could want to add static membership
> leave
> > group logic for more fine-grained use cases."
> >
> But that would only happen if the GroupCoordinator allocates the same
> > member.id (mc1) to the consumer c1, when it rejoins the group in step 5
> > above as a dynamic member, which is very rare as it is randomly
> generated,
> > but possible.
> >
>
>
> > - This raises another question, if the GroupCoordinator assigns a
> > member.id (mc1~) to consumer c1 after step 5. It will join the group and
> > rebalance and the group will become stable, eventually. Now the
> > GroupCoordinator still maintains a mapping of "group.instance.id ->
> > member.id" (c1 -> gc1, c2 -> gc2, c3 -> gc3, c4 -> gc4) internally and
> > af

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Mayuresh Gharat]

Hi Boyang,

Thanks a lot for the reply. I think, I get what you are saying.
IIUC, once the group starts moving to dynamic membership from static
membership, the group will have a list of new member.id's and the earlier
mapping of group.instance.id -> member.id will be outdated and become
inactive from pov of triggering rebalances. This is assuming that the time
interval to move from static to dynamic membership is small.

Thanks,

Mayuresh

On Fri, Dec 28, 2018 at 11:37 PM Boyang Chen  wrote:

> Hey Mayuresh,
>
> thanks for giving another thorough thought on this matter! The session
> timeout is controlled on the member level (or more specifically member.id
> level), which has nothing to do with the static member mapping. Putting
> aside the low possibility of having a duplicate member id assignment
> throughout the restart, if this indeed happens when we switch a member c1
> to dynamic member, the static membership map will only contain [gc1 -> mc1]
> mapping.
> This is because c2~c4 will be evicted during rebalance as their
> corresponding mc2~mc4 are already timeout through heartbeatpugatory. Next
> time the dynamic member restarts while group is in stable, the static map
> will be cleaned up and rebalance will be triggered as expected (we are
> already in dynamic mode now).
>
> I hope this answers your question, thanks!
>
> Boyang
>
> 
> From: Mayuresh Gharat 
> Sent: Saturday, December 22, 2018 2:21 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> Hi Boyang,
>
> Regarding "However, we shall still attempt to remove the member static info
> if the given `member.id` points to an existing `group.instance.id` upon
> LeaveGroupRequest, because I could think of the possibility that in long
> term we could want to add static membership leave group logic for more
> fine-grained use cases."
>
> > I think, there is some confusion here. I am probably not putting it
> > right.
> >
> I agree, If a static member sends LeaveGroupRequest, it should be removed
> > from the group.
> >
> Now getting back to downgrade of static membership to Dynamic membership,
> > with the example described earlier  (copying it again for ease of
> reading)
> > :
> >
>
> >>1. Lets say we have 4 consumers :  c1, c2, c3, c4 in the static
> group.
> >>2. The group.instance.id for each of there are as follows :
> >>   - c1 -> gc1, c2 -> gc2, c3 -> gc3, c4 -> gc4
> >>3. The mapping on the GroupCordinator would be :
> >>   - gc1 -> mc1, gc2 -> mc2, gc3 -> mc3, gc4 -> mc4, where mc1, mc2,
> >>   mc3, mc4 are the randomly generated memberIds for c1, c2, c3, c4
> >>   respectively, by the GroupCoordinator.
> >>4. Now we do a restart to move the group to dynamic membership.
> >>5. We bounce c1 first and it rejoins with UNKNOWN_MEMBERID (since we
> >>don't persist the previously assigned memberId mc1 anywhere on the
> c1).
> >>
> > - We agree that there is no way to recognize that c1 was a part of the
> > group, *earlier*.  If yes, the statement : "The dynamic member rejoins
> > the group without `group.instance.id`. It will be accepted since it is a
> > known member." is not necessarily true, right?
> >
>
>
> > - Now I *agree* with "However, we shall still attempt to remove the
> > member static info if the given `member.id` points to an existing `
> > group.instance.id` upon LeaveGroupRequest, because I could think of the
> > possibility that in long term we could want to add static membership
> leave
> > group logic for more fine-grained use cases."
> >
> But that would only happen if the GroupCoordinator allocates the same
> > member.id (mc1) to the consumer c1, when it rejoins the group in step 5
> > above as a dynamic member, which is very rare as it is randomly
> generated,
> > but possible.
> >
>
>
> > - This raises another question, if the GroupCoordinator assigns a
> > member.id (mc1~) to consumer c1 after step 5. It will join the group and
> > rebalance and the group will become stable, eventually. Now the
> > GroupCoordinator still maintains a mapping of  "group.instance.id ->
> > member.id" (c1 -> gc1, c2 -> gc2, c3 -> gc3, c4 -> gc4) internally and
> > after some time, it realizes that it has not received heartbeat from the
> > consumer with "group.instance.id" = gc1. In that case, it will trigger
> > another rebalance assuming that a static member has lef

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

Hey Mayuresh,

thanks for giving another thorough thought on this matter! The session timeout 
is controlled on the member level (or more specifically member.id level), which 
has nothing to do with the static member mapping. Putting aside the low 
possibility of having a duplicate member id assignment throughout the restart, 
if this indeed happens when we switch a member c1 to dynamic member, the static 
membership map will only contain [gc1 -> mc1] mapping.
This is because c2~c4 will be evicted during rebalance as their corresponding 
mc2~mc4 are already timeout through heartbeatpugatory. Next time the dynamic 
member restarts while group is in stable, the static map will be cleaned up and 
rebalance will be triggered as expected (we are already in dynamic mode now).

I hope this answers your question, thanks!

Boyang


From: Mayuresh Gharat 
Sent: Saturday, December 22, 2018 2:21 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hi Boyang,

Regarding "However, we shall still attempt to remove the member static info
if the given `member.id` points to an existing `group.instance.id` upon
LeaveGroupRequest, because I could think of the possibility that in long
term we could want to add static membership leave group logic for more
fine-grained use cases."

> I think, there is some confusion here. I am probably not putting it
> right.
>
I agree, If a static member sends LeaveGroupRequest, it should be removed
> from the group.
>
Now getting back to downgrade of static membership to Dynamic membership,
> with the example described earlier  (copying it again for ease of reading)
> :
>

>>1. Lets say we have 4 consumers :  c1, c2, c3, c4 in the static group.
>>2. The group.instance.id for each of there are as follows :
>>   - c1 -> gc1, c2 -> gc2, c3 -> gc3, c4 -> gc4
>>3. The mapping on the GroupCordinator would be :
>>   - gc1 -> mc1, gc2 -> mc2, gc3 -> mc3, gc4 -> mc4, where mc1, mc2,
>>   mc3, mc4 are the randomly generated memberIds for c1, c2, c3, c4
>>   respectively, by the GroupCoordinator.
>>4. Now we do a restart to move the group to dynamic membership.
>>5. We bounce c1 first and it rejoins with UNKNOWN_MEMBERID (since we
>>don't persist the previously assigned memberId mc1 anywhere on the c1).
>>
> - We agree that there is no way to recognize that c1 was a part of the
> group, *earlier*.  If yes, the statement : "The dynamic member rejoins
> the group without `group.instance.id`. It will be accepted since it is a
> known member." is not necessarily true, right?
>


> - Now I *agree* with "However, we shall still attempt to remove the
> member static info if the given `member.id` points to an existing `
> group.instance.id` upon LeaveGroupRequest, because I could think of the
> possibility that in long term we could want to add static membership leave
> group logic for more fine-grained use cases."
>
But that would only happen if the GroupCoordinator allocates the same
> member.id (mc1) to the consumer c1, when it rejoins the group in step 5
> above as a dynamic member, which is very rare as it is randomly generated,
> but possible.
>


> - This raises another question, if the GroupCoordinator assigns a
> member.id (mc1~) to consumer c1 after step 5. It will join the group and
> rebalance and the group will become stable, eventually. Now the
> GroupCoordinator still maintains a mapping of  "group.instance.id ->
> member.id" (c1 -> gc1, c2 -> gc2, c3 -> gc3, c4 -> gc4) internally and
> after some time, it realizes that it has not received heartbeat from the
> consumer with "group.instance.id" = gc1. In that case, it will trigger
> another rebalance assuming that a static member has left the group (when
> actually it (c1) has not left the group but moved to dynamic membership).
> This can result in multiple rebalances as the same will happen for c2, c3,
> c4.
>

Thoughts ???
One thing, I can think of right now is to run :
removeMemberFromGroup(String groupId, list
groupInstanceIdsToRemove, RemoveMemberFromGroupOptions options)
with groupInstanceIdsToRemove =  once we have bounced
all the members in the group. This assumes that we will be able to complete
the bounces before the GroupCoordinator realizes that it has not received a
heartbeat for any of . This is tricky and error prone.
Will have to think more on this.

Thanks,

Mayuresh

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Mayuresh Gharat]

Hi Boyang,

Regarding "However, we shall still attempt to remove the member static info
if the given `member.id` points to an existing `group.instance.id` upon
LeaveGroupRequest, because I could think of the possibility that in long
term we could want to add static membership leave group logic for more
fine-grained use cases."

> I think, there is some confusion here. I am probably not putting it
> right.
>
I agree, If a static member sends LeaveGroupRequest, it should be removed
> from the group.
>
Now getting back to downgrade of static membership to Dynamic membership,
> with the example described earlier  (copying it again for ease of reading)
> :
>

>>1. Lets say we have 4 consumers :  c1, c2, c3, c4 in the static group.
>>2. The group.instance.id for each of there are as follows :
>>   - c1 -> gc1, c2 -> gc2, c3 -> gc3, c4 -> gc4
>>3. The mapping on the GroupCordinator would be :
>>   - gc1 -> mc1, gc2 -> mc2, gc3 -> mc3, gc4 -> mc4, where mc1, mc2,
>>   mc3, mc4 are the randomly generated memberIds for c1, c2, c3, c4
>>   respectively, by the GroupCoordinator.
>>4. Now we do a restart to move the group to dynamic membership.
>>5. We bounce c1 first and it rejoins with UNKNOWN_MEMBERID (since we
>>don't persist the previously assigned memberId mc1 anywhere on the c1).
>>
> - We agree that there is no way to recognize that c1 was a part of the
> group, *earlier*.  If yes, the statement : "The dynamic member rejoins
> the group without `group.instance.id`. It will be accepted since it is a
> known member." is not necessarily true, right?
>


> - Now I *agree* with "However, we shall still attempt to remove the
> member static info if the given `member.id` points to an existing `
> group.instance.id` upon LeaveGroupRequest, because I could think of the
> possibility that in long term we could want to add static membership leave
> group logic for more fine-grained use cases."
>
But that would only happen if the GroupCoordinator allocates the same
> member.id (mc1) to the consumer c1, when it rejoins the group in step 5
> above as a dynamic member, which is very rare as it is randomly generated,
> but possible.
>


> - This raises another question, if the GroupCoordinator assigns a
> member.id (mc1~) to consumer c1 after step 5. It will join the group and
> rebalance and the group will become stable, eventually. Now the
> GroupCoordinator still maintains a mapping of  "group.instance.id ->
> member.id" (c1 -> gc1, c2 -> gc2, c3 -> gc3, c4 -> gc4) internally and
> after some time, it realizes that it has not received heartbeat from the
> consumer with "group.instance.id" = gc1. In that case, it will trigger
> another rebalance assuming that a static member has left the group (when
> actually it (c1) has not left the group but moved to dynamic membership).
> This can result in multiple rebalances as the same will happen for c2, c3,
> c4.
>

Thoughts ???
One thing, I can think of right now is to run :
removeMemberFromGroup(String groupId, list
groupInstanceIdsToRemove, RemoveMemberFromGroupOptions options)
with groupInstanceIdsToRemove =  once we have bounced
all the members in the group. This assumes that we will be able to complete
the bounces before the GroupCoordinator realizes that it has not received a
heartbeat for any of . This is tricky and error prone.
Will have to think more on this.

Thanks,

Mayuresh

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Guozhang Wang]

Yes, I think it makes sense to let KafkaStreams to expose embedded consumer
client-id and instance-ids, e.g. at the ThreadMetadata exposed via
`localThreadsMetadata`.

I can open up a new KIP for the change on Streams side so that we can keep
this KIP as on consumer side only.


Guozhang


On Fri, Nov 30, 2018 at 9:31 PM Boyang Chen  wrote:

> Thanks Guozhang and Mayuresh for the followup here.
> > Also I was thinking if we can have a replace API, that takes in a map of
> > old to new instance Ids. Such that we can replace a consumer.
> > IF we have this api, and if a consumer host goes down due to hardware
> > issues, we can have another host spin up and take its place. This is
> like a
> > cold backup which can be a step towards providing the hot backup that we
> > discussed earlier in the KIP.
> I like Mayuresh's suggestion, and I think we could prepare follow-up work
> once 345 is done to add a replace API. For the
> very first version I feel this is not a must-have.
>
> For Streams, I think we do not need an extra config for the instance id,
> instead, we can re-use the way we construct the embedded consumer's client
> id as:
>
> [streams client-id] + "-StreamThread-" + [thread-id] + "-consumer"
>
> So as long as user's specify the unique streams client-id, the resulted
> consumer client-id / instance-id should be unique as well already.
>
> So Guozhang you mean stream will enable static membership automatically
> correct? That would make the logic simpler
> and fewer code change on stream side.
>
> As for the LeaveGroupRequest, as I understand it, your concern is that when
> we are shutting down a single Streams instance that may contain multiple
> threads, shutting down that instance would mean shutting down multiple
> members. Personally I'd prefer to make the LeaveGroupRequest API more
> general and less inclined to Streams (I think Mayuresh also suggested
> this). So I'd suggest that we keep the LeaveGroupRequest API as suggested,
> i.e. a list of member.instance.ids. And in Streams we can add a new API in
> KafkaStreams to expose:
>
> 1) the list of embedded consumer / producer client ids,
> 2) the producer's txn ids if EOS is turned on, and
> 3) the consumer's instance ids.
>
> I agree with the suggestion to make the leave group request change
> generic. So this new Stream API
> will be added on the rest layer to expose the necessary ids correct?
>
> Looking forward to your confirmation 
>
> Best,
> Boyang
>
> --------------
> *From:* Guozhang Wang 
> *Sent:* Saturday, December 1, 2018 7:00 AM
> *To:* dev
> *Subject:* Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> Hi Boyang,
>
> For Streams, I think we do not need an extra config for the instance id,
> instead, we can re-use the way we construct the embedded consumer's client
> id as:
>
> [streams client-id] + "-StreamThread-" + [thread-id] + "-consumer"
>
> So as long as user's specify the unique streams client-id, the resulted
> consumer client-id / instance-id should be unique as well already.
>
> As for the LeaveGroupRequest, as I understand it, your concern is that when
> we are shutting down a single Streams instance that may contain multiple
> threads, shutting down that instance would mean shutting down multiple
> members. Personally I'd prefer to make the LeaveGroupRequest API more
> general and less inclined to Streams (I think Mayuresh also suggested
> this). So I'd suggest that we keep the LeaveGroupRequest API as suggested,
> i.e. a list of member.instance.ids. And in Streams we can add a new API in
> KafkaStreams to expose:
>
> 1) the list of embedded consumer / producer client ids,
> 2) the producer's txn ids if EOS is turned on, and
> 3) the consumer's instance ids.
>
> So that Streams operators can read those values from KafkaStreams directly
> before shutting it down and use the list in the LeaveGroupRequest API. How
> about that?
>
>
> Guozhang
>
>
> On Fri, Nov 30, 2018 at 7:45 AM Mayuresh Gharat <
> gharatmayures...@gmail.com>
> wrote:
>
> > I like Guozhang's suggestion to not have to wait for session timeout in
> > case we know that we want to downsize the consumer group and redistribute
> > the partitions among the remaining consumers.
> > IIUC, with the above suggestions, the admin api
> > "removeMemberFromGroup(groupId, list[instanceId])" or
> > "removeMemberFromGroup(groupId, instanceId)", will automatically cause a
> > rebalance, right?
> > I would prefer ist[instanceid] because that's more general scenario.
> >
> > Also I was thinking 

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Mayuresh Gharat]

Hi Boyang,

Sounds good to me. We can keep this first version simple.
I can also start working on a follow up KIP for hotback and cold backup in
the meantime this KIP goes in once finalized, if its OK with you.

Thanks,

Mayuresh

On Fri, Nov 30, 2018 at 9:31 PM Boyang Chen  wrote:

> Thanks Guozhang and Mayuresh for the followup here.
> > Also I was thinking if we can have a replace API, that takes in a map of
> > old to new instance Ids. Such that we can replace a consumer.
> > IF we have this api, and if a consumer host goes down due to hardware
> > issues, we can have another host spin up and take its place. This is
> like a
> > cold backup which can be a step towards providing the hot backup that we
> > discussed earlier in the KIP.
> I like Mayuresh's suggestion, and I think we could prepare follow-up work
> once 345 is done to add a replace API. For the
> very first version I feel this is not a must-have.
>
> For Streams, I think we do not need an extra config for the instance id,
> instead, we can re-use the way we construct the embedded consumer's client
> id as:
>
> [streams client-id] + "-StreamThread-" + [thread-id] + "-consumer"
>
> So as long as user's specify the unique streams client-id, the resulted
> consumer client-id / instance-id should be unique as well already.
> So Guozhang you mean stream will enable static membership automatically
> correct? That would make the logic simpler
> and fewer code change on stream side.
>
> As for the LeaveGroupRequest, as I understand it, your concern is that when
> we are shutting down a single Streams instance that may contain multiple
> threads, shutting down that instance would mean shutting down multiple
> members. Personally I'd prefer to make the LeaveGroupRequest API more
> general and less inclined to Streams (I think Mayuresh also suggested
> this). So I'd suggest that we keep the LeaveGroupRequest API as suggested,
> i.e. a list of member.instance.ids. And in Streams we can add a new API in
> KafkaStreams to expose:
>
> 1) the list of embedded consumer / producer client ids,
> 2) the producer's txn ids if EOS is turned on, and
> 3) the consumer's instance ids.
> I agree with the suggestion to make the leave group request change
> generic. So this new Stream API
> will be added on the rest layer to expose the necessary ids correct?
>
> Looking forward to your confirmation 
>
> Best,
> Boyang
>
> ____________
> From: Guozhang Wang 
> Sent: Saturday, December 1, 2018 7:00 AM
> To: dev
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> Hi Boyang,
>
> For Streams, I think we do not need an extra config for the instance id,
> instead, we can re-use the way we construct the embedded consumer's client
> id as:
>
> [streams client-id] + "-StreamThread-" + [thread-id] + "-consumer"
>
> So as long as user's specify the unique streams client-id, the resulted
> consumer client-id / instance-id should be unique as well already.
>
> As for the LeaveGroupRequest, as I understand it, your concern is that when
> we are shutting down a single Streams instance that may contain multiple
> threads, shutting down that instance would mean shutting down multiple
> members. Personally I'd prefer to make the LeaveGroupRequest API more
> general and less inclined to Streams (I think Mayuresh also suggested
> this). So I'd suggest that we keep the LeaveGroupRequest API as suggested,
> i.e. a list of member.instance.ids. And in Streams we can add a new API in
> KafkaStreams to expose:
>
> 1) the list of embedded consumer / producer client ids,
> 2) the producer's txn ids if EOS is turned on, and
> 3) the consumer's instance ids.
>
> So that Streams operators can read those values from KafkaStreams directly
> before shutting it down and use the list in the LeaveGroupRequest API. How
> about that?
>
>
> Guozhang
>
>
> On Fri, Nov 30, 2018 at 7:45 AM Mayuresh Gharat <
> gharatmayures...@gmail.com>
> wrote:
>
> > I like Guozhang's suggestion to not have to wait for session timeout in
> > case we know that we want to downsize the consumer group and redistribute
> > the partitions among the remaining consumers.
> > IIUC, with the above suggestions, the admin api
> > "removeMemberFromGroup(groupId, list[instanceId])" or
> > "removeMemberFromGroup(groupId, instanceId)", will automatically cause a
> > rebalance, right?
> > I would prefer ist[instanceid] because that's more general scenario.
> >
> > Also I was thinking if we can have a replace API, that takes in a map of
> > old to new instance 

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Guozhang Wang]

Hi Boyang,

For Streams, I think we do not need an extra config for the instance id,
instead, we can re-use the way we construct the embedded consumer's client
id as:

[streams client-id] + "-StreamThread-" + [thread-id] + "-consumer"

So as long as user's specify the unique streams client-id, the resulted
consumer client-id / instance-id should be unique as well already.

As for the LeaveGroupRequest, as I understand it, your concern is that when
we are shutting down a single Streams instance that may contain multiple
threads, shutting down that instance would mean shutting down multiple
members. Personally I'd prefer to make the LeaveGroupRequest API more
general and less inclined to Streams (I think Mayuresh also suggested
this). So I'd suggest that we keep the LeaveGroupRequest API as suggested,
i.e. a list of member.instance.ids. And in Streams we can add a new API in
KafkaStreams to expose:

1) the list of embedded consumer / producer client ids,
2) the producer's txn ids if EOS is turned on, and
3) the consumer's instance ids.

So that Streams operators can read those values from KafkaStreams directly
before shutting it down and use the list in the LeaveGroupRequest API. How
about that?


Guozhang


On Fri, Nov 30, 2018 at 7:45 AM Mayuresh Gharat 
wrote:

> I like Guozhang's suggestion to not have to wait for session timeout in
> case we know that we want to downsize the consumer group and redistribute
> the partitions among the remaining consumers.
> IIUC, with the above suggestions, the admin api
> "removeMemberFromGroup(groupId, list[instanceId])" or
> "removeMemberFromGroup(groupId, instanceId)", will automatically cause a
> rebalance, right?
> I would prefer ist[instanceid] because that's more general scenario.
>
> Also I was thinking if we can have a replace API, that takes in a map of
> old to new instance Ids. Such that we can replace a consumer.
> IF we have this api, and if a consumer host goes down due to hardware
> issues, we can have another host spin up and take its place. This is like a
> cold backup which can be a step towards providing the hot backup that we
> discussed earlier in the KIP.
> Thoughts?
>
> Thanks,
>
> Mayuresh
>
> On Thu, Nov 29, 2018 at 1:30 AM Boyang Chen  wrote:
>
> > In fact I feel that it's more convenient for user to specify a list of
> > instance id prefixes. Because
> > for general consumer application we couldn't always find a proper prefix
> > to remove a list of consumers.
> > So we are either adding list[instanceid prefix], or we could add two
> > fields: instanceid prefix, and list[instanceid]
> > for clarity purpose. As you know, two options are equivalent since full
> > name is subset of prefix.
> >
> > Let me know your thoughts!
> >
> > Boyang
> > ____
> > From: Boyang Chen 
> > Sent: Thursday, November 29, 2018 3:39 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> > specifying member id
> >
> > Thanks Guozhang for the new proposal here!
> >
> > So I'd like to propose a slightly modified version of LeaveGroupRequest:
> > instead of letting the static member consumer client themselves to send
> the
> > request (which means we still need to have some hidden configs to turn it
> > off like we did today), how about just letting any other client to send
> > this request since the LeaveGroupRequest only requires group.id and
> > member.id? So back to your operational scenarios, if some static member
> > has
> > been found crashed and it is not likely to comeback, or we simply want to
> > shrink the size of the group by shutting down some static members, we can
> > use an admin client to send the LeaveGroupRequest after the instance has
> > been completely shutdown or crashed to kick them out of the group and
> also
> > triggers the rebalance.
> >
> > One issue though, is that users may not know the member id required in
> the
> > LeaveGroupRequest. To work around it we can add the `group.instance.id`
> > along with the member id as well and then allow member id null-able. The
> > coordinator logic would then be modified as 1) if member.id is
> specified,
> > ignore instance.id and always use member.id to find the member to kick
> > out,
> > 2) otherwise, try with the instance.id to find the corresponding
> member.id
> > and kick it out, 3) if none is found, reject with an error code.
> >
> > So in sum the alternative changes are:
> >
> > a) Modify LeaveGroupRequest to add group.instance.id
> > b) Modify coordinator logic to handle such request on the broker side.

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Mayuresh Gharat]

I like Guozhang's suggestion to not have to wait for session timeout in
case we know that we want to downsize the consumer group and redistribute
the partitions among the remaining consumers.
IIUC, with the above suggestions, the admin api
"removeMemberFromGroup(groupId, list[instanceId])" or
"removeMemberFromGroup(groupId, instanceId)", will automatically cause a
rebalance, right?
I would prefer ist[instanceid] because that's more general scenario.

Also I was thinking if we can have a replace API, that takes in a map of
old to new instance Ids. Such that we can replace a consumer.
IF we have this api, and if a consumer host goes down due to hardware
issues, we can have another host spin up and take its place. This is like a
cold backup which can be a step towards providing the hot backup that we
discussed earlier in the KIP.
Thoughts?

Thanks,

Mayuresh

On Thu, Nov 29, 2018 at 1:30 AM Boyang Chen  wrote:

> In fact I feel that it's more convenient for user to specify a list of
> instance id prefixes. Because
> for general consumer application we couldn't always find a proper prefix
> to remove a list of consumers.
> So we are either adding list[instanceid prefix], or we could add two
> fields: instanceid prefix, and list[instanceid]
> for clarity purpose. As you know, two options are equivalent since full
> name is subset of prefix.
>
> Let me know your thoughts!
>
> Boyang
> 
> From: Boyang Chen 
> Sent: Thursday, November 29, 2018 3:39 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> Thanks Guozhang for the new proposal here!
>
> So I'd like to propose a slightly modified version of LeaveGroupRequest:
> instead of letting the static member consumer client themselves to send the
> request (which means we still need to have some hidden configs to turn it
> off like we did today), how about just letting any other client to send
> this request since the LeaveGroupRequest only requires group.id and
> member.id? So back to your operational scenarios, if some static member
> has
> been found crashed and it is not likely to comeback, or we simply want to
> shrink the size of the group by shutting down some static members, we can
> use an admin client to send the LeaveGroupRequest after the instance has
> been completely shutdown or crashed to kick them out of the group and also
> triggers the rebalance.
>
> One issue though, is that users may not know the member id required in the
> LeaveGroupRequest. To work around it we can add the `group.instance.id`
> along with the member id as well and then allow member id null-able. The
> coordinator logic would then be modified as 1) if member.id is specified,
> ignore instance.id and always use member.id to find the member to kick
> out,
> 2) otherwise, try with the instance.id to find the corresponding member.id
> and kick it out, 3) if none is found, reject with an error code.
>
> So in sum the alternative changes are:
>
> a) Modify LeaveGroupRequest to add group.instance.id
> b) Modify coordinator logic to handle such request on the broker side.
> c) Add a new API in AdminClient like "removeMemberFromGroup(groupId,
> instanceId)" which will be translated as a LeaveGroupRequest.
> d) [Optional] we can even batch the request by allowing
> "removeMemberFromGroup(groupId, list[instanceId])" and then make `
> member.id`
> and `instance.id` field of LeaveGroupRequest to be an array instead of a
> single entry.
> e) We can also remove the admin ConsumerRebalanceRequest as well for
> simplicity (why not? paranoid of having as less request protocols as
> possible :), as it is not needed anymore with the above proposal.
> I agree that reusing LeaveGroupRequest is actually a good idea: we only
> need to iterate
> over an existing request format. Also I found that we haven't discussed
> how we want to enable
> this feature on Streaming applications, which is different from common
> consumer application in that
> Stream app uses stream thread as individual consumer.
> For example if user specifies the client id, the stream consumer client id
> will be like:
> User client id + "-StreamThread-" + thread id + "-consumer"
>
> So I'm thinking we should do sth similar for defining group.instance.id
> on Stream. We shall define another
> config called `stream.instance.id` which would be used as prefix, and for
> each thread consumer the formula
> will look like:
> `group.instance.id` = `stream.instance.id` + "-" + thread id + "-consumer"
>
> And for the ease of use, the interface of leave group request could
> include `group.instance.id.prefix` instead of
> `g

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

In fact I feel that it's more convenient for user to specify a list of instance 
id prefixes. Because
for general consumer application we couldn't always find a proper prefix to 
remove a list of consumers.
So we are either adding list[instanceid prefix], or we could add two fields: 
instanceid prefix, and list[instanceid]
for clarity purpose. As you know, two options are equivalent since full name is 
subset of prefix.

Let me know your thoughts!

Boyang

From: Boyang Chen 
Sent: Thursday, November 29, 2018 3:39 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Thanks Guozhang for the new proposal here!

So I'd like to propose a slightly modified version of LeaveGroupRequest:
instead of letting the static member consumer client themselves to send the
request (which means we still need to have some hidden configs to turn it
off like we did today), how about just letting any other client to send
this request since the LeaveGroupRequest only requires group.id and
member.id? So back to your operational scenarios, if some static member has
been found crashed and it is not likely to comeback, or we simply want to
shrink the size of the group by shutting down some static members, we can
use an admin client to send the LeaveGroupRequest after the instance has
been completely shutdown or crashed to kick them out of the group and also
triggers the rebalance.

One issue though, is that users may not know the member id required in the
LeaveGroupRequest. To work around it we can add the `group.instance.id`
along with the member id as well and then allow member id null-able. The
coordinator logic would then be modified as 1) if member.id is specified,
ignore instance.id and always use member.id to find the member to kick out,
2) otherwise, try with the instance.id to find the corresponding member.id
and kick it out, 3) if none is found, reject with an error code.

So in sum the alternative changes are:

a) Modify LeaveGroupRequest to add group.instance.id
b) Modify coordinator logic to handle such request on the broker side.
c) Add a new API in AdminClient like "removeMemberFromGroup(groupId,
instanceId)" which will be translated as a LeaveGroupRequest.
d) [Optional] we can even batch the request by allowing
"removeMemberFromGroup(groupId, list[instanceId])" and then make `member.id`
and `instance.id` field of LeaveGroupRequest to be an array instead of a
single entry.
e) We can also remove the admin ConsumerRebalanceRequest as well for
simplicity (why not? paranoid of having as less request protocols as
possible :), as it is not needed anymore with the above proposal.
I agree that reusing LeaveGroupRequest is actually a good idea: we only need to 
iterate
over an existing request format. Also I found that we haven't discussed how we 
want to enable
this feature on Streaming applications, which is different from common consumer 
application in that
Stream app uses stream thread as individual consumer.
For example if user specifies the client id, the stream consumer client id will 
be like:
User client id + "-StreamThread-" + thread id + "-consumer"

So I'm thinking we should do sth similar for defining group.instance.id on 
Stream. We shall define another
config called `stream.instance.id` which would be used as prefix, and for each 
thread consumer the formula
will look like:
`group.instance.id` = `stream.instance.id` + "-" + thread id + "-consumer"

And for the ease of use, the interface of leave group request could include 
`group.instance.id.prefix` instead of
`group.instance.id` so that we could batch remove consumers relating to a 
single stream instance. This is more intuitive
and flexible since specifying names of 16~32 * n (n = number of stream 
instances to shut down) consumers is not an easy
job without client management tooling.

How does this workaround sound?

Boyang

From: Guozhang Wang 
Sent: Thursday, November 29, 2018 2:38 AM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hi Boyang,

I was thinking that with the optional static members in the admin
ConsumerRebalanceRequest it should be sufficient to kick out the static
member before their session timeout (arguably long in practice) have not
reached. But now I see your concern is that in some situations the admin
operators may not even know the full list of static members, but ONLY know
which static member has failed and hence would like to kick out of the
group.

So I'd like to propose a slightly modified version of LeaveGroupRequest:
instead of letting the static member consumer client themselves to send the
request (which means we still need to have some hidden configs to turn it
off like we did today), how about just letting any other client to send
this request since the LeaveGroupRequest only 

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

Thanks Guozhang for the new proposal here!

So I'd like to propose a slightly modified version of LeaveGroupRequest:
instead of letting the static member consumer client themselves to send the
request (which means we still need to have some hidden configs to turn it
off like we did today), how about just letting any other client to send
this request since the LeaveGroupRequest only requires group.id and
member.id? So back to your operational scenarios, if some static member has
been found crashed and it is not likely to comeback, or we simply want to
shrink the size of the group by shutting down some static members, we can
use an admin client to send the LeaveGroupRequest after the instance has
been completely shutdown or crashed to kick them out of the group and also
triggers the rebalance.

One issue though, is that users may not know the member id required in the
LeaveGroupRequest. To work around it we can add the `group.instance.id`
along with the member id as well and then allow member id null-able. The
coordinator logic would then be modified as 1) if member.id is specified,
ignore instance.id and always use member.id to find the member to kick out,
2) otherwise, try with the instance.id to find the corresponding member.id
and kick it out, 3) if none is found, reject with an error code.

So in sum the alternative changes are:

a) Modify LeaveGroupRequest to add group.instance.id
b) Modify coordinator logic to handle such request on the broker side.
c) Add a new API in AdminClient like "removeMemberFromGroup(groupId,
instanceId)" which will be translated as a LeaveGroupRequest.
d) [Optional] we can even batch the request by allowing
"removeMemberFromGroup(groupId, list[instanceId])" and then make `member.id`
and `instance.id` field of LeaveGroupRequest to be an array instead of a
single entry.
e) We can also remove the admin ConsumerRebalanceRequest as well for
simplicity (why not? paranoid of having as less request protocols as
possible :), as it is not needed anymore with the above proposal.
I agree that reusing LeaveGroupRequest is actually a good idea: we only need to 
iterate
over an existing request format. Also I found that we haven't discussed how we 
want to enable
this feature on Streaming applications, which is different from common consumer 
application in that
Stream app uses stream thread as individual consumer.
For example if user specifies the client id, the stream consumer client id will 
be like:
User client id + "-StreamThread-" + thread id + "-consumer"

So I'm thinking we should do sth similar for defining group.instance.id on 
Stream. We shall define another
config called `stream.instance.id` which would be used as prefix, and for each 
thread consumer the formula
will look like:
`group.instance.id` = `stream.instance.id` + "-" + thread id + "-consumer"

And for the ease of use, the interface of leave group request could include 
`group.instance.id.prefix` instead of
`group.instance.id` so that we could batch remove consumers relating to a 
single stream instance. This is more intuitive
and flexible since specifying names of 16~32 * n (n = number of stream 
instances to shut down) consumers is not an easy
job without client management tooling.

How does this workaround sound?

Boyang

From: Guozhang Wang 
Sent: Thursday, November 29, 2018 2:38 AM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hi Boyang,

I was thinking that with the optional static members in the admin
ConsumerRebalanceRequest it should be sufficient to kick out the static
member before their session timeout (arguably long in practice) have not
reached. But now I see your concern is that in some situations the admin
operators may not even know the full list of static members, but ONLY know
which static member has failed and hence would like to kick out of the
group.

So I'd like to propose a slightly modified version of LeaveGroupRequest:
instead of letting the static member consumer client themselves to send the
request (which means we still need to have some hidden configs to turn it
off like we did today), how about just letting any other client to send
this request since the LeaveGroupRequest only requires group.id and
member.id? So back to your operational scenarios, if some static member has
been found crashed and it is not likely to comeback, or we simply want to
shrink the size of the group by shutting down some static members, we can
use an admin client to send the LeaveGroupRequest after the instance has
been completely shutdown or crashed to kick them out of the group and also
triggers the rebalance.

One issue though, is that users may not know the member id required in the
LeaveGroupRequest. To work around it we can add the `group.instance.id`
along with the member id as well and then allow member id null-able. The
coordinator logic would then be 

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Guozhang Wang]

GroupCoordinator
> > that consumer c4 has been kicked out of the groupId = "GroupA"?
> Previous proposal should be suffice to answer this question 
>
> - Also it looks like the statement "If the `member.id` uses
> > UNKNOWN_MEMBER_NAME, we shall always generate a new member id and replace
> > the one within current map, if `group.member.name` is known. Also once
> we
> > are done with KIP-394
> > <
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-394%253A%2BRequire%2Bmember.id%2Bfor%2Binitial%2Bjoin%2Bgroup%2Brequestdata=02%7C01%7C%7Ccd1e9e2eee0f440987bf08d654aa1dd9%7C84df9e7fe9f640afb435%7C1%7C0%7C636789486977454595sdata=lEv9SuBZgATGhOSe5zUr%2Fqudycoh%2FwfdM%2FhPH5Hp1N4%3Dreserved=0
> > >,
> > all the join group requests are requiring `member.id` to physically
> enter
> > the consumer group. This way the latest joined " is incomplete. Can you
> > take a look at this?
> > Also when we say "all the join group requests are requiring `member.id`
> to
> > physically enter the consumer group." because a newly started consumer
> will
> > not have a "member.id", I assume you mean, once the GroupCoordinator
> > assigns a member.id to the newly started consumer, it has to use it for
> > any
> > future JoinGroupRequests. Is my understanding correct?
> >
> Thanks for catching it! And yes, we shall use one extra round-trip between
> consumer
> and broker to inform the new member id allocation.
>
> Next is the replies to Guozhang's comment:
> 2) I once have a discussion about the LeaveGroupRequest for static members,
> and the reason for not having it for static members is that we'd need to
> make it a configurable behavior as well (i.e. the likelihood that a static
> member may shutdown but come back later may be even larger than the
> likelihood that a shutdown static member would not come back), and when a
> shutdown is complete the instance cannot tell whether or not it will come
> back by itself. And hence letting a third party (think: admin used by K8s
> plugins) issuing a request to indicate static member changes would be more
> plausible.
>
> I think having an optional list of all the static members that are still in
> the group, rather than the members to be removed since the latter looks a
> bit less flexible to me, in the request is a good idea (remember we allow a
> group to have both static and dynamic members at the same time, so when
> receiving the request, we will only do the diff and add / remove the static
> members directly only, while still let the dynamic members to try to
> re-join the group with the rebalance timeout).
> I'm also in favor of storing all the in-group static members. In fact we
> could reuse
> the static membership mapping to store this information. Do you think
> that we should let static member send leave group request to indicate
> their status of "leaving",
> and use ConsumerRebalanceRequest to trigger rebalance without them? I'm
> suggesting we should
> remove those members when kicking off rebalance since we are shutting them
> down already.
>
> 3) personally I favor "ids" over "names" :) Since we already have some
> "ids" and hence it sounds more consistent, plus on the producer side we
> have a `transactional.id` whose semantics is a bit similar to this one,
> i.e. for unique distinguishment of a client which may comes and goes but
> need to be persist over multiple "instance life-times".
> Sure we have enough votes for ids I will finalize the name to `
> group.instance.id`, does that
> sound good?
>
> Best,
> Boyang
> 
> From: Guozhang Wang 
> Sent: Wednesday, November 28, 2018 4:51 AM
> To: dev
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> Regarding Jason's question and Boyang's responses:
>
> 2) I once have a discussion about the LeaveGroupRequest for static members,
> and the reason for not having it for static members is that we'd need to
> make it a configurable behavior as well (i.e. the likelihood that a static
> member may shutdown but come back later may be even larger than the
> likelihood that a shutdown static member would not come back), and when a
> shutdown is complete the instance cannot tell whether or not it will come
> back by itself. And hence letting a third party (think: admin used by K8s
> plugins) issuing a request to indicate static member changes would be more
> plausible.
>
> I think having an optional list of all the static members that are still in
> the group, rather than the

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

reuse
the static membership mapping to store this information. Do you think
that we should let static member send leave group request to indicate their 
status of "leaving",
and use ConsumerRebalanceRequest to trigger rebalance without them? I'm 
suggesting we should
remove those members when kicking off rebalance since we are shutting them down 
already.

3) personally I favor "ids" over "names" :) Since we already have some
"ids" and hence it sounds more consistent, plus on the producer side we
have a `transactional.id` whose semantics is a bit similar to this one,
i.e. for unique distinguishment of a client which may comes and goes but
need to be persist over multiple "instance life-times".
Sure we have enough votes for ids I will finalize the name to 
`group.instance.id`, does that
sound good?

Best,
Boyang

From: Guozhang Wang 
Sent: Wednesday, November 28, 2018 4:51 AM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Regarding Jason's question and Boyang's responses:

2) I once have a discussion about the LeaveGroupRequest for static members,
and the reason for not having it for static members is that we'd need to
make it a configurable behavior as well (i.e. the likelihood that a static
member may shutdown but come back later may be even larger than the
likelihood that a shutdown static member would not come back), and when a
shutdown is complete the instance cannot tell whether or not it will come
back by itself. And hence letting a third party (think: admin used by K8s
plugins) issuing a request to indicate static member changes would be more
plausible.

I think having an optional list of all the static members that are still in
the group, rather than the members to be removed since the latter looks a
bit less flexible to me, in the request is a good idea (remember we allow a
group to have both static and dynamic members at the same time, so when
receiving the request, we will only do the diff and add / remove the static
members directly only, while still let the dynamic members to try to
re-join the group with the rebalance timeout).

3) personally I favor "ids" over "names" :) Since we already have some
"ids" and hence it sounds more consistent, plus on the producer side we
have a `transactional.id` whose semantics is a bit similar to this one,
i.e. for unique distinguishment of a client which may comes and goes but
need to be persist over multiple "instance life-times".


Guozhang


On Tue, Nov 27, 2018 at 10:00 AM Mayuresh Gharat 
wrote:

> Hi Boyang,
>
> Thanks for the replies. Please find the follow up queries below.
>
> 5. Regarding "So in summary, *the member will only be removed due to
> session timeout*. We shall remove it from both in-memory static member name
> mapping and member list." If the rebalance is invoked manually using the
> the admin apis, how long should the group coordinator wait for the members
> of the group to send a JoinGroupRequest for participating in the rebalance?
> How is a lagging consumer handled?
> The plan is to disable member kick out when rebalance.timeout is reached,
> so basically we are not "waiting" any
> join group request from existing members; we shall just rebalance base on
> what we currently have within the group
> metadata. Lagging consumer will trigger rebalance later if session timeout
> > rebalance timeout.
>
> >
> Just wanted to understand this better. Lets take an example, say we have a
> > consumer group "GroupA" with 4 consumers  c1, c2, c3, c4.
> > Everything is running fine and suddenly C4 host has issues and it goes
> > down. Now we notice that we can still operate with c1, c2, c3 and don't
> > want to wait for
> > c4 to come back up. We use the admin api
> > "invokeConsumerRebalance("GroupA")".
> > Now the GroupCoordinator, will ask the members c1, c2, c3 to join the
> > group again (in there heartBeatResponse) as first step of rebalance.
> > Now lets say that c1, c2 immediately send a joinGroupRequest but c3 is
> > delayed. At this stage, if we are not "waiting" on any join group
> request,
> > few things can happen :
> >
> >- c4's partitions are distributed only among c1,c2. c3 maintains its
> >original assignment. c1, c2 will start processing the newly assigned
> >partitions.
> >
> > OR
> >
> >- c4's partitions are distributed among c1, c2, c3. c1 and c2 start
> >processing the newly assigned partitions. c3 gets to know about the
> newly
> >assigned partitions later when it sends the JoinGroupRequest (which
> was
> >delayed).
> >
> > OR
> >
> >

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Guozhang Wang]

making this work is in the error handling of the zombie
> > consumer. If the zombie simply resets its member.id and rejoins to get a
> > new one upon receiving the UNKNOWN_MEMBER_ID error, then it would end up
> > fencing the new member. We want to avoid this. There needs to be an
> > expectation for static members that the member.id of a static member
> will
> > not be changed except when a new member with the same member.name joins
> > the
> > group. Then we can treat UNKNOWN_MEMBER_ID as a fatal error for consumers
> > with static member names.
> >
> > Yep, I like this idea! Keep giving out refresh member.id when facing
> > anonymous request will definitely
> >
> > prevent processing bug due to duplicate consumers, however I don't think
> I
> > fully understand the 3rd paragraph where
> >
> > you mentioned  "There needs to be an expectation for static members that
> > the member.id of a static member will
> >
> > not be changed except when a new member with the same member.name joins
> > the group. "  How do you plan
> > to know whether this member is new member or old member? I feel even with
> > zombie consumer takes the ownership,
> > it should be detected very quickly (as MISMATCH_ID exception trigger
> > original consumer instance dies)
> > and end user will start to fix it right away. Is there any similar logic
> > we applied in fencing duplicate `transaction.id`?
> >
> > 2. The mechanics of the ConsumerRebalance API seem unclear to me. As far
> as
> > I understand it, it is used for scaling down a consumer group and somehow
> > bypasses normal session timeout expiration. I am wondering how critical
> > this piece is and whether we can leave it for future work. If not, then
> it
> > would be helpful to elaborate on its implementation. How would the
> > coordinator know which members to kick out of the group?
> >
> > This API is needed when we need to immediately trigger rebalance instead
> > of waiting session timeout
> >
> > or rebalance timeout (Emergent scale up/down). It is very necessary to
> > have it for
> >
> > management purpose because user could choose when to trigger rebalance
> > pretty freely,
> >
> > gaining more client side control.
> >
> > In the meanwhile I see your point that we need to actually have the
> > ability to kick out members that we plan
> >
> > to scale down fast (as rebalance timeout no longer kicks any offline
> > member out of the group), I will think of adding an optional
> >
> > list of members that are ready to be removed.
> >
> > Another idea is to let static member send `LeaveGroupRequest` when they
> > are going offline (either scale down or bouncing),
> >
> > and broker will cache this information as "OfflineMembers" without
> > triggering rebalance. When handling ConsumerRebalanceRequest broker will
> >
> > kick the static members that are currently offline and trigger rebalance
> > immediately. How does this plan sound?
> >
> > 3. I've been holding back on mentioning this, but I think we should
> > reconsider the name `member.name`. I think we want something that
> suggests
> > its expectation of uniqueness in the group. How about `group.instance.id
> `
> > to go along with `group.id`?
> >
> > Yea, Dong and Stanislav also mentioned this naming. I personally buy in
> > the namespace idea, and
> >
> > since we already use `member.name` in a lot of context, I decide to
> > rename the config to `group.member.name`
> >
> > which should be sufficient for solving all the concerns we have now.
> > Sounds good?
> >
> >
> > Thank you for your great suggestions! Let me know if my reply makes sense
> > her.
> >
> >
> > Best,
> >
> > Boyang
> >
> > 
> > From: Jason Gustafson 
> > Sent: Tuesday, November 27, 2018 7:51 AM
> > To: dev
> > Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> > specifying member id
> >
> > Hi Boyang,
> >
> > Thanks for the updates. Looks like we're headed in the right direction
> and
> > clearly the interest that this KIP is receiving shows how strong the
> > motivation is!
> >
> > I have a few questions:
> >
> > 1. This may be the same thing that Mayuresh is asking about. I think the
> > suggestion in the KIP is that if a consumer sends JoinGroup with a member
> > name, but no member id, then we will return the current member id
> > associat

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Mayuresh Gharat]

CoordinatorRequest logic to the script.
>
> >7. Are there any specific failure scenarios when you say "other
> >potential failure cases."? It would be good to mention them
> explicitly,
> > if
> >you think there are any.
>
> Nah, I'm gonna remove it because it seems causing more confusion than
> making my assumption clear, which is
>
> "there could be other failure cases that I can't enumerate now" 
>
> >8. It would be good to have a rollback plan as you have for roll
> forward
> >in the KIP.
>
> Great suggestion! Added a simple rollback plan.
>
>
> Next is answering Jason's suggestions:
>
> 1. This may be the same thing that Mayuresh is asking about. I think the
> suggestion in the KIP is that if a consumer sends JoinGroup with a member
> name, but no member id, then we will return the current member id
> associated with that name. It seems in this case that we wouldn't be able
> to protect from having two consumers active with the same configured
> member.name? For example, imagine that we had a consumer with member.name
> =A
> which is assigned member.id=1. Suppose it becomes a zombie and a new
> instance starts up with member.name=A. If it is also assigned member.id=1,
> then how can we detect the zombie if it comes back to life? Both instances
> will have the same member.id.
>
> The goal is to avoid a rebalance on a rolling restart, but we still need to
> fence previous members. I am wondering if we can generate a new member.id
> every time we receive a request from a static member with an unknown member
> id. If the old instance with the same member.name attempts any operation,
> then it will be fenced with an UNKNOWN_MEMBER_ID error. As long as the
> subscription of the new instance hasn't changed, then we can skip the
> rebalance and return the current assignment without forcing a rebalance.
>
> The trick to making this work is in the error handling of the zombie
> consumer. If the zombie simply resets its member.id and rejoins to get a
> new one upon receiving the UNKNOWN_MEMBER_ID error, then it would end up
> fencing the new member. We want to avoid this. There needs to be an
> expectation for static members that the member.id of a static member will
> not be changed except when a new member with the same member.name joins
> the
> group. Then we can treat UNKNOWN_MEMBER_ID as a fatal error for consumers
> with static member names.
>
> Yep, I like this idea! Keep giving out refresh member.id when facing
> anonymous request will definitely
>
> prevent processing bug due to duplicate consumers, however I don't think I
> fully understand the 3rd paragraph where
>
> you mentioned  "There needs to be an expectation for static members that
> the member.id of a static member will
>
> not be changed except when a new member with the same member.name joins
> the group. "  How do you plan
> to know whether this member is new member or old member? I feel even with
> zombie consumer takes the ownership,
> it should be detected very quickly (as MISMATCH_ID exception trigger
> original consumer instance dies)
> and end user will start to fix it right away. Is there any similar logic
> we applied in fencing duplicate `transaction.id`?
>
> 2. The mechanics of the ConsumerRebalance API seem unclear to me. As far as
> I understand it, it is used for scaling down a consumer group and somehow
> bypasses normal session timeout expiration. I am wondering how critical
> this piece is and whether we can leave it for future work. If not, then it
> would be helpful to elaborate on its implementation. How would the
> coordinator know which members to kick out of the group?
>
> This API is needed when we need to immediately trigger rebalance instead
> of waiting session timeout
>
> or rebalance timeout (Emergent scale up/down). It is very necessary to
> have it for
>
> management purpose because user could choose when to trigger rebalance
> pretty freely,
>
> gaining more client side control.
>
> In the meanwhile I see your point that we need to actually have the
> ability to kick out members that we plan
>
> to scale down fast (as rebalance timeout no longer kicks any offline
> member out of the group), I will think of adding an optional
>
> list of members that are ready to be removed.
>
> Another idea is to let static member send `LeaveGroupRequest` when they
> are going offline (either scale down or bouncing),
>
> and broker will cache this information as "OfflineMembers" without
> triggering rebalance. When handling ConsumerRebalanceRequest broker will
>
> kick the static members that are currently offline and trigger rebalance
> 

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

but I think we should
reconsider the name `member.name`. I think we want something that suggests
its expectation of uniqueness in the group. How about `group.instance.id`
to go along with `group.id`?

Yea, Dong and Stanislav also mentioned this naming. I personally buy in the 
namespace idea, and

since we already use `member.name` in a lot of context, I decide to rename the 
config to `group.member.name`

which should be sufficient for solving all the concerns we have now. Sounds 
good?


Thank you for your great suggestions! Let me know if my reply makes sense her.


Best,

Boyang


From: Jason Gustafson 
Sent: Tuesday, November 27, 2018 7:51 AM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hi Boyang,

Thanks for the updates. Looks like we're headed in the right direction and
clearly the interest that this KIP is receiving shows how strong the
motivation is!

I have a few questions:

1. This may be the same thing that Mayuresh is asking about. I think the
suggestion in the KIP is that if a consumer sends JoinGroup with a member
name, but no member id, then we will return the current member id
associated with that name. It seems in this case that we wouldn't be able
to protect from having two consumers active with the same configured
member.name? For example, imagine that we had a consumer with member.name=A
which is assigned member.id=1. Suppose it becomes a zombie and a new
instance starts up with member.name=A. If it is also assigned member.id=1,
then how can we detect the zombie if it comes back to life? Both instances
will have the same member.id.

The goal is to avoid a rebalance on a rolling restart, but we still need to
fence previous members. I am wondering if we can generate a new member.id
every time we receive a request from a static member with an unknown member
id. If the old instance with the same member.name attempts any operation,
then it will be fenced with an UNKNOWN_MEMBER_ID error. As long as the
subscription of the new instance hasn't changed, then we can skip the
rebalance and return the current assignment without forcing a rebalance.

The trick to making this work is in the error handling of the zombie
consumer. If the zombie simply resets its member.id and rejoins to get a
new one upon receiving the UNKNOWN_MEMBER_ID error, then it would end up
fencing the new member. We want to avoid this. There needs to be an
expectation for static members that the member.id of a static member will
not be changed except when a new member with the same member.name joins the
group. Then we can treat UNKNOWN_MEMBER_ID as a fatal error for consumers
with static member names.

2. The mechanics of the ConsumerRebalance API seem unclear to me. As far as
I understand it, it is used for scaling down a consumer group and somehow
bypasses normal session timeout expiration. I am wondering how critical
this piece is and whether we can leave it for future work. If not, then it
would be helpful to elaborate on its implementation. How would the
coordinator know which members to kick out of the group?

3. I've been holding back on mentioning this, but I think we should
reconsider the name `member.name`. I think we want something that suggests
its expectation of uniqueness in the group. How about `group.instance.id`
to go along with `group.id`?

Thanks,
Jason



On Mon, Nov 26, 2018 at 10:18 AM Mayuresh Gharat 
wrote:

> Hi Boyang,
>
> Thanks a lot for replying to all the queries and discussions here, so
> patiently.
> Really appreciate it.
>
> Had a few questions and suggestions after rereading the current version of
> the KIP :
>
>
>1. Do you intend to have member.id is a static config like member.name
>after KIP-345 and KIP-394?
>2. Regarding "On client side, we add a new config called MEMBER_NAME in
>ConsumerConfig. On consumer service init, if the MEMBER_NAME config is
> set,
>we will put it in the initial join group request to identify itself as a
>static member (static membership); otherwise, we will still send
>UNKNOWN_MEMBER_ID to ask broker for allocating a new random ID (dynamic
>membership)."
>   - What is the value of member_id sent in the first JoinGroupRequest
>   when member_name is set (using static rebalance)? Is it
> UNKNOW_MEMBER_ID?
>3. Regarding "we are requiring member.id (if not unknown) to match the
>value stored in cache, otherwise reply MEMBER_ID_MISMATCH. The edge case
>that if we could have members with the same `member.name` (for example
>mis-configured instances with a valid member.id but added a used member
>name on runtime). When member name has duplicates, we could refuse join
>request from members with an outdated `member.id` (since we update the
>mapping upon each join group request). In an edge case where the client
>

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Jason Gustafson]

LICATE_STATIC_MEMBER says that "The join group
> contains
> >> member name which is already in the consumer group, however the member
> id
> >> was missing". After a consumer is restarted, it will send a
> >> JoinGroupRequest with an existing memberName (as the coordinator has not
> >> expired this member from the memory) and memberId
> >> = JoinGroupRequest.UNKNOWN_MEMBER_ID (since memberId is not persisted
> >> across consumer restart in the consumer side). Does it mean that
> >> JoinGroupRequest from a newly restarted consumer will always be rejected
> >> until the sessionTimeoutMs has passed?
> >> Same answer as question 3). This part of the logic shall be removed from
> >> the proposal.
> >>
> >> 5) It seems that we always add two methods to the interface
> >> org.apache.kafka.clients.admin.AdminClient.java, one with options and
> the
> >> other without option. Could this be specified in the interface change
> >> section?
> >> Sounds good! Added both methods.
> >>
> >> 6) Do we plan to have off-the-shelf command line tool for SRE to trigger
> >> rebalance? If so, we probably want to specify the command line tool
> >> interface similar to
> >>
> >>
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-113%253A%2BSupport%2Breplicas%2Bmovement%2Bbetween%2Blog%2Bdirectories%23KIP-113%3ASupportreplicasmovementbetweenlogdirectories-Scriptsdata=02%7C01%7C%7Cdde139857e7a4a3a83dd08d651d9c93e%7C84df9e7fe9f640afb435%7C1%7C0%7C636786393153281080sdata=%2BNasMFlJf9rEJc9iDfndcyxA4%2BGWieS1azSKbtdGRW4%3Dreserved=0
> >> .
> >> Added the script.
> >>
> >> 7) Would it be simpler to replace name "forceStaticRebalance" with
> >> "invokeConsumerRebalance"? It is not very clear what is the extra
> meaning
> >> of world "force" as compared to "trigger" or "invoke". And it seems
> >> simpler
> >> to allows this API to trigger rebalance regardless of whether consumer
> is
> >> configured with memberName.
> >> Sounds good. Right now I feel for both static and dynamic membership it
> is
> >> more manageable to introduce the consumer rebalance method through admin
> >> client API.
> >>
> >> 8) It is not very clear how the newly added AdminClient API trigger
> >> rebalance. For example, does it send request? Can this be explained in
> the
> >> KIP?
> >>
> >> Sure, I will add more details to the API.
> >>
> >>
> >> Thanks again for the helpful suggestions!
> >>
> >>
> >> Best,
> >> Boyang
> >>
> >> 
> >> From: Dong Lin 
> >> Sent: Saturday, November 24, 2018 2:54 PM
> >> To: dev
> >> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> >> specifying member id
> >>
> >> Hey Boyang,
> >>
> >> Thanks for the update! Here are some followup comments:
> >>
> >> 1) It is not very clear to the user what is the difference between
> >> member.name and client.id as both seems to be used to identify the
> >> consumer. I am wondering if it would be more intuitive to name it
> >> group.member.name (preferred choice since it matches the current
> group.id
> >> config name) or rebalance.member.name to explicitly show that the id is
> >> solely used for rebalance.
> >>
> >> 2) In the interface change section it is said that
> >> GroupMaxSessionTimeoutMs
> >> will be changed to 30 minutes. It seems to suggest that we will change
> the
> >> default value of this config. It does not seem necessary to increase the
> >> time of consumer failure detection when user doesn't use static
> >> membership.
> >> Also, say static membership is enabled, then this default config change
> >> will cause a partition to be unavailable for consumption for 30 minutes
> if
> >> there is hard consumer failure, which seems to be worse experience than
> >> having unnecessary rebalance (when this timeout is small), particularly
> >> for
> >> new users of Kafka. Could you explain more why we should make this
> change?
> >>
> >> 3) Could we just combine MEMBER_ID_MISMATCH and DUPLICATE_STATIC_MEMBER
> >> into one error? It seems that these two errors are currently handled by
> >> the
> >> consumer in th

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Mayuresh Gharat]

detection when user doesn't use static
>> membership.
>> Also, say static membership is enabled, then this default config change
>> will cause a partition to be unavailable for consumption for 30 minutes if
>> there is hard consumer failure, which seems to be worse experience than
>> having unnecessary rebalance (when this timeout is small), particularly
>> for
>> new users of Kafka. Could you explain more why we should make this change?
>> We are not changing the default session timeout value. We are just
>> changing the
>> cap we are enforcing on the session timeout max value. So this change is
>> not affecting
>> what kind of membership end user is using, and loosing the cap is giving
>> end user
>> more flexibility on trade-off between liveness and stability.
>>
>> 3) Could we just combine MEMBER_ID_MISMATCH and DUPLICATE_STATIC_MEMBER
>> into one error? It seems that these two errors are currently handled by
>> the
>> consumer in the same way. And we don't also don't expect
>> MEMBER_ID_MISMATCH
>> to happen. Thus it is not clear what is the benefit of having two errors.
>> I agree that we should remove DUPLICATE_STATIC_MEMBER error because with
>> the KIP-394<
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-394%3A+Require+member+id+for+initial+join+group+request
>> >
>> we will automatically fence all join requests with UNKNOWN_MEMBER_ID.
>>
>> 4) The doc for DUPLICATE_STATIC_MEMBER says that "The join group contains
>> member name which is already in the consumer group, however the member id
>> was missing". After a consumer is restarted, it will send a
>> JoinGroupRequest with an existing memberName (as the coordinator has not
>> expired this member from the memory) and memberId
>> = JoinGroupRequest.UNKNOWN_MEMBER_ID (since memberId is not persisted
>> across consumer restart in the consumer side). Does it mean that
>> JoinGroupRequest from a newly restarted consumer will always be rejected
>> until the sessionTimeoutMs has passed?
>> Same answer as question 3). This part of the logic shall be removed from
>> the proposal.
>>
>> 5) It seems that we always add two methods to the interface
>> org.apache.kafka.clients.admin.AdminClient.java, one with options and the
>> other without option. Could this be specified in the interface change
>> section?
>> Sounds good! Added both methods.
>>
>> 6) Do we plan to have off-the-shelf command line tool for SRE to trigger
>> rebalance? If so, we probably want to specify the command line tool
>> interface similar to
>>
>> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-113%253A%2BSupport%2Breplicas%2Bmovement%2Bbetween%2Blog%2Bdirectories%23KIP-113%3ASupportreplicasmovementbetweenlogdirectories-Scriptsdata=02%7C01%7C%7Cdde139857e7a4a3a83dd08d651d9c93e%7C84df9e7fe9f640afb435%7C1%7C0%7C636786393153281080sdata=%2BNasMFlJf9rEJc9iDfndcyxA4%2BGWieS1azSKbtdGRW4%3Dreserved=0
>> .
>> Added the script.
>>
>> 7) Would it be simpler to replace name "forceStaticRebalance" with
>> "invokeConsumerRebalance"? It is not very clear what is the extra meaning
>> of world "force" as compared to "trigger" or "invoke". And it seems
>> simpler
>> to allows this API to trigger rebalance regardless of whether consumer is
>> configured with memberName.
>> Sounds good. Right now I feel for both static and dynamic membership it is
>> more manageable to introduce the consumer rebalance method through admin
>> client API.
>>
>> 8) It is not very clear how the newly added AdminClient API trigger
>> rebalance. For example, does it send request? Can this be explained in the
>> KIP?
>>
>> Sure, I will add more details to the API.
>>
>>
>> Thanks again for the helpful suggestions!
>>
>>
>> Best,
>> Boyang
>>
>> 
>> From: Dong Lin 
>> Sent: Saturday, November 24, 2018 2:54 PM
>> To: dev
>> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
>> specifying member id
>>
>> Hey Boyang,
>>
>> Thanks for the update! Here are some followup comments:
>>
>> 1) It is not very clear to the user what is the difference between
>> member.name and client.id as both seems to be used to identify the
>> consumer. I am wondering if it would be more intuitive to name it
>> group.member.name (preferred choice since it matches the current group.id
>> config name) or rebalan

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Mayuresh Gharat]

n this scenario?
> > I have extended the two exceptions within join group response V4.
> > Basically I define both corresponding actions to be immediate failing
> > client application, because so far it is unknown what kind of client
> issue
> > could trigger them. After the first version, we will keep enhance the
> error
> > handling logic!
> >
> > > This would mean that it might take more time to detect unowned topic
> > > partitions and may cause delay for applications that perform data
> > mirroring
> > > tasks. I discussed this with our sre and we have a suggestion to make
> > here
> > > as listed below separately.
> > The goal of extending session timeout cap is for users with good client
> > side monitoring tools that could auto-heal the dead consumers very fast.
> So
> > it is optional (and personal) to extend session timeout to a reasonable
> > number with different client scenarios.
> >
> > > you meant remove unjoined members of the group, right ?
> > Yep, there is a typo. Thanks for catching this!
> >
> > > What do you mean by " Internally we would optimize this logic by having
> > > rebalance timeout only in charge of stopping prepare rebalance stage,
> > > without removing non-responsive members immediately." There would not
> be
> > a
> > > full rebalance if the lagging consumer sent a JoinGroup request later,
> > > right ? If yes, can you highlight this in the KIP ?
> > No, there won't be. We want to limit the rebalance timeout functionality
> > to only use as a timer to
> > end prepare rebalance stage. This way, late joining static members will
> > not trigger further rebalance
> > as long as they are within session timeout. I added your highlight to the
> > KIP!
> >
> > > The KIP talks about scale up scenario but its not quite clear how we
> > > handle it. Are we adding a separate "expansion.timeout" or we adding
> > status
> > > "learner" ?. Can you shed more light on how this is handled in the KIP,
> > if
> > > its handled?
> > Updated the KIP: we shall not cover scale up case in 345, because we
> > believe client side could
> > better handle this logic.
> >
> > > I think Jason had brought this up earlier about having a way to say how
> > > many members/consumer hosts are you choosing to be in the consumer
> group.
> > > If we can do this, then in case of mirroring applications we can do
> this
> > :
> > > Lets say we have a mirroring application that consumes from Kafka
> cluster
> > > A and produces to Kafka cluster B.
> > > Depending on the data and the Kafka cluster configuration, Kafka
> service
> > > providers can set a mirroring group saying that it will take, for
> example
> > > 300 consumer hosts/members to achieve the desired throughput and
> latency
> > > for mirroring and can have additional 10 consumer hosts as spare in the
> > > same group.
> > > So when the first 300 members/consumers to join the group will start
> > > mirroring the data from Kafka cluster A to Kafka cluster B.
> > > The remaining 10 consumer members can sit idle.
> > > The moment one of the consumer (for example: consumer number 54) from
> the
> > > first 300 members go out of the group (crossed session timeout), it
> (the
> > > groupCoordinator) can just assign the topicPartitions from the consumer
> > > member 54 to one of the spare hosts.
> > > Once the consumer member 54 comes back up, it can start as being a part
> > of
> > > the spare pool.
> > > This enables us to have lower session timeouts and low latency
> mirroring,
> > > in cases where the service providers are OK with having spare hosts.
> > > This would mean that we would tolerate n consumer members leaving and
> > > rejoining the group and still provide low latency as long as n <=
> number
> > of
> > > spare consumers.
> > > If there are no spare host available, we can get back to the idea as
> > > described in the KIP.
> > Great idea! In fact on top of static membership we could later introduce
> > APIs to set hard-coded
> > client ids to the group and replace the dead host, or as you proposed to
> > define spare host as
> > what I understood as hot backup. I will put both Jason and your
> > suggestions into a separate section
> > called "Future works". Note that this spare host idea may be also
> solvable
> > through rebalance protocol
> > IMO.
> >
> >

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

Hey Dong, thanks for the follow-up here!


1) It is not very clear to the user what is the difference between
member.name and client.id as both seems to be used to identify the
consumer. I am wondering if it would be more intuitive to name it
group.member.name (preferred choice since it matches the current group.id
config name) or rebalance.member.name to explicitly show that the id is
solely used for rebalance.
Great question. I feel `member.name` is enough to explain itself, it seems not 
very
helpful to make the config name longer. Comparing `name` with `id` gives user 
the
impression that they have the control over it with customized rule than library 
decided.

2) In the interface change section it is said that GroupMaxSessionTimeoutMs
will be changed to 30 minutes. It seems to suggest that we will change the
default value of this config. It does not seem necessary to increase the
time of consumer failure detection when user doesn't use static membership.
Also, say static membership is enabled, then this default config change
will cause a partition to be unavailable for consumption for 30 minutes if
there is hard consumer failure, which seems to be worse experience than
having unnecessary rebalance (when this timeout is small), particularly for
new users of Kafka. Could you explain more why we should make this change?
We are not changing the default session timeout value. We are just changing the
cap we are enforcing on the session timeout max value. So this change is not 
affecting
what kind of membership end user is using, and loosing the cap is giving end 
user
more flexibility on trade-off between liveness and stability.

3) Could we just combine MEMBER_ID_MISMATCH and DUPLICATE_STATIC_MEMBER
into one error? It seems that these two errors are currently handled by the
consumer in the same way. And we don't also don't expect MEMBER_ID_MISMATCH
to happen. Thus it is not clear what is the benefit of having two errors.
I agree that we should remove DUPLICATE_STATIC_MEMBER error because with the 
KIP-394<https://cwiki.apache.org/confluence/display/KAFKA/KIP-394%3A+Require+member+id+for+initial+join+group+request>
we will automatically fence all join requests with UNKNOWN_MEMBER_ID.

4) The doc for DUPLICATE_STATIC_MEMBER says that "The join group contains
member name which is already in the consumer group, however the member id
was missing". After a consumer is restarted, it will send a
JoinGroupRequest with an existing memberName (as the coordinator has not
expired this member from the memory) and memberId
= JoinGroupRequest.UNKNOWN_MEMBER_ID (since memberId is not persisted
across consumer restart in the consumer side). Does it mean that
JoinGroupRequest from a newly restarted consumer will always be rejected
until the sessionTimeoutMs has passed?
Same answer as question 3). This part of the logic shall be removed from the 
proposal.

5) It seems that we always add two methods to the interface
org.apache.kafka.clients.admin.AdminClient.java, one with options and the
other without option. Could this be specified in the interface change
section?
Sounds good! Added both methods.

6) Do we plan to have off-the-shelf command line tool for SRE to trigger
rebalance? If so, we probably want to specify the command line tool
interface similar to
https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-113%253A%2BSupport%2Breplicas%2Bmovement%2Bbetween%2Blog%2Bdirectories%23KIP-113%3ASupportreplicasmovementbetweenlogdirectories-Scriptsdata=02%7C01%7C%7Cdde139857e7a4a3a83dd08d651d9c93e%7C84df9e7fe9f640afb435%7C1%7C0%7C636786393153281080sdata=%2BNasMFlJf9rEJc9iDfndcyxA4%2BGWieS1azSKbtdGRW4%3Dreserved=0
.
Added the script.

7) Would it be simpler to replace name "forceStaticRebalance" with
"invokeConsumerRebalance"? It is not very clear what is the extra meaning
of world "force" as compared to "trigger" or "invoke". And it seems simpler
to allows this API to trigger rebalance regardless of whether consumer is
configured with memberName.
Sounds good. Right now I feel for both static and dynamic membership it is
more manageable to introduce the consumer rebalance method through admin
client API.

8) It is not very clear how the newly added AdminClient API trigger
rebalance. For example, does it send request? Can this be explained in the
KIP?

Sure, I will add more details to the API.


Thanks again for the helpful suggestions!


Best,
Boyang


From: Dong Lin 
Sent: Saturday, November 24, 2018 2:54 PM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hey Boyang,

Thanks for the update! Here are some followup comments:

1) It is not very clear to the user what is the difference between
member.name and client.id as both seems to be used to identify the
consumer. I am wondering if it would be m

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Dong Lin]

this!
>
> > What do you mean by " Internally we would optimize this logic by having
> > rebalance timeout only in charge of stopping prepare rebalance stage,
> > without removing non-responsive members immediately." There would not be
> a
> > full rebalance if the lagging consumer sent a JoinGroup request later,
> > right ? If yes, can you highlight this in the KIP ?
> No, there won't be. We want to limit the rebalance timeout functionality
> to only use as a timer to
> end prepare rebalance stage. This way, late joining static members will
> not trigger further rebalance
> as long as they are within session timeout. I added your highlight to the
> KIP!
>
> > The KIP talks about scale up scenario but its not quite clear how we
> > handle it. Are we adding a separate "expansion.timeout" or we adding
> status
> > "learner" ?. Can you shed more light on how this is handled in the KIP,
> if
> > its handled?
> Updated the KIP: we shall not cover scale up case in 345, because we
> believe client side could
> better handle this logic.
>
> > I think Jason had brought this up earlier about having a way to say how
> > many members/consumer hosts are you choosing to be in the consumer group.
> > If we can do this, then in case of mirroring applications we can do this
> :
> > Lets say we have a mirroring application that consumes from Kafka cluster
> > A and produces to Kafka cluster B.
> > Depending on the data and the Kafka cluster configuration, Kafka service
> > providers can set a mirroring group saying that it will take, for example
> > 300 consumer hosts/members to achieve the desired throughput and latency
> > for mirroring and can have additional 10 consumer hosts as spare in the
> > same group.
> > So when the first 300 members/consumers to join the group will start
> > mirroring the data from Kafka cluster A to Kafka cluster B.
> > The remaining 10 consumer members can sit idle.
> > The moment one of the consumer (for example: consumer number 54) from the
> > first 300 members go out of the group (crossed session timeout), it (the
> > groupCoordinator) can just assign the topicPartitions from the consumer
> > member 54 to one of the spare hosts.
> > Once the consumer member 54 comes back up, it can start as being a part
> of
> > the spare pool.
> > This enables us to have lower session timeouts and low latency mirroring,
> > in cases where the service providers are OK with having spare hosts.
> > This would mean that we would tolerate n consumer members leaving and
> > rejoining the group and still provide low latency as long as n <= number
> of
> > spare consumers.
> > If there are no spare host available, we can get back to the idea as
> > described in the KIP.
> Great idea! In fact on top of static membership we could later introduce
> APIs to set hard-coded
> client ids to the group and replace the dead host, or as you proposed to
> define spare host as
> what I understood as hot backup. I will put both Jason and your
> suggestions into a separate section
> called "Future works". Note that this spare host idea may be also solvable
> through rebalance protocol
> IMO.
>
> Thank you again for the great feedback!
>
> Boyang
> 
> From: Boyang Chen 
> Sent: Thursday, November 22, 2018 3:39 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> Hey Dong, sorry for missing your message. I couldn't find your email on my
> thread, so I will just do a checklist here!
>
>
> 1) The motivation currently explicitly states that the goal is to improve
>
> performance for heavy state application. It seems that the motivation can
>
> be stronger with the following use-case. Currently for MirrorMaker cluster
>
> with e.g. 100 MirrorMaker processes, it will take a long time to rolling
>
> bounce the entire MirrorMaker cluster. Each MirrorMaker process restart
>
> will trigger a rebalance which currently pause the consumption of the all
>
> partitions of the MirrorMaker cluster. With the change stated in this
>
> patch, as long as a MirrorMaker can restart within the specified timeout
>
> (e.g. 2 minutes), then we only need constant number of rebalance (e.g. for
>
> leader restart) for the entire rolling bounce, which will significantly
>
> improves the availability of the MirrorMaker pipeline. In my opinion, the
>
> main benefit of the KIP is to avoid unnecessary rebalance if the consumer
>
> process can be restarted within soon, which helps performance even if
>
> overhead o

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

Hey Mayuresh,


thanks for your feedbacks! I will try do another checklist here.


> By this you mean, even if the application has not called
> KafkaConsumer.poll() within session timeout, it will not be sending the
> LeaveGroup request, right?

Yep it's true, we will prevent client from sending leave group request when 
they are set with `member.name`.


> When is the member.name removed from this map?
Good question, we will only kick off member due to session timeout within 
static membership. Let me update the KIP to clearly assert that.

> How is this case (missing member id) handled on the client side? What is the 
> application that
> is using the KafkaConsumer suppose to do in this scenario?
I have extended the two exceptions within join group response V4. Basically I 
define both corresponding actions to be immediate failing client application, 
because so far it is unknown what kind of client issue could trigger them. 
After the first version, we will keep enhance the error handling logic!

> This would mean that it might take more time to detect unowned topic
> partitions and may cause delay for applications that perform data mirroring
> tasks. I discussed this with our sre and we have a suggestion to make here
> as listed below separately.
The goal of extending session timeout cap is for users with good client side 
monitoring tools that could auto-heal the dead consumers very fast. So it is 
optional (and personal) to extend session timeout to a reasonable number with 
different client scenarios.

> you meant remove unjoined members of the group, right ?
Yep, there is a typo. Thanks for catching this!

> What do you mean by " Internally we would optimize this logic by having
> rebalance timeout only in charge of stopping prepare rebalance stage,
> without removing non-responsive members immediately." There would not be a
> full rebalance if the lagging consumer sent a JoinGroup request later,
> right ? If yes, can you highlight this in the KIP ?
No, there won't be. We want to limit the rebalance timeout functionality to 
only use as a timer to
end prepare rebalance stage. This way, late joining static members will not 
trigger further rebalance
as long as they are within session timeout. I added your highlight to the KIP!

> The KIP talks about scale up scenario but its not quite clear how we
> handle it. Are we adding a separate "expansion.timeout" or we adding status
> "learner" ?. Can you shed more light on how this is handled in the KIP, if
> its handled?
Updated the KIP: we shall not cover scale up case in 345, because we believe 
client side could
better handle this logic.

> I think Jason had brought this up earlier about having a way to say how
> many members/consumer hosts are you choosing to be in the consumer group.
> If we can do this, then in case of mirroring applications we can do this :
> Lets say we have a mirroring application that consumes from Kafka cluster
> A and produces to Kafka cluster B.
> Depending on the data and the Kafka cluster configuration, Kafka service
> providers can set a mirroring group saying that it will take, for example
> 300 consumer hosts/members to achieve the desired throughput and latency
> for mirroring and can have additional 10 consumer hosts as spare in the
> same group.
> So when the first 300 members/consumers to join the group will start
> mirroring the data from Kafka cluster A to Kafka cluster B.
> The remaining 10 consumer members can sit idle.
> The moment one of the consumer (for example: consumer number 54) from the
> first 300 members go out of the group (crossed session timeout), it (the
> groupCoordinator) can just assign the topicPartitions from the consumer
> member 54 to one of the spare hosts.
> Once the consumer member 54 comes back up, it can start as being a part of
> the spare pool.
> This enables us to have lower session timeouts and low latency mirroring,
> in cases where the service providers are OK with having spare hosts.
> This would mean that we would tolerate n consumer members leaving and
> rejoining the group and still provide low latency as long as n <= number of
> spare consumers.
> If there are no spare host available, we can get back to the idea as
> described in the KIP.
Great idea! In fact on top of static membership we could later introduce APIs 
to set hard-coded
client ids to the group and replace the dead host, or as you proposed to define 
spare host as
what I understood as hot backup. I will put both Jason and your suggestions 
into a separate section
called "Future works". Note that this spare host idea may be also solvable 
through rebalance protocol
IMO.

Thank you again for the great feedback!

Boyang
____________
From: Boyang Chen 
Sent: Thursday, November 22, 2018 3:39 PM
To: dev@kafka.apache.or

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

out as a workaround for triggering 
multiple
rebalances when scaling up members are not joining at the same time. It is 
decided to
be addressed by client side protocol change, so we will not introduce expansion 
timeout.

8) It is mentioned that "To distinguish between previous version of
protocol, we will also increase the join group request version to v4 when
MEMBER_NAME is set" and "If the broker version is not the latest (< v4),
the join group request shall be downgraded to v3 without setting the member
Id". It is probably simpler to just say that this feature is enabled if
JoinGroupRequest V4 is supported on both client and broker and MEMBER_NAME
is configured with non-empty string.
Yep, addressed this!

9) It is mentioned that broker may return NO_STATIC_MEMBER_INFO_SET error
in OffsetCommitResponse for "commit requests under static membership". Can
you clarify how broker determines whether the commit request is under
static membership?

We have agreed that commit request shouldn't be affected by the new membership, 
thus
removing it here. Thanks for catching this!

Let me know if you have further suggestions or concerns. Thank you for your 
valuable feedback
to help me design the KIP better! (And I will try to address your feedbacks in 
next round Mayuresh )

Best,
Boyang

From: Mayuresh Gharat 
Sent: Wednesday, November 21, 2018 7:50 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hi Boyang,

Thanks for updating the KIP. This is a step good direction for stateful
applications and also mirroring applications whose latency is affected due
to the rebalance issues that we have today.

I had a few questions on the current version of the KIP :
For the effectiveness of the KIP, consumer with member.name set will *not
send leave group request* when they go offline

> By this you mean, even if the application has not called
> KafkaConsumer.poll() within session timeout, it will not be sending the
> LeaveGroup request, right?
>

Broker will maintain an in-memory mapping of {member.name → member.id} to
track member uniqueness.

> When is the member.name removed from this map?
>

Member.id must be set if the *member.name 
<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmember.namedata=02%7C01%7C%7Cd9b758a0a8be4f85e16c08d64f430e40%7C84df9e7fe9f640afb435%7C1%7C0%7C636783546751462147sdata=39mD7qTQCN6ZB04ilVcHdlngjicF5335YBun9n9HIR8%3Dreserved=0>
 *is already
within the map. Otherwise reply MISSING_MEMBER_ID

> How is this case handled on the client side? What is the application that
> is using the KafkaConsumer suppose to do in this scenario?
>

Session timeout is the timeout we will trigger rebalance when a member goes
offline for too long (not sending heartbeat request). To make static
membership effective, we should increase the default max session timeout to
30 min so that end user could config it freely.

> This would mean that it might take more time to detect unowned topic
> partitions and may cause delay for applications that perform data mirroring
> tasks. I discussed this with our sre and we have a suggestion to make here
> as listed below separately.
>

Currently there is a config called *rebalance timeout* which is configured
by consumer *max.poll.intervals*. The reason we set it to poll interval is
because consumer could only send request within the call of poll() and we
want to wait sufficient time for the join group request. When reaching
rebalance timeout, the group will move towards completingRebalance stage
and remove unjoined groups

> you meant remove unjoined members of the group, right ?
>

Currently there is a config called *rebalance timeout* which is configured
by consumer *max.poll.intervals*. The reason we set it to poll interval is
because consumer could only send request within the call of poll() and we
want to wait sufficient time for the join group request. When reaching
rebalance timeout, the group will move towards completingRebalance stage
and remove unjoined groups. This is actually conflicting with the design of
static membership, because those temporarily unavailable members will
potentially reattempt the join group and trigger extra rebalances.
Internally we would optimize this logic by having rebalance timeout only in
charge of stopping prepare rebalance stage, without removing non-responsive
members immediately.

> What do you mean by " Internally we would optimize this logic by having
> rebalance timeout only in charge of stopping prepare rebalance stage,
> without removing non-responsive members immediately." There would not be a
> full rebalance if the lagging consumer sent a JoinGroup request later,
> right ? If yes, can you highlight this in the KIP ?
>

Scale Up

> The KIP talks about scale up scenario but its no

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Mayuresh Gharat]

 pretty
> > independent. I could start drafting the KIP once the current proposal is
> > approved.
> >   5.  I believe that we don't need fencing in offset commit request,
> since
> > duplicate member.name issue could be handled by join group request. We
> > shall reject join group with known member name but no member id (which
> > means we already have an active member using this identity).
> >   6.  I agree to remove that internal config once we move forward with
> > static membership. And I already removed the entire section from the KIP.
> >
> > Let me know if you have other concerns.
> >
> > Best,
> > Boyang
> > 
> > From: Guozhang Wang 
> > Sent: Tuesday, November 20, 2018 4:21 PM
> > To: dev
> > Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> > specifying member id
> >
> > Hello Boyang,
> >
> > Thanks a lot for the KIP! It is a great write-up and I appreciate your
> > patience answering to the feedbacks from the community. I'd like to add
> my
> > 2cents here:
> >
> > 1. By introducing another two timeout configs, registration_timeout and
> > expansion_timeout, we are effectively having four timeout configs:
> session
> > timeout, rebalance timeout (configured as "max.poll.interval.ms" on
> client
> > side), and these two. Interplaying these timeout configs can be quite
> hard
> > for users with such complexity, and hence I'm wondering if we can
> simplify
> > the situation with as less possible timeout configs as possible. Here is
> a
> > concrete suggestion I'd like propose:
> >
> > 1.a) Instead of introducing a registration_timeout in addition to the
> > session_timeout for static members, we can just reuse the session_timeout
> > and ask users to set it to a larger value when they are upgrading a
> dynamic
> > client to a static client by setting the "member.name" at the same time.
> > By
> > default, the broker-side min.session.timeout is 6 seconds and
> > max.session.timeout is 5 minutes, which seems reasonable to me (we can of
> > course modify this broker config to enlarge the valid interval if we want
> > in practice). And then we should also consider removing the condition for
> > marking a client as failed if the rebalance timeout has reached while the
> > JoinGroup was not received, so that the semantics of session_timeout and
> > rebalance_timeout are totally separated: the former is only used to
> > determine if a consumer member of the group should be marked as failed
> and
> > kicked out of the group, and the latter is only used to determine the
> > longest time coordinator should wait for PREPARE_REBALANCE phase. In
> other
> > words if a member did not send the JoinGroup in time of the
> > rebalance_timeout, we still include it in the new generation of the group
> > and use its old subscription info to send to leader for assignment. Later
> > if the member came back with HeartBeat request, we can still follow the
> > normal path to bring it to the latest generation while checking that its
> > sent JoinGroup request contains the same subscription info as we used to
> > assign the partitions previously (which should be likely the case in
> > practice). In addition, we should let static members to not send the
> > LeaveGroup request when it is gracefully shutdown, so that a static
> member
> > can only be leaving the group if its session has timed out, OR it has
> been
> > indicated to not exist in the group any more (details below).
> >
> > 1.b) We have a parallel discussion about Incremental Cooperative
> > Rebalancing, in which we will encode the "when to rebalance" logic at the
> > application level, instead of at the protocol level. By doing this we can
> > also enable a few other optimizations, e.g. at the Streams level to first
> > build up the state store as standby tasks and then trigger a second
> > rebalance to actually migrate the active tasks while keeping the actual
> > rebalance latency and hence unavailability window to be small (
> >
> >
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FKAFKA-6145data=02%7C01%7C%7C7272fe45059f4afa205208d64ec12c1e%7C84df9e7fe9f640afb435%7C1%7C0%7C636782988907214421sdata=Etn5Y%2BVFTo0GQ1BydPVMUUYbEf04wyhqzKrs3OjCmC8%3Dreserved=0
> ).
> > I'd propose we align
> > KIP-345 along with this idea, and hence do not add the expansion_timeout
> as
> > part of the protocol layer, but only do that at the application's
> >

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

Thanks Konstantine for correcting the details within proposal for me! Addressed 
them below:


> In a few places the new/proposed changes are referred to as "current".
> Which is a bit confusing considering that there is a protocol in place
> already, and by "current" someone might understand the existing one.

Fixed the `current` usage with `proposed`!

> There's the following sentence in the "Public Interfaces" section:
>"Since for many stateful consumer/stream applications, the state shuffling
> is more painful than short time partial unavailability."
> However, my understanding is that the changes proposed with KIP-345 will
> not exploit any partial availability.
We are proposing to extend session timeout inside static membership and change 
rebalance timeout not to remove unjoined members, which means we would detect a 
consumer failure slower from broker side perspective. This is what I mean by 
"partial unavailability": some topic partition is not making progress due to 
consumer dead/hanging.

> In the rejected alternatives, under point 2) I read "we can copy the member
> id to the config files". I believe it means to say "member name" unless I'm
> missing something about reusing member ids
Updated that section!

Let me know if this makes sense to you!

Boyang

From: Konstantine Karantasis 
Sent: Wednesday, November 21, 2018 2:18 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hi Boyang.

Thanks for preparing this KIP! It is making good progress and will be a
great improvement for stateful Kafka applications.

Apologies for my late reply, I was away for a while. Lots of great comments
so far, so I'll probably second most of them in what I suggest below at
this point.

When I first read the KIP, I wanted to start at the end with something that
wasn't highlighted a lot. That was the topic related to handling duplicate
members. I see now that the initial suggestion of handling this situation
during offset commit has been removed, and I agree with that. Issues
related to membership seem to be handled better when the member joins the
group rather than when it tries to commit offsets. This also simplifies how
many request types need to change in order to incorporate the new member
name field.

I also agree with what Jason and Guozhang have said regarding timeouts.
Although semantically, it's easier to think of every operation having its
own timeout, operationally this can become a burden. Thus, consolidation
seems preferable here. The definition of embedded protocols on top of the
base group membership protocol for rebalancing gives enough flexibility to
address such needs in each client component separately.

Finally, some minor comments:
In a few places the new/proposed changes are referred to as "current".
Which is a bit confusing considering that there is a protocol in place
already, and by "current" someone might understand the existing one. I'd
recommend using new/proposed or equivalent when referring to changes
introduced with KIP-345 and current/existing or equivalent when referring
to existing behavior.

There's the following sentence in the "Public Interfaces" section:
"Since for many stateful consumer/stream applications, the state shuffling
is more painful than short time partial unavailability."
However, my understanding is that the changes proposed with KIP-345 will
not exploit any partial availability. A suggestion for dealing with
temporary imbalances has been made in "Incremental Cooperative Rebalancing"
which can work well with KIP-345, but here I don't see proposed changes
that suggest that some resources (e.g. partitions) will keep being used
while others will not be utilized. Thus, you might want to adjust this
sentence. Correct me if I'm missing something related to that.

In the rejected alternatives, under point 2) I read "we can copy the member
id to the config files". I believe it means to say "member name" unless I'm
missing something about reusing member ids. Also below I read: "By allowing
consumers to optionally specifying a member id" which probably implies
"member name" again. In a sense this section highlights a potential
confusion between member name and member id. I wonder if we could come up
with a better term for the new field. StaticTag, StaticLabel, or even
StaticName are some suggestions that could potentially help with confusion
between MemberId and MemberName and what corresponds to what. But I
wouldn't like to disrupt the discussion with naming conventions too much at
this point. I just mention it here as a thought.

Looking forward to see the final details of this KIP. Great work so far!

Konstantine


On Tue, Nov 20, 2018 at 4:23 AM Boyang Chen  wrote:

>

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Konstantine Karantasis]

Hi Boyang.

Thanks for preparing this KIP! It is making good progress and will be a
great improvement for stateful Kafka applications.

Apologies for my late reply, I was away for a while. Lots of great comments
so far, so I'll probably second most of them in what I suggest below at
this point.

When I first read the KIP, I wanted to start at the end with something that
wasn't highlighted a lot. That was the topic related to handling duplicate
members. I see now that the initial suggestion of handling this situation
during offset commit has been removed, and I agree with that. Issues
related to membership seem to be handled better when the member joins the
group rather than when it tries to commit offsets. This also simplifies how
many request types need to change in order to incorporate the new member
name field.

I also agree with what Jason and Guozhang have said regarding timeouts.
Although semantically, it's easier to think of every operation having its
own timeout, operationally this can become a burden. Thus, consolidation
seems preferable here. The definition of embedded protocols on top of the
base group membership protocol for rebalancing gives enough flexibility to
address such needs in each client component separately.

Finally, some minor comments:
In a few places the new/proposed changes are referred to as "current".
Which is a bit confusing considering that there is a protocol in place
already, and by "current" someone might understand the existing one. I'd
recommend using new/proposed or equivalent when referring to changes
introduced with KIP-345 and current/existing or equivalent when referring
to existing behavior.

There's the following sentence in the "Public Interfaces" section:
"Since for many stateful consumer/stream applications, the state shuffling
is more painful than short time partial unavailability."
However, my understanding is that the changes proposed with KIP-345 will
not exploit any partial availability. A suggestion for dealing with
temporary imbalances has been made in "Incremental Cooperative Rebalancing"
which can work well with KIP-345, but here I don't see proposed changes
that suggest that some resources (e.g. partitions) will keep being used
while others will not be utilized. Thus, you might want to adjust this
sentence. Correct me if I'm missing something related to that.

In the rejected alternatives, under point 2) I read "we can copy the member
id to the config files". I believe it means to say "member name" unless I'm
missing something about reusing member ids. Also below I read: "By allowing
consumers to optionally specifying a member id" which probably implies
"member name" again. In a sense this section highlights a potential
confusion between member name and member id. I wonder if we could come up
with a better term for the new field. StaticTag, StaticLabel, or even
StaticName are some suggestions that could potentially help with confusion
between MemberId and MemberName and what corresponds to what. But I
wouldn't like to disrupt the discussion with naming conventions too much at
this point. I just mention it here as a thought.

Looking forward to see the final details of this KIP. Great work so far!

Konstantine


On Tue, Nov 20, 2018 at 4:23 AM Boyang Chen  wrote:

> Thanks Guozhang for the great summary here, and I have been following up
> the action items here.
>
>
>   1.  I already updated the KIP to remove the expansion timeout and
> registration timeout. Great to see them being addressed in client side!
>   2.  I double checked the design and I believe that it is ok to have both
> static member and dynamic member co-exist in the same group. So the upgrade
> shouldn't be destructive and we are removing the two membership protocol
> switching APIs.
>   3.  I only have question about this one. I'm still reading the KafkaApis
> code here. Should I just use the same authorization logic for
> ForceStaticRebalanceRequest as JoinGroupRequest?
>   4.  I'm very excited to see this work with K8! Like you suggested, this
> feature could be better addressed in a separate KIP because it is pretty
> independent. I could start drafting the KIP once the current proposal is
> approved.
>   5.  I believe that we don't need fencing in offset commit request, since
> duplicate member.name issue could be handled by join group request. We
> shall reject join group with known member name but no member id (which
> means we already have an active member using this identity).
>   6.  I agree to remove that internal config once we move forward with
> static membership. And I already removed the entire section from the KIP.
>
> Let me know if you have other concerns.
>
> Best,
> Boyang
> ____________
> From: Guozhang Wang 
> Sent: Tuesday, November 20, 2018 4:21 PM
> To: d

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

Thanks Guozhang for the great summary here, and I have been following up the 
action items here.


  1.  I already updated the KIP to remove the expansion timeout and 
registration timeout. Great to see them being addressed in client side!
  2.  I double checked the design and I believe that it is ok to have both 
static member and dynamic member co-exist in the same group. So the upgrade 
shouldn't be destructive and we are removing the two membership protocol 
switching APIs.
  3.  I only have question about this one. I'm still reading the KafkaApis code 
here. Should I just use the same authorization logic for 
ForceStaticRebalanceRequest as JoinGroupRequest?
  4.  I'm very excited to see this work with K8! Like you suggested, this 
feature could be better addressed in a separate KIP because it is pretty 
independent. I could start drafting the KIP once the current proposal is 
approved.
  5.  I believe that we don't need fencing in offset commit request, since 
duplicate member.name issue could be handled by join group request. We shall 
reject join group with known member name but no member id (which means we 
already have an active member using this identity).
  6.  I agree to remove that internal config once we move forward with static 
membership. And I already removed the entire section from the KIP.

Let me know if you have other concerns.

Best,
Boyang

From: Guozhang Wang 
Sent: Tuesday, November 20, 2018 4:21 PM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hello Boyang,

Thanks a lot for the KIP! It is a great write-up and I appreciate your
patience answering to the feedbacks from the community. I'd like to add my
2cents here:

1. By introducing another two timeout configs, registration_timeout and
expansion_timeout, we are effectively having four timeout configs: session
timeout, rebalance timeout (configured as "max.poll.interval.ms" on client
side), and these two. Interplaying these timeout configs can be quite hard
for users with such complexity, and hence I'm wondering if we can simplify
the situation with as less possible timeout configs as possible. Here is a
concrete suggestion I'd like propose:

1.a) Instead of introducing a registration_timeout in addition to the
session_timeout for static members, we can just reuse the session_timeout
and ask users to set it to a larger value when they are upgrading a dynamic
client to a static client by setting the "member.name" at the same time. By
default, the broker-side min.session.timeout is 6 seconds and
max.session.timeout is 5 minutes, which seems reasonable to me (we can of
course modify this broker config to enlarge the valid interval if we want
in practice). And then we should also consider removing the condition for
marking a client as failed if the rebalance timeout has reached while the
JoinGroup was not received, so that the semantics of session_timeout and
rebalance_timeout are totally separated: the former is only used to
determine if a consumer member of the group should be marked as failed and
kicked out of the group, and the latter is only used to determine the
longest time coordinator should wait for PREPARE_REBALANCE phase. In other
words if a member did not send the JoinGroup in time of the
rebalance_timeout, we still include it in the new generation of the group
and use its old subscription info to send to leader for assignment. Later
if the member came back with HeartBeat request, we can still follow the
normal path to bring it to the latest generation while checking that its
sent JoinGroup request contains the same subscription info as we used to
assign the partitions previously (which should be likely the case in
practice). In addition, we should let static members to not send the
LeaveGroup request when it is gracefully shutdown, so that a static member
can only be leaving the group if its session has timed out, OR it has been
indicated to not exist in the group any more (details below).

1.b) We have a parallel discussion about Incremental Cooperative
Rebalancing, in which we will encode the "when to rebalance" logic at the
application level, instead of at the protocol level. By doing this we can
also enable a few other optimizations, e.g. at the Streams level to first
build up the state store as standby tasks and then trigger a second
rebalance to actually migrate the active tasks while keeping the actual
rebalance latency and hence unavailability window to be small (
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FKAFKA-6145data=02%7C01%7C%7C7272fe45059f4afa205208d64ec12c1e%7C84df9e7fe9f640afb435%7C1%7C0%7C636782988907214421sdata=Etn5Y%2BVFTo0GQ1BydPVMUUYbEf04wyhqzKrs3OjCmC8%3Dreserved=0).
 I'd propose we align
KIP-345 along with this idea, and hence do not add the expansion_timeout as
part of the protocol layer, but only do that at

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Guozhang Wang]

Hello Boyang,

Thanks a lot for the KIP! It is a great write-up and I appreciate your
patience answering to the feedbacks from the community. I'd like to add my
2cents here:

1. By introducing another two timeout configs, registration_timeout and
expansion_timeout, we are effectively having four timeout configs: session
timeout, rebalance timeout (configured as "max.poll.interval.ms" on client
side), and these two. Interplaying these timeout configs can be quite hard
for users with such complexity, and hence I'm wondering if we can simplify
the situation with as less possible timeout configs as possible. Here is a
concrete suggestion I'd like propose:

1.a) Instead of introducing a registration_timeout in addition to the
session_timeout for static members, we can just reuse the session_timeout
and ask users to set it to a larger value when they are upgrading a dynamic
client to a static client by setting the "member.name" at the same time. By
default, the broker-side min.session.timeout is 6 seconds and
max.session.timeout is 5 minutes, which seems reasonable to me (we can of
course modify this broker config to enlarge the valid interval if we want
in practice). And then we should also consider removing the condition for
marking a client as failed if the rebalance timeout has reached while the
JoinGroup was not received, so that the semantics of session_timeout and
rebalance_timeout are totally separated: the former is only used to
determine if a consumer member of the group should be marked as failed and
kicked out of the group, and the latter is only used to determine the
longest time coordinator should wait for PREPARE_REBALANCE phase. In other
words if a member did not send the JoinGroup in time of the
rebalance_timeout, we still include it in the new generation of the group
and use its old subscription info to send to leader for assignment. Later
if the member came back with HeartBeat request, we can still follow the
normal path to bring it to the latest generation while checking that its
sent JoinGroup request contains the same subscription info as we used to
assign the partitions previously (which should be likely the case in
practice). In addition, we should let static members to not send the
LeaveGroup request when it is gracefully shutdown, so that a static member
can only be leaving the group if its session has timed out, OR it has been
indicated to not exist in the group any more (details below).

1.b) We have a parallel discussion about Incremental Cooperative
Rebalancing, in which we will encode the "when to rebalance" logic at the
application level, instead of at the protocol level. By doing this we can
also enable a few other optimizations, e.g. at the Streams level to first
build up the state store as standby tasks and then trigger a second
rebalance to actually migrate the active tasks while keeping the actual
rebalance latency and hence unavailability window to be small (
https://issues.apache.org/jira/browse/KAFKA-6145). I'd propose we align
KIP-345 along with this idea, and hence do not add the expansion_timeout as
part of the protocol layer, but only do that at the application's
coordinator / assignor layer (Connect, Streams, etc). We can still,
deprecate the "*group.initial.rebalance.delay.ms
*" though as part of this KIP
since we have discussed about its limit and think it is actually not a very
good design and could be replaced with client-side logic above.


2. I'd like to see your thoughts on the upgrade path for this KIP. More
specifically, let's say after we have upgraded broker version to be able to
recognize the new versions of JoinGroup request and the admin requests, how
should we upgrade the clients and enable static groups? On top of my head
if we do a rolling bounce in which we set the member.name config as well as
optionally increase the session.timeout config when we bounce each
instance, then during this rolling bounces we will have a group contained
with both dynamic members and static members. It means that we should have
the group to allow such scenario (i.e. we cannot reject JoinGroup requests
from dynamic members), and hence the "member.name" -> "member.id" mapping
will only be partial at this scenario. Also could you describe if the
upgrade to the first version that support this feature would ever get any
benefits, or only the future upgrade path for rolling bounces could get
benefits out of this feature?

If that's the case and we will do 1) as suggested above, do we still need
the enableStaticMembership and enableDynamicMembership admin requests any
more? Seems it is not necessary any more as we will only have the notion of
"dynamic or static members" that can co-exist in a group while there no
notion of "dynamic or static groups", and hence these two requests are not
needed anymore.


3. We need to briefly talk about the implications for ACL as we introduce
new admin requests that are related to a specific group.id. 

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Dong Lin]

Hey Boyang,

Thanks for the proposal! This is very useful. I have some comments below:

1) The motivation currently explicitly states that the goal is to improve
performance for heavy state application. It seems that the motivation can
be stronger with the following use-case. Currently for MirrorMaker cluster
with e.g. 100 MirrorMaker processes, it will take a long time to rolling
bounce the entire MirrorMaker cluster. Each MirrorMaker process restart
will trigger a rebalance which currently pause the consumption of the all
partitions of the MirrorMaker cluster. With the change stated in this
patch, as long as a MirrorMaker can restart within the specified timeout
(e.g. 2 minutes), then we only need constant number of rebalance (e.g. for
leader restart) for the entire rolling bounce, which will significantly
improves the availability of the MirrorMaker pipeline. In my opinion, the
main benefit of the KIP is to avoid unnecessary rebalance if the consumer
process can be restarted within soon, which helps performance even if
overhead of state shuffling for a given process is small.

2) In order to simplify the KIP reading, can you follow the writeup style
of other KIP (e.g. KIP-98) and list the interface change such as new
configs (e.g. registration timeout), new request/response, new AdminClient
API and new error code (e.g. DUPLICATE_STATIC_MEMBER)? Currently some of
these are specified in the Proposed Change section which makes it a bit
inconvenient to understand the new interface that will be exposed to user.
Explanation of the current two-phase rebalance protocol probably can be
moved out of public interface section.

3) There are currently two version of JoinGroupRequest in the KIP and only
one of them has field memberId. This seems confusing.

4) It is mentioned in the KIP that "An admin API to force rebalance could
be helpful here, but we will make a call once we finished the major
implementation". So this seems to be still an open question in the current
design. We probably want to agree on this before voting for the KIP.

5) The KIP currently adds new config MEMBER_NAME for consumer. Can you
specify the name of the config key and the default config value? Possible
default values include empty string or null (similar to transaction.id in
producer config).

6) Regarding the use of the topic "static_member_map" to persist member
name map, currently if consumer coordinator broker goes offline, rebalance
is triggered and consumers will try connect to the new coordinator. If
these consumers can connect to the new coordinator within
max.poll.interval.ms which by default is 5 minutes, given that broker can
use a deterministic algorithm to determine the partition -> member_name
mapping, each consumer should get assigned the same set of partitions
without requiring state shuffling. So it is not clear whether we have a
strong use-case for this new logic. Can you help clarify what is the
benefit of using topic "static_member_map" to persist member name map?

7) Regarding the introduction of the expensionTimeoutMs config, it is
mentioned that "we are using expansion timeout to replace rebalance
timeout, which is configured by max.poll.intervals from client side, and
using registration timeout to replace session timeout". Currently the
default max.poll.interval.ms is configured to be 5 minutes and there will
be only one rebalance if all new consumers can join within 5 minutes. So it
is not clear whether we have a strong use-case for this new config. Can you
explain what is the benefit of introducing this new config?

8) It is mentioned that "To distinguish between previous version of
protocol, we will also increase the join group request version to v4 when
MEMBER_NAME is set" and "If the broker version is not the latest (< v4),
the join group request shall be downgraded to v3 without setting the member
Id". It is probably simpler to just say that this feature is enabled if
JoinGroupRequest V4 is supported on both client and broker and MEMBER_NAME
is configured with non-empty string.

9) It is mentioned that broker may return NO_STATIC_MEMBER_INFO_SET error
in OffsetCommitResponse for "commit requests under static membership". Can
you clarify how broker determines whether the commit request is under
static membership?

Thanks,
Dong

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

Thanks Colin and Jason for the further inputs!

I understand that k8 integration will be the long term solution for KStream to 
serve as a streaming platform, and I buy in the idea of "pre-registration" 
which could save us from using timeout tricks.


> If we have a way to indicate the
> expected group members, then the group can respond to a change much more
> quickly. There would be no need to wait 5 minutes for all members to join
> and it would be robust in the presence of failures. Ironically, static
> membership in this case makes the group more dynamic ;).

The caveat here is that existing KStream users are probably not easy to migrate 
to K8 or config other management tools immediately (including us), so in the 
static membership design I always try to minimize the amount of operation work 
needed by the client, hoping the user just needs to provide unique id 
generation and two timeout configs to easy onboard. This way it would be much 
easier to onboard static membership for all KStream users, not only benefiting 
K8 equipped users. The expected member list would be nice to have, which I 
would definitely like to discuss in a follow-up KIP. Do you think this makes 
sense?

> I think both of these issues could be solved by having some broker-side 
> metadata about groups which is configured through the admin client.  If there 
> was an "expected group size," stored on the broker-side, then we could 
> rebalance immediately whenever the group size reached that size.  Otherwise, 
> we could apply the rebalance delay, like now.  This would give lower latency 
> when setting things up.

Setting expected group size in administrative way is a good suggestion, however 
we need to realize that in daily operation, we could always have transit 
failures or instance unavailable (AWS EC2 spin up failed) which invalidates our 
expectation. For example we are scaling up from 4 to 8 hosts, we would expect 
following operations:

  1.  Use admin tool to set group size from 4 to 8
  2.  Spin up new hosts...
  3.  Group will rebalance when we hit 8 join group requests

If in step 2 we failed to spin up one single host, the scale up mechanism will 
fail and get stuck until we fixed the problem by forcing rebalance the fleet. 
To fast scale up, 7 hosts is still better than 4 hosts IMO, and without 
expansion timeout we are paying extra loading time and human monitoring effort 
to trigger rebalance if necessary (we are introducing a forceStaticRebalance 
API here). So expansion timeout and registration timeout are useful to reduce 
mental burden when one operates the member group, which should be helpful.

> Expected group size is just an expectation, so the group would be allowed to 
> get bigger than that.  We could also have another number which was the > 
> maximum group size.  This really would be a hard upper limit on the size of 
> the group, which admins could optionally configure.

The maximum group size is discussed in this 
JIRA<https://issues.apache.org/jira/browse/KAFKA-7641>. It is also useful for 
dynamic membership, thanks for the suggestion!

> When a new client joined a group, the server could send back a unique random 
> 64-bit member ID.  The client could hold on to this ID and use it whenever it 
> > rejoined the group after a failure.  Since the ID is random and provided by 
> the server, it can't be spoofed or accidentally reused by a misconfigured 
> client.

It is actually the current consumer behavior, and we are discussing whether to 
enforce member id when accepting join group request in this 
JIRA<https://issues.apache.org/jira/browse/KAFKA-7610>. Note that we don't have 
way to persist member ids generated by broker on client side through restarts. 
Each time restarted consumers will use "unknown member id" to join again. This 
approach has been discussed<https://github.com/apache/kafka/pull/5176> before 
and we believe it is fragile to remember ids on client side. That's why we 
propose KIP-345 on top of it.

Thanks a lot!

Boyang
________
From: Colin McCabe 
Sent: Saturday, November 17, 2018 2:39 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

As Jason said, there are definitely scenarios where we know how many group 
members we expect ahead of time.  It would be nice if we could distinguish 
between the error case of "we expected 5 clients in the group, but one failed" 
and a case like "4 clients started up quickly but the 5th took an extra 2 
seconds."  We can sandbag the group rebalance delay, but that's a hack which 
has clear disadvantages.

It would also be nice to be able to detect when a group member left the group 
briefly but then came back.

I think both of these issues could be solved by having some broker-side 
metadata about groups whic

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Colin McCabe]

would be an attractive
> safety net in some cases. Perhaps it would be good enough if we have a way
> to pre-register group members administratively? Members can still be
> expired due to inactivity and we would have a way to get around the
> rebalance delays. Would that work?
> 
> Thanks,
> Jason
> 
> 
> On Wed, Nov 14, 2018 at 10:24 PM, Boyang Chen  wrote:
> 
> > Thank you for the clarification Jason! The proposals make sense here and
> > let me continue the discussion.
> >
> > > Then the ids would be determined using some convention. Most likely, we
> > would just use sequential numbers 0, 1, 2,
> > > etc. We do the same thing for partition ids.
> >
> >
> > If we initialize a set of member names (I assume ids = names here) on
> > broker through Admin API, the client needs to pick up this information
> > simultaneously which I doubt if there is a generic way to achieve that? It
> > would also make the scaling operations difficult if we need to define the
> > member names every time we change the member set which is an extra
> > operation burden. From my daily ops experience, dynamically generate member
> > names on client side would be easier. Is there a good approach to address
> > this issue?
> >
> > > I was thinking that the registration is specified ahead of time and
> > remains valid until changed. It would be more like a
> > > replica assignment. We don't move partitions just because a broker is
> > down.
> > > The expectation is that it the broker will eventually return.
> >
> >
> > At Pinterest we maintain 24-7 SLA for 15 - 30 minutes reaction to all the
> > critical streaming services abnormality. One of the burden was the night
> > shift which requires the oncaller to quickly resolve the issue and get the
> > streaming application back on track, however there is a chance of miss. My
> > concern was that if we forfeit the timeout on static membership to trigger
> > rebalance, missing some pages during midnight could be negatively impacting
> > the system performance since we may realize that some partitions stop
> > working for a couple of hours already until next morning. So registration
> > timeout serves as the "last line of defense" to guarantee liveness if no
> > human intervention jumps in.
> >
> >
> > I'm very interested in the replication protocol currently implemented on
> > Kafka, but I'm not familiar with it. If we do have mechanism to handle
> > issues like I mentioned above for replication (auto healing during
> > mid-night if one broker is never back), we could continue discussing the
> > new approaches to have basic guarantee of consumer group liveness.
> >
> >
> > The discussion so far is to make sure that all the design approaches we
> > have taken are pointing to real scenarios. Once we clarify the scenarios,
> > we would definitely propose better solution on top of it. I hope these
> > discussions make sense. Thanks again for helping make the design solid!
> >
> >
> > Boyang
> >
> > 
> > From: Jason Gustafson 
> > Sent: Thursday, November 15, 2018 9:54 AM
> > To: dev
> > Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> > specifying member id
> >
> > >
> > > I feel this would make the current protocol harder to use. For example,
> > on
> > > KStream we typically would expect (number of instances * number of
> > threads
> > > per instance) consumers. Giving out all this many member names in a list
> > > may not be easy, compared with dynamic generation of member names, at
> > least
> > > for KStream use case.
> >
> >
> > That's a fair point. What I had in mind is for the API to specify the
> > number of consumers in the group. Then the ids would be determined using
> > some convention. Most likely, we would just use sequential numbers 0, 1, 2,
> > etc. We do the same thing for partition ids. Streams folks can chime in and
> > say whether that would work or not.
> >
> > So this suggests we will not rely on heartbeat and commit interval to
> > > trigger rebalance? Even in static membership, I feel tracking active
> > > members is still required to be handled by broker since not all users are
> > > fully equipped with monitoring tools, otherwise users will feel
> > > uncomfortable using static membership solely for reducing rebalance
> > purpose.
> >
> >
> > Let me clarify. I think it is still useful to distinguish the liveness of
> > the consum

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Jason Gustafson]

 on static membership to trigger
> rebalance, missing some pages during midnight could be negatively impacting
> the system performance since we may realize that some partitions stop
> working for a couple of hours already until next morning. So registration
> timeout serves as the "last line of defense" to guarantee liveness if no
> human intervention jumps in.
>
>
> I'm very interested in the replication protocol currently implemented on
> Kafka, but I'm not familiar with it. If we do have mechanism to handle
> issues like I mentioned above for replication (auto healing during
> mid-night if one broker is never back), we could continue discussing the
> new approaches to have basic guarantee of consumer group liveness.
>
>
> The discussion so far is to make sure that all the design approaches we
> have taken are pointing to real scenarios. Once we clarify the scenarios,
> we would definitely propose better solution on top of it. I hope these
> discussions make sense. Thanks again for helping make the design solid!
>
>
> Boyang
>
> 
> From: Jason Gustafson 
> Sent: Thursday, November 15, 2018 9:54 AM
> To: dev
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> >
> > I feel this would make the current protocol harder to use. For example,
> on
> > KStream we typically would expect (number of instances * number of
> threads
> > per instance) consumers. Giving out all this many member names in a list
> > may not be easy, compared with dynamic generation of member names, at
> least
> > for KStream use case.
>
>
> That's a fair point. What I had in mind is for the API to specify the
> number of consumers in the group. Then the ids would be determined using
> some convention. Most likely, we would just use sequential numbers 0, 1, 2,
> etc. We do the same thing for partition ids. Streams folks can chime in and
> say whether that would work or not.
>
> So this suggests we will not rely on heartbeat and commit interval to
> > trigger rebalance? Even in static membership, I feel tracking active
> > members is still required to be handled by broker since not all users are
> > fully equipped with monitoring tools, otherwise users will feel
> > uncomfortable using static membership solely for reducing rebalance
> purpose.
>
>
> Let me clarify. I think it is still useful to distinguish the liveness of
> the consumer using the heartbeat mechanism and to propagate that
> information to the leader during rebalances. This gives the group leader
> the option (but not the requirement) to change the partition assignment for
> inactive members. My suggestion was more about how long the static
> registration would remain valid. In the current proposal we have a
> registration timeout, and I think Mayuresh is fair to point out the
> potential confusion with the session timeout. It's kind of the same, but
> not exactly. So instead, I was thinking that the registration is specified
> ahead of time and remains valid until changed. It would be more like a
> replica assignment. We don't move partitions just because a broker is down.
> The expectation is that it the broker will eventually return. Similarly for
> stateful applications, we would have the option to do the same thing. The
> registration is fixed until someone changes it.
>
> Does that make sense?
>
> Thanks,
> Jason
>
>
> On Wed, Nov 14, 2018 at 2:46 PM, Boyang Chen  wrote:
>
> > Thanks Jason for the suggestions! I update the KIP with full schema
> > changes.
> >
> > > we offer an admin API that lets a user define the expected members of
> > the group.
> >
> > I feel this would make the current protocol harder to use. For example,
> on
> > KStream we typically would expect (number of instances * number of
> threads
> > per instance) consumers. Giving out all this many member names in a list
> > may not be easy, compared with dynamic generation of member names, at
> least
> > for KStream use case.
> >
> > > For the sake of discussion, I was wondering if we could just say that
> > static members do not expire.
> >
> >
> > So this suggests we will not rely on heartbeat and commit interval to
> > trigger rebalance? Even in static membership, I feel tracking active
> > members is still required to be handled by broker since not all users are
> > fully equipped with monitoring tools, otherwise users will feel
> > uncomfortable using static membership solely for reducing rebalance
> purpose.
> >
> >
> > For the admin API design, I'm simplifying the join group request
> handling

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

Thank you for the clarification Jason! The proposals make sense here and let me 
continue the discussion.

> Then the ids would be determined using some convention. Most likely, we would 
> just use sequential numbers 0, 1, 2,
> etc. We do the same thing for partition ids.


If we initialize a set of member names (I assume ids = names here) on broker 
through Admin API, the client needs to pick up this information simultaneously 
which I doubt if there is a generic way to achieve that? It would also make the 
scaling operations difficult if we need to define the member names every time 
we change the member set which is an extra operation burden. From my daily ops 
experience, dynamically generate member names on client side would be easier. 
Is there a good approach to address this issue?

> I was thinking that the registration is specified ahead of time and remains 
> valid until changed. It would be more like a
> replica assignment. We don't move partitions just because a broker is down.
> The expectation is that it the broker will eventually return.


At Pinterest we maintain 24-7 SLA for 15 - 30 minutes reaction to all the 
critical streaming services abnormality. One of the burden was the night shift 
which requires the oncaller to quickly resolve the issue and get the streaming 
application back on track, however there is a chance of miss. My concern was 
that if we forfeit the timeout on static membership to trigger rebalance, 
missing some pages during midnight could be negatively impacting the system 
performance since we may realize that some partitions stop working for a couple 
of hours already until next morning. So registration timeout serves as the 
"last line of defense" to guarantee liveness if no human intervention jumps in.


I'm very interested in the replication protocol currently implemented on Kafka, 
but I'm not familiar with it. If we do have mechanism to handle issues like I 
mentioned above for replication (auto healing during mid-night if one broker is 
never back), we could continue discussing the new approaches to have basic 
guarantee of consumer group liveness.


The discussion so far is to make sure that all the design approaches we have 
taken are pointing to real scenarios. Once we clarify the scenarios, we would 
definitely propose better solution on top of it. I hope these discussions make 
sense. Thanks again for helping make the design solid!


Boyang


From: Jason Gustafson 
Sent: Thursday, November 15, 2018 9:54 AM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

>
> I feel this would make the current protocol harder to use. For example, on
> KStream we typically would expect (number of instances * number of threads
> per instance) consumers. Giving out all this many member names in a list
> may not be easy, compared with dynamic generation of member names, at least
> for KStream use case.


That's a fair point. What I had in mind is for the API to specify the
number of consumers in the group. Then the ids would be determined using
some convention. Most likely, we would just use sequential numbers 0, 1, 2,
etc. We do the same thing for partition ids. Streams folks can chime in and
say whether that would work or not.

So this suggests we will not rely on heartbeat and commit interval to
> trigger rebalance? Even in static membership, I feel tracking active
> members is still required to be handled by broker since not all users are
> fully equipped with monitoring tools, otherwise users will feel
> uncomfortable using static membership solely for reducing rebalance purpose.


Let me clarify. I think it is still useful to distinguish the liveness of
the consumer using the heartbeat mechanism and to propagate that
information to the leader during rebalances. This gives the group leader
the option (but not the requirement) to change the partition assignment for
inactive members. My suggestion was more about how long the static
registration would remain valid. In the current proposal we have a
registration timeout, and I think Mayuresh is fair to point out the
potential confusion with the session timeout. It's kind of the same, but
not exactly. So instead, I was thinking that the registration is specified
ahead of time and remains valid until changed. It would be more like a
replica assignment. We don't move partitions just because a broker is down.
The expectation is that it the broker will eventually return. Similarly for
stateful applications, we would have the option to do the same thing. The
registration is fixed until someone changes it.

Does that make sense?

Thanks,
Jason


On Wed, Nov 14, 2018 at 2:46 PM, Boyang Chen  wrote:

> Thanks Jason for the suggestions! I update the KIP with full schema
> changes.
>
> > we offer an admin API that lets a user define the expected members of
> the gr

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Jason Gustafson]

>
> I feel this would make the current protocol harder to use. For example, on
> KStream we typically would expect (number of instances * number of threads
> per instance) consumers. Giving out all this many member names in a list
> may not be easy, compared with dynamic generation of member names, at least
> for KStream use case.


That's a fair point. What I had in mind is for the API to specify the
number of consumers in the group. Then the ids would be determined using
some convention. Most likely, we would just use sequential numbers 0, 1, 2,
etc. We do the same thing for partition ids. Streams folks can chime in and
say whether that would work or not.

So this suggests we will not rely on heartbeat and commit interval to
> trigger rebalance? Even in static membership, I feel tracking active
> members is still required to be handled by broker since not all users are
> fully equipped with monitoring tools, otherwise users will feel
> uncomfortable using static membership solely for reducing rebalance purpose.


Let me clarify. I think it is still useful to distinguish the liveness of
the consumer using the heartbeat mechanism and to propagate that
information to the leader during rebalances. This gives the group leader
the option (but not the requirement) to change the partition assignment for
inactive members. My suggestion was more about how long the static
registration would remain valid. In the current proposal we have a
registration timeout, and I think Mayuresh is fair to point out the
potential confusion with the session timeout. It's kind of the same, but
not exactly. So instead, I was thinking that the registration is specified
ahead of time and remains valid until changed. It would be more like a
replica assignment. We don't move partitions just because a broker is down.
The expectation is that it the broker will eventually return. Similarly for
stateful applications, we would have the option to do the same thing. The
registration is fixed until someone changes it.

Does that make sense?

Thanks,
Jason


On Wed, Nov 14, 2018 at 2:46 PM, Boyang Chen  wrote:

> Thanks Jason for the suggestions! I update the KIP with full schema
> changes.
>
> > we offer an admin API that lets a user define the expected members of
> the group.
>
> I feel this would make the current protocol harder to use. For example, on
> KStream we typically would expect (number of instances * number of threads
> per instance) consumers. Giving out all this many member names in a list
> may not be easy, compared with dynamic generation of member names, at least
> for KStream use case.
>
> > For the sake of discussion, I was wondering if we could just say that
> static members do not expire.
>
>
> So this suggests we will not rely on heartbeat and commit interval to
> trigger rebalance? Even in static membership, I feel tracking active
> members is still required to be handled by broker since not all users are
> fully equipped with monitoring tools, otherwise users will feel
> uncomfortable using static membership solely for reducing rebalance purpose.
>
>
> For the admin API design, I'm simplifying the join group request handling,
> while using admin tool to switch between static and dynamic membership and
> set the two corresponding timeouts. Do you think this approach makes sense?
> The version one implementation will be much more clean if we handle
> membership change through user intervention.
>
>
> Best,
>
> Boyang
>
> ____________
> From: Jason Gustafson 
> Sent: Wednesday, November 14, 2018 9:31 AM
> To: dev
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> Hey Boyang,
>
> Thanks for the updates. From a high level, I think this actually
> complements Konstantine's writeup on incremental rebalancing. The gap we're
> addressing is providing a way to bind the the partition assignment of a
> group to a set of user-provided ids so that we are not so reliant on the
> group's immediate state. For example, these ids might identify the state
> store volume for particular streams instances. This is basically what you
> need to work well with k8s stateful sets (as far as I understand them).
>
> One key decision is how we would define and update the expected static
> members in a consumer group. The mechanics of the registration and
> expansion timeouts feel a little bit clunky. For the sake of discussion, I
> was wondering if we could just say that static members do not expire.
> Instead, we offer an admin API that lets a user define the expected members
> of the group. This API could be used to both grow and shrink a group. This
> would solve the rebalancing problems when applications are initially
> bootstrapped or when they are restarted beca

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

Thanks Jason for the suggestions! I update the KIP with full schema changes.

> we offer an admin API that lets a user define the expected members of the 
> group.

I feel this would make the current protocol harder to use. For example, on 
KStream we typically would expect (number of instances * number of threads per 
instance) consumers. Giving out all this many member names in a list may not be 
easy, compared with dynamic generation of member names, at least for KStream 
use case.

> For the sake of discussion, I was wondering if we could just say that static 
> members do not expire.


So this suggests we will not rely on heartbeat and commit interval to trigger 
rebalance? Even in static membership, I feel tracking active members is still 
required to be handled by broker since not all users are fully equipped with 
monitoring tools, otherwise users will feel uncomfortable using static 
membership solely for reducing rebalance purpose.


For the admin API design, I'm simplifying the join group request handling, 
while using admin tool to switch between static and dynamic membership and set 
the two corresponding timeouts. Do you think this approach makes sense? The 
version one implementation will be much more clean if we handle membership 
change through user intervention.


Best,

Boyang


From: Jason Gustafson 
Sent: Wednesday, November 14, 2018 9:31 AM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hey Boyang,

Thanks for the updates. From a high level, I think this actually
complements Konstantine's writeup on incremental rebalancing. The gap we're
addressing is providing a way to bind the the partition assignment of a
group to a set of user-provided ids so that we are not so reliant on the
group's immediate state. For example, these ids might identify the state
store volume for particular streams instances. This is basically what you
need to work well with k8s stateful sets (as far as I understand them).

One key decision is how we would define and update the expected static
members in a consumer group. The mechanics of the registration and
expansion timeouts feel a little bit clunky. For the sake of discussion, I
was wondering if we could just say that static members do not expire.
Instead, we offer an admin API that lets a user define the expected members
of the group. This API could be used to both grow and shrink a group. This
would solve the rebalancing problems when applications are initially
bootstrapped or when they are restarted because we would always know how
many members should be in a group. What do you think?

By the way, it would be helpful to include the full schema definition for
any protocol changes in the proposal.

Thanks,
Jason


On Mon, Nov 12, 2018 at 8:56 AM, Boyang Chen  wrote:

> Thanks Mayuresh for the feedback! Do you have a quick example for passing
> in consumer config dynamically? I mainly use Kafka Streams at my daily work
> so probably missing the idea how to do it in the current consumer setting.
>
>
> For differentiating session timeout and registration timeout, I would try
> to enhance the documentation in the first stage to see how people react to
> the confusion (would be great if they feel straightforward!). Since one
> doesn't have to fully understand the difference unless defining the new
> config "member name", for current users we could buy some time to listen to
> their understandings and improve our documentation correspondingly in the
> follow-up KIPs.
>
>
> Boyang
>
> 
> From: Mayuresh Gharat 
> Sent: Sunday, November 11, 2018 1:06 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> Hi Boyang,
>
> Thanks for the reply.
>
> Please find the replies inline below :
> For having a consumer config at runtime, I think it's not necessary to
> address in this KIP because most companies run sidecar jobs through daemon
> software like puppet. It should be easy to change the config through script
> or UI without actual code change. We still want to leave flexibility for
> user to define member name as they like.
>  This might be little different for companies that use configuration
> management tools that does not allow the applications to define/change the
> configs dynamically. For example, if we use something similar to spring to
> pull in the configs for the KafkaConsumer and pass it to the constructor to
> create the KafkaConsumer object, it will be hard to specify a unique value
> to the "MEMBER_NAME" config unless someone deploying the app generates a
> unique string for this config outside the deployment workflow and copies it
> statically before starting up each consumer instance. Unless we can l

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Jason Gustafson]

Hey Boyang,

Thanks for the updates. From a high level, I think this actually
complements Konstantine's writeup on incremental rebalancing. The gap we're
addressing is providing a way to bind the the partition assignment of a
group to a set of user-provided ids so that we are not so reliant on the
group's immediate state. For example, these ids might identify the state
store volume for particular streams instances. This is basically what you
need to work well with k8s stateful sets (as far as I understand them).

One key decision is how we would define and update the expected static
members in a consumer group. The mechanics of the registration and
expansion timeouts feel a little bit clunky. For the sake of discussion, I
was wondering if we could just say that static members do not expire.
Instead, we offer an admin API that lets a user define the expected members
of the group. This API could be used to both grow and shrink a group. This
would solve the rebalancing problems when applications are initially
bootstrapped or when they are restarted because we would always know how
many members should be in a group. What do you think?

By the way, it would be helpful to include the full schema definition for
any protocol changes in the proposal.

Thanks,
Jason


On Mon, Nov 12, 2018 at 8:56 AM, Boyang Chen  wrote:

> Thanks Mayuresh for the feedback! Do you have a quick example for passing
> in consumer config dynamically? I mainly use Kafka Streams at my daily work
> so probably missing the idea how to do it in the current consumer setting.
>
>
> For differentiating session timeout and registration timeout, I would try
> to enhance the documentation in the first stage to see how people react to
> the confusion (would be great if they feel straightforward!). Since one
> doesn't have to fully understand the difference unless defining the new
> config "member name", for current users we could buy some time to listen to
> their understandings and improve our documentation correspondingly in the
> follow-up KIPs.
>
>
> Boyang
>
> 
> From: Mayuresh Gharat 
> Sent: Sunday, November 11, 2018 1:06 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> Hi Boyang,
>
> Thanks for the reply.
>
> Please find the replies inline below :
> For having a consumer config at runtime, I think it's not necessary to
> address in this KIP because most companies run sidecar jobs through daemon
> software like puppet. It should be easy to change the config through script
> or UI without actual code change. We still want to leave flexibility for
> user to define member name as they like.
>  This might be little different for companies that use configuration
> management tools that does not allow the applications to define/change the
> configs dynamically. For example, if we use something similar to spring to
> pull in the configs for the KafkaConsumer and pass it to the constructor to
> create the KafkaConsumer object, it will be hard to specify a unique value
> to the "MEMBER_NAME" config unless someone deploying the app generates a
> unique string for this config outside the deployment workflow and copies it
> statically before starting up each consumer instance. Unless we can loosen
> the criteria for uniqueness of this config value, for each consumer
> instance in the consumer group, I am not sure of a better way of
> addressing this. If we don't want to loosen the criteria, then providing a
> dynamic way to pass this in at runtime, would put the onus of having the
> same unique value each time a consumer is restarted, on to the application
> that is running the consumer.
>
> I just updated the kip about having both "registration timeout" and
> "session timeout". The benefit of having two configs instead of one is to
> reduce the mental burden for operation, for example user just needs to
> unset "member name" to cast back to dynamic membership without worrying
> about tuning the "session timeout" back to a smaller value.
> --- That is a good point. I was thinking, if both the configs are
> specified, it would be confusing for the end user without understanding the
> internals of the consumer and its interaction with group coordinator, as
> which takes precedence when and how it affects the consumer behavior. Just
> my 2 cents.
>
> Thanks,
>
> Mayuresh
>
> On Fri, Nov 9, 2018 at 8:27 PM Boyang Chen  wrote:
>
> > Hey Mayuresh,
> >
> >
> > thanks for the thoughtful questions! Let me try to answer your questions
> > one by one.
> >
> >
> > For having a consumer config at runtime, I think it's not necessary to
> > addres

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

Thanks Mayuresh for the feedback! Do you have a quick example for passing in 
consumer config dynamically? I mainly use Kafka Streams at my daily work so 
probably missing the idea how to do it in the current consumer setting.


For differentiating session timeout and registration timeout, I would try to 
enhance the documentation in the first stage to see how people react to the 
confusion (would be great if they feel straightforward!). Since one doesn't 
have to fully understand the difference unless defining the new config "member 
name", for current users we could buy some time to listen to their 
understandings and improve our documentation correspondingly in the follow-up 
KIPs.


Boyang


From: Mayuresh Gharat 
Sent: Sunday, November 11, 2018 1:06 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hi Boyang,

Thanks for the reply.

Please find the replies inline below :
For having a consumer config at runtime, I think it's not necessary to
address in this KIP because most companies run sidecar jobs through daemon
software like puppet. It should be easy to change the config through script
or UI without actual code change. We still want to leave flexibility for
user to define member name as they like.
 This might be little different for companies that use configuration
management tools that does not allow the applications to define/change the
configs dynamically. For example, if we use something similar to spring to
pull in the configs for the KafkaConsumer and pass it to the constructor to
create the KafkaConsumer object, it will be hard to specify a unique value
to the "MEMBER_NAME" config unless someone deploying the app generates a
unique string for this config outside the deployment workflow and copies it
statically before starting up each consumer instance. Unless we can loosen
the criteria for uniqueness of this config value, for each consumer
instance in the consumer group, I am not sure of a better way of
addressing this. If we don't want to loosen the criteria, then providing a
dynamic way to pass this in at runtime, would put the onus of having the
same unique value each time a consumer is restarted, on to the application
that is running the consumer.

I just updated the kip about having both "registration timeout" and
"session timeout". The benefit of having two configs instead of one is to
reduce the mental burden for operation, for example user just needs to
unset "member name" to cast back to dynamic membership without worrying
about tuning the "session timeout" back to a smaller value.
--- That is a good point. I was thinking, if both the configs are
specified, it would be confusing for the end user without understanding the
internals of the consumer and its interaction with group coordinator, as
which takes precedence when and how it affects the consumer behavior. Just
my 2 cents.

Thanks,

Mayuresh

On Fri, Nov 9, 2018 at 8:27 PM Boyang Chen  wrote:

> Hey Mayuresh,
>
>
> thanks for the thoughtful questions! Let me try to answer your questions
> one by one.
>
>
> For having a consumer config at runtime, I think it's not necessary to
> address in this KIP because most companies run sidecar jobs through daemon
> software like puppet. It should be easy to change the config through script
> or UI without actual code change. We still want to leave flexibility for
> user to define member name as they like.
>
>
> I just updated the kip about having both "registration timeout" and
> "session timeout". The benefit of having two configs instead of one is to
> reduce the mental burden for operation, for example user just needs to
> unset "member name" to cast back to dynamic membership without worrying
> about tuning the "session timeout" back to a smaller value.
>
>
> For backup topic, I think it's a low-level detail which could be addressed
> in the implementation. I feel no preference of adding a new topic vs reuse
> consumer offsets topic. I will do more analysis and make a trade-off
> comparison. Nice catch!
>
>
> I hope the explanations make sense to you. I will keep polishing on the
> edge cases and details.
>
>
> Best,
>
> Boyang
>
> ____________
> From: Mayuresh Gharat 
> Sent: Saturday, November 10, 2018 10:25 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> Hi Boyang,
>
> Thanks for the KIP and sorry for being late to the party. This KIP is
> really useful for us at Linkedin.
>
> I had a few questions :
>
> The idea of having static member name seems nice, but instead of a config,
> would it be possible for it to be pass

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Mayuresh Gharat]

Hi Boyang,

Thanks for the reply.

Please find the replies inline below :
For having a consumer config at runtime, I think it's not necessary to
address in this KIP because most companies run sidecar jobs through daemon
software like puppet. It should be easy to change the config through script
or UI without actual code change. We still want to leave flexibility for
user to define member name as they like.
 This might be little different for companies that use configuration
management tools that does not allow the applications to define/change the
configs dynamically. For example, if we use something similar to spring to
pull in the configs for the KafkaConsumer and pass it to the constructor to
create the KafkaConsumer object, it will be hard to specify a unique value
to the "MEMBER_NAME" config unless someone deploying the app generates a
unique string for this config outside the deployment workflow and copies it
statically before starting up each consumer instance. Unless we can loosen
the criteria for uniqueness of this config value, for each consumer
instance in the consumer group, I am not sure of a better way of
addressing this. If we don't want to loosen the criteria, then providing a
dynamic way to pass this in at runtime, would put the onus of having the
same unique value each time a consumer is restarted, on to the application
that is running the consumer.

I just updated the kip about having both "registration timeout" and
"session timeout". The benefit of having two configs instead of one is to
reduce the mental burden for operation, for example user just needs to
unset "member name" to cast back to dynamic membership without worrying
about tuning the "session timeout" back to a smaller value.
--- That is a good point. I was thinking, if both the configs are
specified, it would be confusing for the end user without understanding the
internals of the consumer and its interaction with group coordinator, as
which takes precedence when and how it affects the consumer behavior. Just
my 2 cents.

Thanks,

Mayuresh

On Fri, Nov 9, 2018 at 8:27 PM Boyang Chen  wrote:

> Hey Mayuresh,
>
>
> thanks for the thoughtful questions! Let me try to answer your questions
> one by one.
>
>
> For having a consumer config at runtime, I think it's not necessary to
> address in this KIP because most companies run sidecar jobs through daemon
> software like puppet. It should be easy to change the config through script
> or UI without actual code change. We still want to leave flexibility for
> user to define member name as they like.
>
>
> I just updated the kip about having both "registration timeout" and
> "session timeout". The benefit of having two configs instead of one is to
> reduce the mental burden for operation, for example user just needs to
> unset "member name" to cast back to dynamic membership without worrying
> about tuning the "session timeout" back to a smaller value.
>
>
> For backup topic, I think it's a low-level detail which could be addressed
> in the implementation. I feel no preference of adding a new topic vs reuse
> consumer offsets topic. I will do more analysis and make a trade-off
> comparison. Nice catch!
>
>
> I hope the explanations make sense to you. I will keep polishing on the
> edge cases and details.
>
>
> Best,
>
> Boyang
>
> ____________
> From: Mayuresh Gharat 
> Sent: Saturday, November 10, 2018 10:25 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> Hi Boyang,
>
> Thanks for the KIP and sorry for being late to the party. This KIP is
> really useful for us at Linkedin.
>
> I had a few questions :
>
> The idea of having static member name seems nice, but instead of a config,
> would it be possible for it to be passed in to the consumer at runtime?
> This is because an app might want to decide the config value at runtime
> using its host information for example, to generate the unique member name.
>
> Also the KIP talks about using the "REGISTRATION_TIMEOUT_MS". I was
> wondering if we can reuse the session timeout here. This might help us to
> have one less config on the consumer.
>
> The KIP also talks about adding another internal topic "static_member_map".
> Would the semantics (GroupCoordinator broker, topic configs) be the same as
> __consumer_offsets topic?
>
> Thanks,
>
> Mayuresh
>
>
> On Wed, Nov 7, 2018 at 12:17 AM Boyang Chen  wrote:
>
> > I took a quick pass of the proposal. First I would say it's a very
> > brilliant initiative from Konstantine and Confluent folks. To draft up a
> > proposal like this needs deep understand

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

Hey Mayuresh,


thanks for the thoughtful questions! Let me try to answer your questions one by 
one.


For having a consumer config at runtime, I think it's not necessary to address 
in this KIP because most companies run sidecar jobs through daemon software 
like puppet. It should be easy to change the config through script or UI 
without actual code change. We still want to leave flexibility for user to 
define member name as they like.


I just updated the kip about having both "registration timeout" and "session 
timeout". The benefit of having two configs instead of one is to reduce the 
mental burden for operation, for example user just needs to unset "member name" 
to cast back to dynamic membership without worrying about tuning the "session 
timeout" back to a smaller value.


For backup topic, I think it's a low-level detail which could be addressed in 
the implementation. I feel no preference of adding a new topic vs reuse 
consumer offsets topic. I will do more analysis and make a trade-off 
comparison. Nice catch!


I hope the explanations make sense to you. I will keep polishing on the edge 
cases and details.


Best,

Boyang


From: Mayuresh Gharat 
Sent: Saturday, November 10, 2018 10:25 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hi Boyang,

Thanks for the KIP and sorry for being late to the party. This KIP is
really useful for us at Linkedin.

I had a few questions :

The idea of having static member name seems nice, but instead of a config,
would it be possible for it to be passed in to the consumer at runtime?
This is because an app might want to decide the config value at runtime
using its host information for example, to generate the unique member name.

Also the KIP talks about using the "REGISTRATION_TIMEOUT_MS". I was
wondering if we can reuse the session timeout here. This might help us to
have one less config on the consumer.

The KIP also talks about adding another internal topic "static_member_map".
Would the semantics (GroupCoordinator broker, topic configs) be the same as
__consumer_offsets topic?

Thanks,

Mayuresh


On Wed, Nov 7, 2018 at 12:17 AM Boyang Chen  wrote:

> I took a quick pass of the proposal. First I would say it's a very
> brilliant initiative from Konstantine and Confluent folks. To draft up a
> proposal like this needs deep understanding of the rebalance protocol! I
> summarized some thoughts here.
>
>
> Overall the motivations of the two proposals align on that:
>
>   1.  Both believe the invariant resource (belonging to the same process)
> should be preserved across rebalance.
>   2.  Transit failures (K8 thread death) shouldn't trigger resource
> redistribution. I don't use rebalance here since part one of the
> cooperative proposal could potentially introduce more rebalances but only
> on must-move resources.
>   3.  Scale up/down and rolling bounce are causing unnecessary resource
> shuffling that need to be mitigated.
>
>
> On motivation level, I think both approach could solve/mitigate the above
> issues. They are just different in design philosophy, or I would say the
> perspective difference between framework user and algorithm designer.
>
>
> Two proposals have different focuses. KIP-345 is trying to place more
> fine-grained control on the broker side to reduce the unnecessary
> rebalances, while keeping the client logic intact. This is pretty intuitive
> cause-effect for normal developers who are not very familiar with rebalance
> protocol. As a developer working with Kafka Streams daily, I'd be happy to
> see a simplified rebalance protocol and just focus on maintaining the
> stream/consumer jobs. Too many rebalances raised my concern on the job
> health. To be concise, static membership has the advantage of reducing
> mental burden.
>
>
> Cooperative proposal takes thoughtful approach on client side. We want to
> have fine-grained control on the join/exit group behaviors and make the
> current dynamic membership better to address above issues. I do feel our
> idea crossed on the delayed rebalance when we scale up/down, which could
> potentially reduce the state shuffling and decouple the behavior from
> session timeout which is already overloaded.  In this sense, I believe both
> approaches would serve well in making "reasonable rebalance" happen at the
> "right timing".
>
>
> However, based on my understanding, either 345 or cooperative rebalancing
> is not solving the problem Mike has proposed: could we do a better job at
> scaling up/down in ideal timing? My initial response was to introduce an
> admin API which now I feel is sub-optimal, in that the goal of smooth
> transition is to make sure the newly u

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Mayuresh Gharat]

focus on judging best timing of scale up/down, the two efforts
> could be aligned. In long term, I feel the more complex improvement of
> consumer protocol should happen on client side instead of server side which
> is easier to test and has less global impact for the entire Kafka
> production cluster.
>
>
> Thanks again to Konstantine, Matthias and other folks in coming up with
> this great client proposal. This is great complementation to KIP 345. In a
> high level, we are not having any collision on the path and both proposals
> are making sense here. Just need better sync to avoid duplicate effort :)
>
>
> Best,
>
> Boyang
>
>
> 
> From: Boyang Chen 
> Sent: Wednesday, November 7, 2018 1:57 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> Thanks Matthias for bringing this awesome proposal up! I shall take a
> deeper look and make a comparison between the two proposals.
>
>
> Meanwhile for the scale down specifically for stateful streaming, we could
> actually introduce a new status called "learner" where the newly up hosts
> could try to catch up with the assigned task progress first before
> triggering the rebalance, from which we don't see a sudden dip on the
> progress. However, it is built on top of the success of KIP-345.
>
>
> 
> From: Matthias J. Sax 
> Sent: Wednesday, November 7, 2018 7:02 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> Hey,
>
> there was quite a pause on this KIP discussion and in the mean time, a
> new design for incremental cooporative rebalance was suggested:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/Incremental+Cooperative+Rebalancing%3A+Support+and+Policies
>
>
> We should make sure that the proposal and this KIP align to each other.
> Thoughts?
>
>
> -Matthias
>
> On 11/5/18 7:31 PM, Boyang Chen wrote:
> > Hey Mike,
> >
> >
> > thanks for the feedback, the two question are very thoughtful!
> >
> >
> >> 1) I am a little confused about the distinction for the leader. If the
> consumer node that was assigned leader does a bounce (goes down and quickly
> comes up) to update application code, will a rebalance be triggered? I > do
> not think a bounce of the leader should trigger a rebalance.
> >
> > For Q1 my intention was to minimize the change within one KIP, since the
> leader rejoining case could be addressed separately.
> >
> >
> >> 2) The timeout for shrink up makes a lot of sense and allows to
> gracefully increase the number of nodes in the cluster. I think we need to
> support graceful shrink down as well. If I set the registration timeout to
> 5 minutes > to handle rolling restarts or intermittent failures without
> shuffling state, I don't want to wait 5 minutes in order for the group to
> rebalance if I am intentionally removing a node from the cluster. I am not
> sure the best way to > do this. One idea I had was adding the ability for a
> CLI or Admin API to force a rebalance of the group. This would allow for an
> admin to trigger the rebalance manually without waiting the entire
> registration timeout on > shrink down. What do you think?
> >
> > For 2) my understanding is that for scaling down case it is better to be
> addressed by CLI tool than code logic, since only by human evaluation we
> could decide whether it is a "right timing" -- the time when all the
> scaling down consumers are offline -- to kick in rebalance. Unless we
> introduce another term on coordinator which indicates the target consumer
> group size, broker will find it hard to decide when to start rebalance. So
> far I prefer to hold the implementation for that, but agree we could
> discuss whether we want to introduce admin API in this KIP or a separate
> one.
> >
> >
> > Thanks again for the proposed ideas!
> >
> >
> > Boyang
> >
> > 
> > From: Mike Freyberger 
> > Sent: Monday, November 5, 2018 6:13 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
> >
> > Boyang,
> >
> > Thanks for updating the KIP. It's shaping up well. Two things:
> >
> > 1) I am a little confused about the distinction for the leader. If the
> consumer node that was assigned leader does a bounce (goes down and quickly
> comes up) to update application code, will a rebalance be triggered? I do
> not 

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

I took a quick pass of the proposal. First I would say it's a very brilliant 
initiative from Konstantine and Confluent folks. To draft up a proposal like 
this needs deep understanding of the rebalance protocol! I summarized some 
thoughts here.


Overall the motivations of the two proposals align on that:

  1.  Both believe the invariant resource (belonging to the same process) 
should be preserved across rebalance.
  2.  Transit failures (K8 thread death) shouldn't trigger resource 
redistribution. I don't use rebalance here since part one of the cooperative 
proposal could potentially introduce more rebalances but only on must-move 
resources.
  3.  Scale up/down and rolling bounce are causing unnecessary resource 
shuffling that need to be mitigated.


On motivation level, I think both approach could solve/mitigate the above 
issues. They are just different in design philosophy, or I would say the 
perspective difference between framework user and algorithm designer.


Two proposals have different focuses. KIP-345 is trying to place more 
fine-grained control on the broker side to reduce the unnecessary rebalances, 
while keeping the client logic intact. This is pretty intuitive cause-effect 
for normal developers who are not very familiar with rebalance protocol. As a 
developer working with Kafka Streams daily, I'd be happy to see a simplified 
rebalance protocol and just focus on maintaining the stream/consumer jobs. Too 
many rebalances raised my concern on the job health. To be concise, static 
membership has the advantage of reducing mental burden.


Cooperative proposal takes thoughtful approach on client side. We want to have 
fine-grained control on the join/exit group behaviors and make the current 
dynamic membership better to address above issues. I do feel our idea crossed 
on the delayed rebalance when we scale up/down, which could potentially reduce 
the state shuffling and decouple the behavior from session timeout which is 
already overloaded.  In this sense, I believe both approaches would serve well 
in making "reasonable rebalance" happen at the "right timing".


However, based on my understanding, either 345 or cooperative rebalancing is 
not solving the problem Mike has proposed: could we do a better job at scaling 
up/down in ideal timing? My initial response was to introduce an admin API 
which now I feel is sub-optimal, in that the goal of smooth transition is to 
make sure the newly up hosts are actually "ready". For example:


We have 4 instance reading from 8 topic partitions (= 8 tasks). At some time we 
would like to scale up to 8 hosts, with the current improvements we could 
reduce 4 potential rebalances to a single one. But the new hosts are yet 
unknown to be "ready" if they need to reconstruct the local state. To be 
actually ready, we need 4 standby tasks running on those empty hosts and leader 
needs to wait for the signal of "replay/reconstruct complete" to actually 
involve them into the main consumer group. Otherwise, rebalance just kills our 
performance since we need to wait indefinite long for task migration.


The scale down is also tricky such that we are not able to define a "true" 
leave of a member. Rebalance immediately after "true" leaves are most optimal 
comparing with human intervention. Does this make sense?


My intuition is that cooperative approach which was implemented on the client 
side could better handle scaling cases than KIP 345, since it involves a lot of 
algorithmic changes to define "replaying" stage, which I feel would 
over-complicate broker logic if implemented on coordinator. If we let 345 focus 
on reducing unnecessary rebalance, and let cooperative approach focus on 
judging best timing of scale up/down, the two efforts could be aligned. In long 
term, I feel the more complex improvement of consumer protocol should happen on 
client side instead of server side which is easier to test and has less global 
impact for the entire Kafka production cluster.


Thanks again to Konstantine, Matthias and other folks in coming up with this 
great client proposal. This is great complementation to KIP 345. In a high 
level, we are not having any collision on the path and both proposals are 
making sense here. Just need better sync to avoid duplicate effort :)


Best,

Boyang



From: Boyang Chen 
Sent: Wednesday, November 7, 2018 1:57 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Thanks Matthias for bringing this awesome proposal up! I shall take a deeper 
look and make a comparison between the two proposals.


Meanwhile for the scale down specifically for stateful streaming, we could 
actually introduce a new status called "learner" where the newly up hosts could 
try to catch up with the assigned task progress first before triggering the 
reb

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

Thanks Matthias for bringing this awesome proposal up! I shall take a deeper 
look and make a comparison between the two proposals.


Meanwhile for the scale down specifically for stateful streaming, we could 
actually introduce a new status called "learner" where the newly up hosts could 
try to catch up with the assigned task progress first before triggering the 
rebalance, from which we don't see a sudden dip on the progress. However, it is 
built on top of the success of KIP-345.



From: Matthias J. Sax 
Sent: Wednesday, November 7, 2018 7:02 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hey,

there was quite a pause on this KIP discussion and in the mean time, a
new design for incremental cooporative rebalance was suggested:

https://cwiki.apache.org/confluence/display/KAFKA/Incremental+Cooperative+Rebalancing%3A+Support+and+Policies


We should make sure that the proposal and this KIP align to each other.
Thoughts?


-Matthias

On 11/5/18 7:31 PM, Boyang Chen wrote:
> Hey Mike,
>
>
> thanks for the feedback, the two question are very thoughtful!
>
>
>> 1) I am a little confused about the distinction for the leader. If the 
>> consumer node that was assigned leader does a bounce (goes down and quickly 
>> comes up) to update application code, will a rebalance be triggered? I > do 
>> not think a bounce of the leader should trigger a rebalance.
>
> For Q1 my intention was to minimize the change within one KIP, since the 
> leader rejoining case could be addressed separately.
>
>
>> 2) The timeout for shrink up makes a lot of sense and allows to gracefully 
>> increase the number of nodes in the cluster. I think we need to support 
>> graceful shrink down as well. If I set the registration timeout to 5 minutes 
>> > to handle rolling restarts or intermittent failures without shuffling 
>> state, I don't want to wait 5 minutes in order for the group to rebalance if 
>> I am intentionally removing a node from the cluster. I am not sure the best 
>> way to > do this. One idea I had was adding the ability for a CLI or Admin 
>> API to force a rebalance of the group. This would allow for an admin to 
>> trigger the rebalance manually without waiting the entire registration 
>> timeout on > shrink down. What do you think?
>
> For 2) my understanding is that for scaling down case it is better to be 
> addressed by CLI tool than code logic, since only by human evaluation we 
> could decide whether it is a "right timing" -- the time when all the scaling 
> down consumers are offline -- to kick in rebalance. Unless we introduce 
> another term on coordinator which indicates the target consumer group size, 
> broker will find it hard to decide when to start rebalance. So far I prefer 
> to hold the implementation for that, but agree we could discuss whether we 
> want to introduce admin API in this KIP or a separate one.
>
>
> Thanks again for the proposed ideas!
>
>
> Boyang
>
> ____________
> From: Mike Freyberger 
> Sent: Monday, November 5, 2018 6:13 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
> specifying member id
>
> Boyang,
>
> Thanks for updating the KIP. It's shaping up well. Two things:
>
> 1) I am a little confused about the distinction for the leader. If the 
> consumer node that was assigned leader does a bounce (goes down and quickly 
> comes up) to update application code, will a rebalance be triggered? I do not 
> think a bounce of the leader should trigger a rebalance.
>
> 2) The timeout for shrink up makes a lot of sense and allows to gracefully 
> increase the number of nodes in the cluster. I think we need to support 
> graceful shrink down as well. If I set the registration timeout to 5 minutes 
> to handle rolling restarts or intermittent failures without shuffling state, 
> I don't want to wait 5 minutes in order for the group to rebalance if I am 
> intentionally removing a node from the cluster. I am not sure the best way to 
> do this. One idea I had was adding the ability for a CLI or Admin API to 
> force a rebalance of the group. This would allow for an admin to trigger the 
> rebalance manually without waiting the entire registration timeout on shrink 
> down. What do you think?
>
> Mike
>
> On 10/30/18, 1:55 AM, "Boyang Chen"  wrote:
>
> Btw, I updated KIP 345 based on my understanding. Feel free to take 
> another round of look:
>
> 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances
> KIP-345:

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Matthias J. Sax]

Hey,

there was quite a pause on this KIP discussion and in the mean time, a
new design for incremental cooporative rebalance was suggested:

https://cwiki.apache.org/confluence/display/KAFKA/Incremental+Cooperative+Rebalancing%3A+Support+and+Policies


We should make sure that the proposal and this KIP align to each other.
Thoughts?


-Matthias

On 11/5/18 7:31 PM, Boyang Chen wrote:
> Hey Mike,
> 
> 
> thanks for the feedback, the two question are very thoughtful!
> 
> 
>> 1) I am a little confused about the distinction for the leader. If the 
>> consumer node that was assigned leader does a bounce (goes down and quickly 
>> comes up) to update application code, will a rebalance be triggered? I > do 
>> not think a bounce of the leader should trigger a rebalance.
> 
> For Q1 my intention was to minimize the change within one KIP, since the 
> leader rejoining case could be addressed separately.
> 
> 
>> 2) The timeout for shrink up makes a lot of sense and allows to gracefully 
>> increase the number of nodes in the cluster. I think we need to support 
>> graceful shrink down as well. If I set the registration timeout to 5 minutes 
>> > to handle rolling restarts or intermittent failures without shuffling 
>> state, I don't want to wait 5 minutes in order for the group to rebalance if 
>> I am intentionally removing a node from the cluster. I am not sure the best 
>> way to > do this. One idea I had was adding the ability for a CLI or Admin 
>> API to force a rebalance of the group. This would allow for an admin to 
>> trigger the rebalance manually without waiting the entire registration 
>> timeout on > shrink down. What do you think?
> 
> For 2) my understanding is that for scaling down case it is better to be 
> addressed by CLI tool than code logic, since only by human evaluation we 
> could decide whether it is a "right timing" -- the time when all the scaling 
> down consumers are offline -- to kick in rebalance. Unless we introduce 
> another term on coordinator which indicates the target consumer group size, 
> broker will find it hard to decide when to start rebalance. So far I prefer 
> to hold the implementation for that, but agree we could discuss whether we 
> want to introduce admin API in this KIP or a separate one.
> 
> 
> Thanks again for the proposed ideas!
> 
> 
> Boyang
> 
> ____________
> From: Mike Freyberger 
> Sent: Monday, November 5, 2018 6:13 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
> specifying member id
> 
> Boyang,
> 
> Thanks for updating the KIP. It's shaping up well. Two things:
> 
> 1) I am a little confused about the distinction for the leader. If the 
> consumer node that was assigned leader does a bounce (goes down and quickly 
> comes up) to update application code, will a rebalance be triggered? I do not 
> think a bounce of the leader should trigger a rebalance.
> 
> 2) The timeout for shrink up makes a lot of sense and allows to gracefully 
> increase the number of nodes in the cluster. I think we need to support 
> graceful shrink down as well. If I set the registration timeout to 5 minutes 
> to handle rolling restarts or intermittent failures without shuffling state, 
> I don't want to wait 5 minutes in order for the group to rebalance if I am 
> intentionally removing a node from the cluster. I am not sure the best way to 
> do this. One idea I had was adding the ability for a CLI or Admin API to 
> force a rebalance of the group. This would allow for an admin to trigger the 
> rebalance manually without waiting the entire registration timeout on shrink 
> down. What do you think?
> 
> Mike
> 
> On 10/30/18, 1:55 AM, "Boyang Chen"  wrote:
> 
> Btw, I updated KIP 345 based on my understanding. Feel free to take 
> another round of look:
> 
> 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances
> KIP-345: Introduce static membership protocol to reduce 
> ...<https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances>
> cwiki.apache.org
> For stateful applications, one of the biggest performance bottleneck is the 
> state shuffling. In Kafka consumer, there is a concept called "rebalance" 
> which means that for given M partitions and N consumers in one consumer 
> group, Kafka will try to balance the load between consumers and ideally have 
> ...
> 
> 
> 
> 
> KIP-345: Introduce static membership protocol to reduce 
> ...<https://cwiki.apache.org/confluence

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

Hey Mike,


thanks for the feedback, the two question are very thoughtful!


> 1) I am a little confused about the distinction for the leader. If the 
> consumer node that was assigned leader does a bounce (goes down and quickly 
> comes up) to update application code, will a rebalance be triggered? I > do 
> not think a bounce of the leader should trigger a rebalance.

For Q1 my intention was to minimize the change within one KIP, since the leader 
rejoining case could be addressed separately.


> 2) The timeout for shrink up makes a lot of sense and allows to gracefully 
> increase the number of nodes in the cluster. I think we need to support 
> graceful shrink down as well. If I set the registration timeout to 5 minutes 
> > to handle rolling restarts or intermittent failures without shuffling 
> state, I don't want to wait 5 minutes in order for the group to rebalance if 
> I am intentionally removing a node from the cluster. I am not sure the best 
> way to > do this. One idea I had was adding the ability for a CLI or Admin 
> API to force a rebalance of the group. This would allow for an admin to 
> trigger the rebalance manually without waiting the entire registration 
> timeout on > shrink down. What do you think?

For 2) my understanding is that for scaling down case it is better to be 
addressed by CLI tool than code logic, since only by human evaluation we could 
decide whether it is a "right timing" -- the time when all the scaling down 
consumers are offline -- to kick in rebalance. Unless we introduce another term 
on coordinator which indicates the target consumer group size, broker will find 
it hard to decide when to start rebalance. So far I prefer to hold the 
implementation for that, but agree we could discuss whether we want to 
introduce admin API in this KIP or a separate one.


Thanks again for the proposed ideas!


Boyang


From: Mike Freyberger 
Sent: Monday, November 5, 2018 6:13 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Boyang,

Thanks for updating the KIP. It's shaping up well. Two things:

1) I am a little confused about the distinction for the leader. If the consumer 
node that was assigned leader does a bounce (goes down and quickly comes up) to 
update application code, will a rebalance be triggered? I do not think a bounce 
of the leader should trigger a rebalance.

2) The timeout for shrink up makes a lot of sense and allows to gracefully 
increase the number of nodes in the cluster. I think we need to support 
graceful shrink down as well. If I set the registration timeout to 5 minutes to 
handle rolling restarts or intermittent failures without shuffling state, I 
don't want to wait 5 minutes in order for the group to rebalance if I am 
intentionally removing a node from the cluster. I am not sure the best way to 
do this. One idea I had was adding the ability for a CLI or Admin API to force 
a rebalance of the group. This would allow for an admin to trigger the 
rebalance manually without waiting the entire registration timeout on shrink 
down. What do you think?

Mike

On 10/30/18, 1:55 AM, "Boyang Chen"  wrote:

Btw, I updated KIP 345 based on my understanding. Feel free to take another 
round of look:


https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances
KIP-345: Introduce static membership protocol to reduce 
...<https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances>
cwiki.apache.org
For stateful applications, one of the biggest performance bottleneck is the 
state shuffling. In Kafka consumer, there is a concept called "rebalance" which 
means that for given M partitions and N consumers in one consumer group, Kafka 
will try to balance the load between consumers and ideally have ...




KIP-345: Introduce static membership protocol to reduce 
...<https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances>
cwiki.apache.org
For stateful applications, one of the biggest performance bottleneck is the 
state shuffling. In Kafka consumer, there is a concept called "rebalance" which 
means that for given M partitions and N consumers in one consumer group, Kafka 
will try to balance the load between consumers and ideally have ...






From: Boyang Chen 
    Sent: Monday, October 29, 2018 12:34 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Thanks everyone for the input on this thread! (Sorry it's been a while) I 
feel that we are very close to the final solution.


Hey Jason and Mike, 

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Mike Freyberger]

Boyang,

Thanks for updating the KIP. It's shaping up well. Two things:

1) I am a little confused about the distinction for the leader. If the consumer 
node that was assigned leader does a bounce (goes down and quickly comes up) to 
update application code, will a rebalance be triggered? I do not think a bounce 
of the leader should trigger a rebalance.

2) The timeout for shrink up makes a lot of sense and allows to gracefully 
increase the number of nodes in the cluster. I think we need to support 
graceful shrink down as well. If I set the registration timeout to 5 minutes to 
handle rolling restarts or intermittent failures without shuffling state, I 
don't want to wait 5 minutes in order for the group to rebalance if I am 
intentionally removing a node from the cluster. I am not sure the best way to 
do this. One idea I had was adding the ability for a CLI or Admin API to force 
a rebalance of the group. This would allow for an admin to trigger the 
rebalance manually without waiting the entire registration timeout on shrink 
down. What do you think?

Mike

On 10/30/18, 1:55 AM, "Boyang Chen"  wrote:

Btw, I updated KIP 345 based on my understanding. Feel free to take another 
round of look:


https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances

KIP-345: Introduce static membership protocol to reduce 
...<https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances>
cwiki.apache.org
For stateful applications, one of the biggest performance bottleneck is the 
state shuffling. In Kafka consumer, there is a concept called "rebalance" which 
means that for given M partitions and N consumers in one consumer group, Kafka 
will try to balance the load between consumers and ideally have ...






From: Boyang Chen 
Sent: Monday, October 29, 2018 12:34 PM
To: dev@kafka.apache.org
    Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Thanks everyone for the input on this thread! (Sorry it's been a while) I 
feel that we are very close to the final solution.


Hey Jason and Mike, I have two quick questions on the new features here:

  1.  so our proposal is that until we add a new static member into the 
group (scale up), we will not trigger rebalance until the "registration 
timeout"( the member has been offline for too long)? How about leader's rejoin 
request, I think we should still trigger rebalance when that happens, since the 
consumer group may have new topics to consume?
  2.  I'm not very clear on the scale up scenario in static membership 
here. Should we fallback to dynamic membership while adding/removing hosts (by 
setting member.name = null), or we still want to add instances with 
`member.name` so that we eventually expand/shrink the static membership? I 
personally feel the easier solution is to spin up new members and wait until 
either the same "registration timeout" or a "scale up timeout" before starting 
the rebalance. What do you think?

Meanwhile I will go ahead to make changes to the KIP with our newly 
discussed items and details. Really excited to see the design has become more 
solid.

Best,
Boyang


From: Jason Gustafson 
Sent: Saturday, August 25, 2018 6:04 AM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hey Mike,

Yeah, that's a good point. A long "registration timeout" may not be a great
idea. Perhaps in practice you'd set it long enough to be able to detect a
failure and provision a new instance. Maybe on the order of 10 minutes is
more reasonable.

In any case, it's probably a good idea to have an administrative way to
force deregistration. One option is to extend the DeleteGroups API with a
list of members names.

-Jason



On Fri, Aug 24, 2018 at 2:21 PM, Mike Freyberger 
wrote:

> Jason,
>
> Regarding step 4 in your proposal which suggests beginning a long timer
> (30 minutes) when a static member leaves the group, would there also be 
the
> ability for an admin to force a static membership expiration?
>
> I’m thinking that during particular types of outages or upgrades users
> would want forcefully remove a static member from the group.
>
> So the user would shut the consumer down normally, which wouldn’t trigger
> a rebalance. Then the user could use an admin CLI tool to force remove 
that
> consumer from the group, so the TopicPartitions that were previously owned
> by that consumer can be releas

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

Btw, I updated KIP 345 based on my understanding. Feel free to take another 
round of look:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances

KIP-345: Introduce static membership protocol to reduce 
...<https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances>
cwiki.apache.org
For stateful applications, one of the biggest performance bottleneck is the 
state shuffling. In Kafka consumer, there is a concept called "rebalance" which 
means that for given M partitions and N consumers in one consumer group, Kafka 
will try to balance the load between consumers and ideally have ...






From: Boyang Chen 
Sent: Monday, October 29, 2018 12:34 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Thanks everyone for the input on this thread! (Sorry it's been a while) I feel 
that we are very close to the final solution.


Hey Jason and Mike, I have two quick questions on the new features here:

  1.  so our proposal is that until we add a new static member into the group 
(scale up), we will not trigger rebalance until the "registration timeout"( the 
member has been offline for too long)? How about leader's rejoin request, I 
think we should still trigger rebalance when that happens, since the consumer 
group may have new topics to consume?
  2.  I'm not very clear on the scale up scenario in static membership here. 
Should we fallback to dynamic membership while adding/removing hosts (by 
setting member.name = null), or we still want to add instances with 
`member.name` so that we eventually expand/shrink the static membership? I 
personally feel the easier solution is to spin up new members and wait until 
either the same "registration timeout" or a "scale up timeout" before starting 
the rebalance. What do you think?

Meanwhile I will go ahead to make changes to the KIP with our newly discussed 
items and details. Really excited to see the design has become more solid.

Best,
Boyang


From: Jason Gustafson 
Sent: Saturday, August 25, 2018 6:04 AM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hey Mike,

Yeah, that's a good point. A long "registration timeout" may not be a great
idea. Perhaps in practice you'd set it long enough to be able to detect a
failure and provision a new instance. Maybe on the order of 10 minutes is
more reasonable.

In any case, it's probably a good idea to have an administrative way to
force deregistration. One option is to extend the DeleteGroups API with a
list of members names.

-Jason



On Fri, Aug 24, 2018 at 2:21 PM, Mike Freyberger 
wrote:

> Jason,
>
> Regarding step 4 in your proposal which suggests beginning a long timer
> (30 minutes) when a static member leaves the group, would there also be the
> ability for an admin to force a static membership expiration?
>
> I’m thinking that during particular types of outages or upgrades users
> would want forcefully remove a static member from the group.
>
> So the user would shut the consumer down normally, which wouldn’t trigger
> a rebalance. Then the user could use an admin CLI tool to force remove that
> consumer from the group, so the TopicPartitions that were previously owned
> by that consumer can be released.
>
> At a high level, we need consumer groups to gracefully handle intermittent
> failures and permanent failures. Currently, the consumer group protocol
> handles permanent failures well, but does not handle intermittent failures
> well (it creates unnecessary rebalances). I want to make sure the overall
> solution here handles both intermittent failures and permanent failures,
> rather than sacrificing support for permanent failures in order to provide
> support for intermittent failures.
>
> Mike
>
> Sent from my iPhone
>
> > On Aug 24, 2018, at 3:03 PM, Jason Gustafson  wrote:
> >
> > Hey Guozhang,
> >
> > Responses below:
> >
> > Originally I was trying to kill more birds with one stone with KIP-345,
> >> e.g. to fix the multi-rebalance issue on starting up / shutting down a
> >> multi-instance client (mentioned as case 1)/2) in my early email), and
> >> hence proposing to have a pure static-membership protocol. But thinking
> >> twice about it I now feel it may be too ambitious and worth fixing in
> >> another KIP.
> >
> >
> > I was considering an extension to support pre-initialization of the
> static
> > members of the group, but I agree we should probably leave this problem
> for
> > future work.
> >
> > 1. Ho

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

Thanks everyone for the input on this thread! (Sorry it's been a while) I feel 
that we are very close to the final solution.


Hey Jason and Mike, I have two quick questions on the new features here:

  1.  so our proposal is that until we add a new static member into the group 
(scale up), we will not trigger rebalance until the "registration timeout"( the 
member has been offline for too long)? How about leader's rejoin request, I 
think we should still trigger rebalance when that happens, since the consumer 
group may have new topics to consume?
  2.  I'm not very clear on the scale up scenario in static membership here. 
Should we fallback to dynamic membership while adding/removing hosts (by 
setting member.name = null), or we still want to add instances with 
`member.name` so that we eventually expand/shrink the static membership? I 
personally feel the easier solution is to spin up new members and wait until 
either the same "registration timeout" or a "scale up timeout" before starting 
the rebalance. What do you think?

Meanwhile I will go ahead to make changes to the KIP with our newly discussed 
items and details. Really excited to see the design has become more solid.

Best,
Boyang


From: Jason Gustafson 
Sent: Saturday, August 25, 2018 6:04 AM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hey Mike,

Yeah, that's a good point. A long "registration timeout" may not be a great
idea. Perhaps in practice you'd set it long enough to be able to detect a
failure and provision a new instance. Maybe on the order of 10 minutes is
more reasonable.

In any case, it's probably a good idea to have an administrative way to
force deregistration. One option is to extend the DeleteGroups API with a
list of members names.

-Jason



On Fri, Aug 24, 2018 at 2:21 PM, Mike Freyberger 
wrote:

> Jason,
>
> Regarding step 4 in your proposal which suggests beginning a long timer
> (30 minutes) when a static member leaves the group, would there also be the
> ability for an admin to force a static membership expiration?
>
> I’m thinking that during particular types of outages or upgrades users
> would want forcefully remove a static member from the group.
>
> So the user would shut the consumer down normally, which wouldn’t trigger
> a rebalance. Then the user could use an admin CLI tool to force remove that
> consumer from the group, so the TopicPartitions that were previously owned
> by that consumer can be released.
>
> At a high level, we need consumer groups to gracefully handle intermittent
> failures and permanent failures. Currently, the consumer group protocol
> handles permanent failures well, but does not handle intermittent failures
> well (it creates unnecessary rebalances). I want to make sure the overall
> solution here handles both intermittent failures and permanent failures,
> rather than sacrificing support for permanent failures in order to provide
> support for intermittent failures.
>
> Mike
>
> Sent from my iPhone
>
> > On Aug 24, 2018, at 3:03 PM, Jason Gustafson  wrote:
> >
> > Hey Guozhang,
> >
> > Responses below:
> >
> > Originally I was trying to kill more birds with one stone with KIP-345,
> >> e.g. to fix the multi-rebalance issue on starting up / shutting down a
> >> multi-instance client (mentioned as case 1)/2) in my early email), and
> >> hence proposing to have a pure static-membership protocol. But thinking
> >> twice about it I now feel it may be too ambitious and worth fixing in
> >> another KIP.
> >
> >
> > I was considering an extension to support pre-initialization of the
> static
> > members of the group, but I agree we should probably leave this problem
> for
> > future work.
> >
> > 1. How this longish static member expiration timeout defined? Is it via a
> >> broker, hence global config, or via a client config which can be
> >> communicated to broker via JoinGroupRequest?
> >
> >
> > I am not too sure. I tend to lean toward server-side configs because they
> > are easier to evolve. If we have to add something to the protocol, then
> > we'll be stuck with it forever.
> >
> > 2. Assuming that for static members, LEAVE_GROUP request will not
> trigger a
> >> rebalance immediately either, similar to session timeout, but only the
> >> longer member expiration timeout, can we remove the internal "
> >> internal.leave.group.on.close" config, which is a quick walk-around
> then?
> >
> >
> > Yeah, I hope we can ultimately get rid of it, but we may need it for
> > compatibility with older brokers. A related questio

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Jason Gustafson]

; will be rejected, right?
> >>>>
> >>>> 4. "just erase the cached mapping, and wait for session timeout to
> >>> trigger
> >>>> rebalance should be sufficient." this is also a bit unclear to me: who
> >>> will
> >>>> erase the cached mapping? Since it is on the broker-side I assume that
> >>>> broker has to do it. Are you suggesting to use a new request for it?
> >>>>
> >>>> 5. "Halfway switch": following 3) above, if your proposal is basically
> >> to
> >>>> let "first join-request wins", and the strategy will stay as is until
> >> all
> >>>> members are gone, then this will also not happen since whoever used
> >>>> different strategy as the first guy who sends join-group request will
> >> be
> >>>> rejected right?
> >>>>
> >>>>
> >>>> Guozhang
> >>>>
> >>>>
> >>>> On Tue, Aug 21, 2018 at 9:28 AM, John Roesler 
> >> wrote:
> >>>>
> >>>>> This sounds good to me!
> >>>>>
> >>>>> Thanks for the time you've spent on it,
> >>>>> -John
> >>>>>
> >>>>> On Tue, Aug 21, 2018 at 12:13 AM Boyang Chen 
> >>>> wrote:
> >>>>>
> >>>>>> Thanks Matthias for the input. Sorry I was busy recently and
> >> haven't
> >>>> got
> >>>>>> time to update this thread. To summarize what we come up so far,
> >> here
> >>>> is
> >>>>> a
> >>>>>> draft updated plan:
> >>>>>>
> >>>>>>
> >>>>>> Introduce a new config called `member.name` which is supposed to
> >> be
> >>>>>> provided uniquely by the consumer client. The broker will maintain
> >> a
> >>>>> cache
> >>>>>> with [key:member.name, value:member.id]. A join group request with
> >>>>>> member.name set will be treated as `static-membership` strategy,
> >> and
> >>>>> will
> >>>>>> reject any join group request without member.name. So this
> >>>> coordination
> >>>>>> change will be differentiated from the `dynamic-membership`
> >> protocol
> >>> we
> >>>>>> currently have.
> >>>>>>
> >>>>>>
> >>>>>> When handling static join group request:
> >>>>>>
> >>>>>>  1.   The broker will check the membership to see whether this is
> >> a
> >>>> new
> >>>>>> member. If new, broker allocate a unique member id, cache the
> >> mapping
> >>>> and
> >>>>>> move to rebalance stage.
> >>>>>>  2.   Following 1, if this is an existing member, broker will not
> >>>> change
> >>>>>> group state, and return its cached member.id and current
> >> assignment.
> >>>>>> (unless this is leader, we shall trigger rebalance)
> >>>>>>  3.   Although Guozhang has mentioned we could rejoin with pair
> >>> member
> >>>>>> name and id, I think for join group request it is ok to leave
> >> member
> >>> id
> >>>>>> blank as member name is the unique identifier. In commit offset
> >>> request
> >>>>> we
> >>>>>> *must* have both.
> >>>>>>
> >>>>>>
> >>>>>> When handling commit offset request, if enabled with static
> >>> membership,
> >>>>>> each time the commit request must have both member.name and
> >>> member.id
> >>>> to
> >>>>>> be identified as a `certificated member`. If not, this means there
> >>> are
> >>>>>> duplicate consumer members with same member name and the request
> >> will
> >>>> be
> >>>>>> rejected to guarantee consumption uniqueness.
> >>>>>>
> >>>>>>
> >>>>>> When rolling restart/shutting down gracefully, the client will
> >> send a
> >>>>>> leave group request (static membership mode). In static membership,
> >>> we
> >>>>&g

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Mike Freyberger]

t; Thanks Matthias for the input. Sorry I was busy recently and
>> haven't
>>>> got
>>>>>> time to update this thread. To summarize what we come up so far,
>> here
>>>> is
>>>>> a
>>>>>> draft updated plan:
>>>>>> 
>>>>>> 
>>>>>> Introduce a new config called `member.name` which is supposed to
>> be
>>>>>> provided uniquely by the consumer client. The broker will maintain
>> a
>>>>> cache
>>>>>> with [key:member.name, value:member.id]. A join group request with
>>>>>> member.name set will be treated as `static-membership` strategy,
>> and
>>>>> will
>>>>>> reject any join group request without member.name. So this
>>>> coordination
>>>>>> change will be differentiated from the `dynamic-membership`
>> protocol
>>> we
>>>>>> currently have.
>>>>>> 
>>>>>> 
>>>>>> When handling static join group request:
>>>>>> 
>>>>>>  1.   The broker will check the membership to see whether this is
>> a
>>>> new
>>>>>> member. If new, broker allocate a unique member id, cache the
>> mapping
>>>> and
>>>>>> move to rebalance stage.
>>>>>>  2.   Following 1, if this is an existing member, broker will not
>>>> change
>>>>>> group state, and return its cached member.id and current
>> assignment.
>>>>>> (unless this is leader, we shall trigger rebalance)
>>>>>>  3.   Although Guozhang has mentioned we could rejoin with pair
>>> member
>>>>>> name and id, I think for join group request it is ok to leave
>> member
>>> id
>>>>>> blank as member name is the unique identifier. In commit offset
>>> request
>>>>> we
>>>>>> *must* have both.
>>>>>> 
>>>>>> 
>>>>>> When handling commit offset request, if enabled with static
>>> membership,
>>>>>> each time the commit request must have both member.name and
>>> member.id
>>>> to
>>>>>> be identified as a `certificated member`. If not, this means there
>>> are
>>>>>> duplicate consumer members with same member name and the request
>> will
>>>> be
>>>>>> rejected to guarantee consumption uniqueness.
>>>>>> 
>>>>>> 
>>>>>> When rolling restart/shutting down gracefully, the client will
>> send a
>>>>>> leave group request (static membership mode). In static membership,
>>> we
>>>>> will
>>>>>> also define `change-group-timeout` to hold on rebalance provided by
>>>>> leader.
>>>>>> So we will wait for all the members to rejoin the group and do
>>> exactly
>>>>> one
>>>>>> rebalance since all members are expected to rejoin within timeout.
>> If
>>>>>> consumer crashes, the join group request from the restarted
>> consumer
>>>> will
>>>>>> be recognized as an existing member and be handled as above
>> condition
>>>> 1;
>>>>>> However, if the consumer takes longer than session timeout to
>> return,
>>>> we
>>>>>> shall still trigger rebalance but it could still try to catch
>>>>>> `change-group-timeout`. If it failed to catch second timeout, its
>>>> cached
>>>>>> state on broker will be garbage collected and trigger a new
>> rebalance
>>>>> when
>>>>>> it finally joins.
>>>>>> 
>>>>>> 
>>>>>> And consider the switch between dynamic to static membership.
>>>>>> 
>>>>>>  1.  Dynamic to static: the first joiner shall revise the
>> membership
>>>> to
>>>>>> static and wait for all the current members to restart, since their
>>>>>> membership is still dynamic. Here our assumption is that the
>> restart
>>>>>> process shouldn't take a long time, as long restart is breaking the
>>>>>> `rebalance timeout` in whatever membership protocol we are using.
>>>> Before
>>>>>> restart, all dynamic member join requests will be rejected.
>>>>>>  2. 

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Jason Gustafson]

; > > name and id, I think for join group request it is ok to leave
> member
> > id
> > > > > blank as member name is the unique identifier. In commit offset
> > request
> > > > we
> > > > > *must* have both.
> > > > >
> > > > >
> > > > > When handling commit offset request, if enabled with static
> > membership,
> > > > > each time the commit request must have both member.name and
> > member.id
> > > to
> > > > > be identified as a `certificated member`. If not, this means there
> > are
> > > > > duplicate consumer members with same member name and the request
> will
> > > be
> > > > > rejected to guarantee consumption uniqueness.
> > > > >
> > > > >
> > > > > When rolling restart/shutting down gracefully, the client will
> send a
> > > > > leave group request (static membership mode). In static membership,
> > we
> > > > will
> > > > > also define `change-group-timeout` to hold on rebalance provided by
> > > > leader.
> > > > > So we will wait for all the members to rejoin the group and do
> > exactly
> > > > one
> > > > > rebalance since all members are expected to rejoin within timeout.
> If
> > > > > consumer crashes, the join group request from the restarted
> consumer
> > > will
> > > > > be recognized as an existing member and be handled as above
> condition
> > > 1;
> > > > > However, if the consumer takes longer than session timeout to
> return,
> > > we
> > > > > shall still trigger rebalance but it could still try to catch
> > > > > `change-group-timeout`. If it failed to catch second timeout, its
> > > cached
> > > > > state on broker will be garbage collected and trigger a new
> rebalance
> > > > when
> > > > > it finally joins.
> > > > >
> > > > >
> > > > > And consider the switch between dynamic to static membership.
> > > > >
> > > > >   1.  Dynamic to static: the first joiner shall revise the
> membership
> > > to
> > > > > static and wait for all the current members to restart, since their
> > > > > membership is still dynamic. Here our assumption is that the
> restart
> > > > > process shouldn't take a long time, as long restart is breaking the
> > > > > `rebalance timeout` in whatever membership protocol we are using.
> > > Before
> > > > > restart, all dynamic member join requests will be rejected.
> > > > >   2.  Static to dynamic: this is more like a downgrade which should
> > be
> > > > > smooth: just erase the cached mapping, and wait for session timeout
> > to
> > > > > trigger rebalance should be sufficient. (Fallback to current
> > behavior)
> > > > >   3.  Halfway switch: a corner case is like some clients keep
> dynamic
> > > > > membership while some keep static membership. This will cause the
> > group
> > > > > rebalance forever without progress because dynamic/static states
> are
> > > > > bouncing each other. This could guarantee that we will not make the
> > > > > consumer group work in a wrong state by having half static and half
> > > > dynamic.
> > > > >
> > > > > To guarantee correctness, we will also push the member name/id pair
> > to
> > > > > _consumed_offsets topic (as Matthias pointed out) and upgrade the
> API
> > > > > version, these details will be further discussed back in the KIP.
> > > > >
> > > > >
> > > > > Are there any concern for this high level proposal? Just want to
> > > > reiterate
> > > > > on the core idea of the KIP: "If the broker recognize this consumer
> > as
> > > an
> > > > > existing member, it shouldn't trigger rebalance".
> > > > >
> > > > > Thanks a lot for everyone's input! I feel this proposal is much
> more
> > > > > robust than previous one!
> > > > >
> > > > >
> > > > > Best,
> > > > >
> > > > > Boyang
> > > > >
> > > > > 
> > > > > From: Matthias J. Sax 
> > > > > S

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Guozhang Wang]

ocol we are using.
> > Before
> > > > restart, all dynamic member join requests will be rejected.
> > > >   2.  Static to dynamic: this is more like a downgrade which should
> be
> > > > smooth: just erase the cached mapping, and wait for session timeout
> to
> > > > trigger rebalance should be sufficient. (Fallback to current
> behavior)
> > > >   3.  Halfway switch: a corner case is like some clients keep dynamic
> > > > membership while some keep static membership. This will cause the
> group
> > > > rebalance forever without progress because dynamic/static states are
> > > > bouncing each other. This could guarantee that we will not make the
> > > > consumer group work in a wrong state by having half static and half
> > > dynamic.
> > > >
> > > > To guarantee correctness, we will also push the member name/id pair
> to
> > > > _consumed_offsets topic (as Matthias pointed out) and upgrade the API
> > > > version, these details will be further discussed back in the KIP.
> > > >
> > > >
> > > > Are there any concern for this high level proposal? Just want to
> > > reiterate
> > > > on the core idea of the KIP: "If the broker recognize this consumer
> as
> > an
> > > > existing member, it shouldn't trigger rebalance".
> > > >
> > > > Thanks a lot for everyone's input! I feel this proposal is much more
> > > > robust than previous one!
> > > >
> > > >
> > > > Best,
> > > >
> > > > Boyang
> > > >
> > > > 
> > > > From: Matthias J. Sax 
> > > > Sent: Friday, August 10, 2018 2:24 AM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances
> by
> > > > specifying member id
> > > >
> > > > Hi,
> > > >
> > > > thanks for the detailed discussion. I learned a lot about internals
> > again
> > > > :)
> > > >
> > > > I like the idea or a user config `member.name` and to keep `
> member.id`
> > > > internal. Also agree with Guozhang, that reusing `client.id` might
> not
> > > > be a good idea.
> > > >
> > > > To clarify the algorithm, each time we generate a new `member.id`,
> we
> > > > also need to update the "group membership" information (ie, mapping
> > > > [member.id, Assignment]), right? Ie, the new `member.id` replaces
> the
> > > > old entry in the cache.
> > > >
> > > > I also think, we need to preserve the `member.name -> member.id`
> > mapping
> > > > in the `__consumer_offset` topic. The KIP should mention this IMHO.
> > > >
> > > > For changing the default value of config `leave.group.on.close`. I
> > agree
> > > > with John, that we should not change the default config, because it
> > > > would impact all consumer groups with dynamic assignment. However, I
> > > > think we can document, that if static assignment is used (ie,
> > > > `member.name` is configured) we never send a LeaveGroupRequest
> > > > regardless of the config. Note, that the config is internal, so not
> > sure
> > > > how to document this in detail. We should not expose the internal
> > config
> > > > in the docs.
> > > >
> > > > About upgrading: why do we need have two rolling bounces and encode
> > > > "static" vs "dynamic" in the JoinGroupRequest?
> > > >
> > > > If we upgrade an existing consumer group from dynamic to static, I
> > don't
> > > > see any reason why both should not work together and single rolling
> > > > bounce would not be sufficient? If we bounce the first consumer and
> > > > switch from dynamic to static, it sends a `member.name` and the
> broker
> > > > registers the [member.name, member.id] in the cache. Why would this
> > > > interfere with all other consumer that use dynamic assignment?
> > > >
> > > > Also, Guozhang mentioned that for all other request, we need to check
> > if
> > > > the mapping [member.name, member.id] contains the send `member.id`
> --
> > I
> > > > don't think this is necessary -- it seems to be sufficient to check
> the
> > > > `member.id` from the [member.id, Assignment] mapping as be do to

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Jason Gustafson]

The broker will check the membership to see whether this is a
> new
> > > member. If new, broker allocate a unique member id, cache the mapping
> and
> > > move to rebalance stage.
> > >   2.   Following 1, if this is an existing member, broker will not
> change
> > > group state, and return its cached member.id and current assignment.
> > > (unless this is leader, we shall trigger rebalance)
> > >   3.   Although Guozhang has mentioned we could rejoin with pair member
> > > name and id, I think for join group request it is ok to leave member id
> > > blank as member name is the unique identifier. In commit offset request
> > we
> > > *must* have both.
> > >
> > >
> > > When handling commit offset request, if enabled with static membership,
> > > each time the commit request must have both member.name and member.id
> to
> > > be identified as a `certificated member`. If not, this means there are
> > > duplicate consumer members with same member name and the request will
> be
> > > rejected to guarantee consumption uniqueness.
> > >
> > >
> > > When rolling restart/shutting down gracefully, the client will send a
> > > leave group request (static membership mode). In static membership, we
> > will
> > > also define `change-group-timeout` to hold on rebalance provided by
> > leader.
> > > So we will wait for all the members to rejoin the group and do exactly
> > one
> > > rebalance since all members are expected to rejoin within timeout. If
> > > consumer crashes, the join group request from the restarted consumer
> will
> > > be recognized as an existing member and be handled as above condition
> 1;
> > > However, if the consumer takes longer than session timeout to return,
> we
> > > shall still trigger rebalance but it could still try to catch
> > > `change-group-timeout`. If it failed to catch second timeout, its
> cached
> > > state on broker will be garbage collected and trigger a new rebalance
> > when
> > > it finally joins.
> > >
> > >
> > > And consider the switch between dynamic to static membership.
> > >
> > >   1.  Dynamic to static: the first joiner shall revise the membership
> to
> > > static and wait for all the current members to restart, since their
> > > membership is still dynamic. Here our assumption is that the restart
> > > process shouldn't take a long time, as long restart is breaking the
> > > `rebalance timeout` in whatever membership protocol we are using.
> Before
> > > restart, all dynamic member join requests will be rejected.
> > >   2.  Static to dynamic: this is more like a downgrade which should be
> > > smooth: just erase the cached mapping, and wait for session timeout to
> > > trigger rebalance should be sufficient. (Fallback to current behavior)
> > >   3.  Halfway switch: a corner case is like some clients keep dynamic
> > > membership while some keep static membership. This will cause the group
> > > rebalance forever without progress because dynamic/static states are
> > > bouncing each other. This could guarantee that we will not make the
> > > consumer group work in a wrong state by having half static and half
> > dynamic.
> > >
> > > To guarantee correctness, we will also push the member name/id pair to
> > > _consumed_offsets topic (as Matthias pointed out) and upgrade the API
> > > version, these details will be further discussed back in the KIP.
> > >
> > >
> > > Are there any concern for this high level proposal? Just want to
> > reiterate
> > > on the core idea of the KIP: "If the broker recognize this consumer as
> an
> > > existing member, it shouldn't trigger rebalance".
> > >
> > > Thanks a lot for everyone's input! I feel this proposal is much more
> > > robust than previous one!
> > >
> > >
> > > Best,
> > >
> > > Boyang
> > >
> > > 
> > > From: Matthias J. Sax 
> > > Sent: Friday, August 10, 2018 2:24 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> > > specifying member id
> > >
> > > Hi,
> > >
> > > thanks for the detailed discussion. I learned a lot about internals
> again
> > > :)
> > >
> > > I like the idea or a user config `member.name` and to keep `member.id`

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Guozhang Wang]

embership to
> > static and wait for all the current members to restart, since their
> > membership is still dynamic. Here our assumption is that the restart
> > process shouldn't take a long time, as long restart is breaking the
> > `rebalance timeout` in whatever membership protocol we are using. Before
> > restart, all dynamic member join requests will be rejected.
> >   2.  Static to dynamic: this is more like a downgrade which should be
> > smooth: just erase the cached mapping, and wait for session timeout to
> > trigger rebalance should be sufficient. (Fallback to current behavior)
> >   3.  Halfway switch: a corner case is like some clients keep dynamic
> > membership while some keep static membership. This will cause the group
> > rebalance forever without progress because dynamic/static states are
> > bouncing each other. This could guarantee that we will not make the
> > consumer group work in a wrong state by having half static and half
> dynamic.
> >
> > To guarantee correctness, we will also push the member name/id pair to
> > _consumed_offsets topic (as Matthias pointed out) and upgrade the API
> > version, these details will be further discussed back in the KIP.
> >
> >
> > Are there any concern for this high level proposal? Just want to
> reiterate
> > on the core idea of the KIP: "If the broker recognize this consumer as an
> > existing member, it shouldn't trigger rebalance".
> >
> > Thanks a lot for everyone's input! I feel this proposal is much more
> > robust than previous one!
> >
> >
> > Best,
> >
> > Boyang
> >
> > 
> > From: Matthias J. Sax 
> > Sent: Friday, August 10, 2018 2:24 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> > specifying member id
> >
> > Hi,
> >
> > thanks for the detailed discussion. I learned a lot about internals again
> > :)
> >
> > I like the idea or a user config `member.name` and to keep `member.id`
> > internal. Also agree with Guozhang, that reusing `client.id` might not
> > be a good idea.
> >
> > To clarify the algorithm, each time we generate a new `member.id`, we
> > also need to update the "group membership" information (ie, mapping
> > [member.id, Assignment]), right? Ie, the new `member.id` replaces the
> > old entry in the cache.
> >
> > I also think, we need to preserve the `member.name -> member.id` mapping
> > in the `__consumer_offset` topic. The KIP should mention this IMHO.
> >
> > For changing the default value of config `leave.group.on.close`. I agree
> > with John, that we should not change the default config, because it
> > would impact all consumer groups with dynamic assignment. However, I
> > think we can document, that if static assignment is used (ie,
> > `member.name` is configured) we never send a LeaveGroupRequest
> > regardless of the config. Note, that the config is internal, so not sure
> > how to document this in detail. We should not expose the internal config
> > in the docs.
> >
> > About upgrading: why do we need have two rolling bounces and encode
> > "static" vs "dynamic" in the JoinGroupRequest?
> >
> > If we upgrade an existing consumer group from dynamic to static, I don't
> > see any reason why both should not work together and single rolling
> > bounce would not be sufficient? If we bounce the first consumer and
> > switch from dynamic to static, it sends a `member.name` and the broker
> > registers the [member.name, member.id] in the cache. Why would this
> > interfere with all other consumer that use dynamic assignment?
> >
> > Also, Guozhang mentioned that for all other request, we need to check if
> > the mapping [member.name, member.id] contains the send `member.id` -- I
> > don't think this is necessary -- it seems to be sufficient to check the
> > `member.id` from the [member.id, Assignment] mapping as be do today --
> > thus, checking `member.id` does not require any change IMHO.
> >
> >
> > -Matthias
> >
> >
> > On 8/7/18 7:13 PM, Guozhang Wang wrote:
> > > @James
> > >
> > > What you described is true: the transition from dynamic to static
> > > memberships are not thought through yet. But I do not think it is an
> > > impossible problem: note that we indeed moved the offset commit from ZK
> > to
> > > kafka coordinator in 0.8.2 :) The migration plan is to first to
> > > double-commits on both zk 

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[John Roesler]

This sounds good to me!

Thanks for the time you've spent on it,
-John

On Tue, Aug 21, 2018 at 12:13 AM Boyang Chen  wrote:

> Thanks Matthias for the input. Sorry I was busy recently and haven't got
> time to update this thread. To summarize what we come up so far, here is a
> draft updated plan:
>
>
> Introduce a new config called `member.name` which is supposed to be
> provided uniquely by the consumer client. The broker will maintain a cache
> with [key:member.name, value:member.id]. A join group request with
> member.name set will be treated as `static-membership` strategy, and will
> reject any join group request without member.name. So this coordination
> change will be differentiated from the `dynamic-membership` protocol we
> currently have.
>
>
> When handling static join group request:
>
>   1.   The broker will check the membership to see whether this is a new
> member. If new, broker allocate a unique member id, cache the mapping and
> move to rebalance stage.
>   2.   Following 1, if this is an existing member, broker will not change
> group state, and return its cached member.id and current assignment.
> (unless this is leader, we shall trigger rebalance)
>   3.   Although Guozhang has mentioned we could rejoin with pair member
> name and id, I think for join group request it is ok to leave member id
> blank as member name is the unique identifier. In commit offset request we
> *must* have both.
>
>
> When handling commit offset request, if enabled with static membership,
> each time the commit request must have both member.name and member.id to
> be identified as a `certificated member`. If not, this means there are
> duplicate consumer members with same member name and the request will be
> rejected to guarantee consumption uniqueness.
>
>
> When rolling restart/shutting down gracefully, the client will send a
> leave group request (static membership mode). In static membership, we will
> also define `change-group-timeout` to hold on rebalance provided by leader.
> So we will wait for all the members to rejoin the group and do exactly one
> rebalance since all members are expected to rejoin within timeout. If
> consumer crashes, the join group request from the restarted consumer will
> be recognized as an existing member and be handled as above condition 1;
> However, if the consumer takes longer than session timeout to return, we
> shall still trigger rebalance but it could still try to catch
> `change-group-timeout`. If it failed to catch second timeout, its cached
> state on broker will be garbage collected and trigger a new rebalance when
> it finally joins.
>
>
> And consider the switch between dynamic to static membership.
>
>   1.  Dynamic to static: the first joiner shall revise the membership to
> static and wait for all the current members to restart, since their
> membership is still dynamic. Here our assumption is that the restart
> process shouldn't take a long time, as long restart is breaking the
> `rebalance timeout` in whatever membership protocol we are using. Before
> restart, all dynamic member join requests will be rejected.
>   2.  Static to dynamic: this is more like a downgrade which should be
> smooth: just erase the cached mapping, and wait for session timeout to
> trigger rebalance should be sufficient. (Fallback to current behavior)
>   3.  Halfway switch: a corner case is like some clients keep dynamic
> membership while some keep static membership. This will cause the group
> rebalance forever without progress because dynamic/static states are
> bouncing each other. This could guarantee that we will not make the
> consumer group work in a wrong state by having half static and half dynamic.
>
> To guarantee correctness, we will also push the member name/id pair to
> _consumed_offsets topic (as Matthias pointed out) and upgrade the API
> version, these details will be further discussed back in the KIP.
>
>
> Are there any concern for this high level proposal? Just want to reiterate
> on the core idea of the KIP: "If the broker recognize this consumer as an
> existing member, it shouldn't trigger rebalance".
>
> Thanks a lot for everyone's input! I feel this proposal is much more
> robust than previous one!
>
>
> Best,
>
> Boyang
>
> 
> From: Matthias J. Sax 
> Sent: Friday, August 10, 2018 2:24 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> Hi,
>
> thanks for the detailed discussion. I learned a lot about internals again
> :)
>
> I like the idea or a user config `member.name` and to keep `member.id`
> internal. Also agree with Guozhang, that reusing `

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

Thanks Matthias for the input. Sorry I was busy recently and haven't got time 
to update this thread. To summarize what we come up so far, here is a draft 
updated plan:


Introduce a new config called `member.name` which is supposed to be provided 
uniquely by the consumer client. The broker will maintain a cache with 
[key:member.name, value:member.id]. A join group request with member.name set 
will be treated as `static-membership` strategy, and will reject any join group 
request without member.name. So this coordination change will be differentiated 
from the `dynamic-membership` protocol we currently have.


When handling static join group request:

  1.   The broker will check the membership to see whether this is a new 
member. If new, broker allocate a unique member id, cache the mapping and move 
to rebalance stage.
  2.   Following 1, if this is an existing member, broker will not change group 
state, and return its cached member.id and current assignment. (unless this is 
leader, we shall trigger rebalance)
  3.   Although Guozhang has mentioned we could rejoin with pair member name 
and id, I think for join group request it is ok to leave member id blank as 
member name is the unique identifier. In commit offset request we *must* have 
both.


When handling commit offset request, if enabled with static membership, each 
time the commit request must have both member.name and member.id to be 
identified as a `certificated member`. If not, this means there are duplicate 
consumer members with same member name and the request will be rejected to 
guarantee consumption uniqueness.


When rolling restart/shutting down gracefully, the client will send a leave 
group request (static membership mode). In static membership, we will also 
define `change-group-timeout` to hold on rebalance provided by leader. So we 
will wait for all the members to rejoin the group and do exactly one rebalance 
since all members are expected to rejoin within timeout. If consumer crashes, 
the join group request from the restarted consumer will be recognized as an 
existing member and be handled as above condition 1; However, if the consumer 
takes longer than session timeout to return, we shall still trigger rebalance 
but it could still try to catch `change-group-timeout`. If it failed to catch 
second timeout, its cached state on broker will be garbage collected and 
trigger a new rebalance when it finally joins.


And consider the switch between dynamic to static membership.

  1.  Dynamic to static: the first joiner shall revise the membership to static 
and wait for all the current members to restart, since their membership is 
still dynamic. Here our assumption is that the restart process shouldn't take a 
long time, as long restart is breaking the `rebalance timeout` in whatever 
membership protocol we are using. Before restart, all dynamic member join 
requests will be rejected.
  2.  Static to dynamic: this is more like a downgrade which should be smooth: 
just erase the cached mapping, and wait for session timeout to trigger 
rebalance should be sufficient. (Fallback to current behavior)
  3.  Halfway switch: a corner case is like some clients keep dynamic 
membership while some keep static membership. This will cause the group 
rebalance forever without progress because dynamic/static states are bouncing 
each other. This could guarantee that we will not make the consumer group work 
in a wrong state by having half static and half dynamic.

To guarantee correctness, we will also push the member name/id pair to 
_consumed_offsets topic (as Matthias pointed out) and upgrade the API version, 
these details will be further discussed back in the KIP.


Are there any concern for this high level proposal? Just want to reiterate on 
the core idea of the KIP: "If the broker recognize this consumer as an existing 
member, it shouldn't trigger rebalance".

Thanks a lot for everyone's input! I feel this proposal is much more robust 
than previous one!


Best,

Boyang


From: Matthias J. Sax 
Sent: Friday, August 10, 2018 2:24 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hi,

thanks for the detailed discussion. I learned a lot about internals again :)

I like the idea or a user config `member.name` and to keep `member.id`
internal. Also agree with Guozhang, that reusing `client.id` might not
be a good idea.

To clarify the algorithm, each time we generate a new `member.id`, we
also need to update the "group membership" information (ie, mapping
[member.id, Assignment]), right? Ie, the new `member.id` replaces the
old entry in the cache.

I also think, we need to preserve the `member.name -> member.id` mapping
in the `__consumer_offset` topic. The KIP should mention this IMHO.

For changing the default value of config `leave.group.on.close`. I agree
with John, that we should not change the 

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Matthias J. Sax]

Hi,

thanks for the detailed discussion. I learned a lot about internals again :)

I like the idea or a user config `member.name` and to keep `member.id`
internal. Also agree with Guozhang, that reusing `client.id` might not
be a good idea.

To clarify the algorithm, each time we generate a new `member.id`, we
also need to update the "group membership" information (ie, mapping
[member.id, Assignment]), right? Ie, the new `member.id` replaces the
old entry in the cache.

I also think, we need to preserve the `member.name -> member.id` mapping
in the `__consumer_offset` topic. The KIP should mention this IMHO.

For changing the default value of config `leave.group.on.close`. I agree
with John, that we should not change the default config, because it
would impact all consumer groups with dynamic assignment. However, I
think we can document, that if static assignment is used (ie,
`member.name` is configured) we never send a LeaveGroupRequest
regardless of the config. Note, that the config is internal, so not sure
how to document this in detail. We should not expose the internal config
in the docs.

About upgrading: why do we need have two rolling bounces and encode
"static" vs "dynamic" in the JoinGroupRequest?

If we upgrade an existing consumer group from dynamic to static, I don't
see any reason why both should not work together and single rolling
bounce would not be sufficient? If we bounce the first consumer and
switch from dynamic to static, it sends a `member.name` and the broker
registers the [member.name, member.id] in the cache. Why would this
interfere with all other consumer that use dynamic assignment?

Also, Guozhang mentioned that for all other request, we need to check if
the mapping [member.name, member.id] contains the send `member.id` -- I
don't think this is necessary -- it seems to be sufficient to check the
`member.id` from the [member.id, Assignment] mapping as be do today --
thus, checking `member.id` does not require any change IMHO.


-Matthias


On 8/7/18 7:13 PM, Guozhang Wang wrote:
> @James
> 
> What you described is true: the transition from dynamic to static
> memberships are not thought through yet. But I do not think it is an
> impossible problem: note that we indeed moved the offset commit from ZK to
> kafka coordinator in 0.8.2 :) The migration plan is to first to
> double-commits on both zk and coordinator, and then do a second round to
> turn the zk off.
> 
> So just to throw a wild idea here: also following a two-rolling-bounce
> manner, in the JoinGroupRequest we can set the flag to "static" while keep
> the registry-id field empty still, in this case, the coordinator still
> follows the logic of "dynamic", accepting the request while allowing the
> protocol to be set to "static"; after the first rolling bounce, the group
> protocol is already "static", then a second rolling bounce is triggered and
> this time we set the registry-id.
> 
> 
> Guozhang
> 
> On Tue, Aug 7, 2018 at 1:19 AM, James Cheng  wrote:
> 
>> Guozhang, in a previous message, you proposed said this:
>>
>>> On Jul 30, 2018, at 3:56 PM, Guozhang Wang  wrote:
>>>
>>> 1. We bump up the JoinGroupRequest with additional fields:
>>>
>>>  1.a) a flag indicating "static" or "dynamic" membership protocols.
>>>  1.b) with "static" membership, we also add the pre-defined member id.
>>>  1.c) with "static" membership, we also add an optional
>>> "group-change-timeout" value.
>>>
>>> 2. On the broker side, we enforce only one of the two protocols for all
>>> group members: we accept the protocol on the first joined member of the
>>> group, and if later joining members indicate a different membership
>>> protocol, we reject it. If the group-change-timeout value was different
>> to
>>> the first joined member, we reject it as well.
>>
>>
>> What will happen if we have an already-deployed application that wants to
>> switch to using static membership? Let’s say there are 10 instances of it.
>> As the instances go through a rolling restart, they will switch from
>> dynamic membership (the default?) to static membership. As each one leaves
>> the group and restarts, they will be rejected from the group (because the
>> group is currently using dynamic membership). The group will shrink down
>> until there is 1 node handling all the traffic. After that one restarts,
>> the group will switch over to static membership.
>>
>> Is that right? That means that the transition plan from dynamic to static
>> membership isn’t very smooth.
>>
>> I’m not really sure what can be done in this case. This reminds me of the
>> transition plans that were discussed for moving from zookeeper-based
>> consumers to kafka-coordinator-based consumers. That was also hard, and
>> ultimately we decided not to build that.
>>
>> -James
>>
>>
> 
> 



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Guozhang Wang]

@James

What you described is true: the transition from dynamic to static
memberships are not thought through yet. But I do not think it is an
impossible problem: note that we indeed moved the offset commit from ZK to
kafka coordinator in 0.8.2 :) The migration plan is to first to
double-commits on both zk and coordinator, and then do a second round to
turn the zk off.

So just to throw a wild idea here: also following a two-rolling-bounce
manner, in the JoinGroupRequest we can set the flag to "static" while keep
the registry-id field empty still, in this case, the coordinator still
follows the logic of "dynamic", accepting the request while allowing the
protocol to be set to "static"; after the first rolling bounce, the group
protocol is already "static", then a second rolling bounce is triggered and
this time we set the registry-id.


Guozhang

On Tue, Aug 7, 2018 at 1:19 AM, James Cheng  wrote:

> Guozhang, in a previous message, you proposed said this:
>
> > On Jul 30, 2018, at 3:56 PM, Guozhang Wang  wrote:
> >
> > 1. We bump up the JoinGroupRequest with additional fields:
> >
> >  1.a) a flag indicating "static" or "dynamic" membership protocols.
> >  1.b) with "static" membership, we also add the pre-defined member id.
> >  1.c) with "static" membership, we also add an optional
> > "group-change-timeout" value.
> >
> > 2. On the broker side, we enforce only one of the two protocols for all
> > group members: we accept the protocol on the first joined member of the
> > group, and if later joining members indicate a different membership
> > protocol, we reject it. If the group-change-timeout value was different
> to
> > the first joined member, we reject it as well.
>
>
> What will happen if we have an already-deployed application that wants to
> switch to using static membership? Let’s say there are 10 instances of it.
> As the instances go through a rolling restart, they will switch from
> dynamic membership (the default?) to static membership. As each one leaves
> the group and restarts, they will be rejected from the group (because the
> group is currently using dynamic membership). The group will shrink down
> until there is 1 node handling all the traffic. After that one restarts,
> the group will switch over to static membership.
>
> Is that right? That means that the transition plan from dynamic to static
> membership isn’t very smooth.
>
> I’m not really sure what can be done in this case. This reminds me of the
> transition plans that were discussed for moving from zookeeper-based
> consumers to kafka-coordinator-based consumers. That was also hard, and
> ultimately we decided not to build that.
>
> -James
>
>


-- 
-- Guozhang

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[James Cheng]

Guozhang, in a previous message, you proposed said this:

> On Jul 30, 2018, at 3:56 PM, Guozhang Wang  wrote:
> 
> 1. We bump up the JoinGroupRequest with additional fields:
> 
>  1.a) a flag indicating "static" or "dynamic" membership protocols.
>  1.b) with "static" membership, we also add the pre-defined member id.
>  1.c) with "static" membership, we also add an optional
> "group-change-timeout" value.
> 
> 2. On the broker side, we enforce only one of the two protocols for all
> group members: we accept the protocol on the first joined member of the
> group, and if later joining members indicate a different membership
> protocol, we reject it. If the group-change-timeout value was different to
> the first joined member, we reject it as well.


What will happen if we have an already-deployed application that wants to 
switch to using static membership? Let’s say there are 10 instances of it. As 
the instances go through a rolling restart, they will switch from dynamic 
membership (the default?) to static membership. As each one leaves the group 
and restarts, they will be rejected from the group (because the group is 
currently using dynamic membership). The group will shrink down until there is 
1 node handling all the traffic. After that one restarts, the group will switch 
over to static membership.

Is that right? That means that the transition plan from dynamic to static 
membership isn’t very smooth. 

I’m not really sure what can be done in this case. This reminds me of the 
transition plans that were discussed for moving from zookeeper-based consumers 
to kafka-coordinator-based consumers. That was also hard, and ultimately we 
decided not to build that.

-James

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

Sounds good to me Guozhang. I feel clear and confident about this enhanced 
design.


Kindly asking everyone's opinion on this thread for the updated approach. If we 
are ok here, I will go ahead to update the KIP and plan out the implementation.


Best,

Boyang


From: Guozhang Wang 
Sent: Monday, August 6, 2018 8:46 AM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Re: reuse of "client.id": good question. Although client.id could be unique
to guarantee fencing, by using it we are binding the user's behavior of
specifically setting client id to the indication of using static
membership. There are, however, cases where users want to specify client id
but NOT want to use static membership protocols we proposed here. This was
also why in transactional messaging we chose to not reuse client.id as
transactional.id since the latter also indicate that "I want to turn on
transactional messaging". If we agree that here setting this "id" has
similar semantics of enabling certain additional feature, then we should
not piggy-back that on the existing client.id.


Guozhang

On Sat, Aug 4, 2018 at 10:13 PM, Boyang Chen  wrote:

> Hey Guozhang,
>
>
> I think this makes sense. I'm wondering whether we could reuse the
> consumer client id as unique identifier instead of introducing new terms.
> From my experience, they should be unique in either Stream or Consumer use
> case.
>
>
> Thanks a lot for the clarification!
>
> 
> From: Guozhang Wang 
> Sent: Saturday, August 4, 2018 5:25 AM
> To: dev
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> @Boyang,
>
> * Yes registry id should be generated via a config, similar to what you
> proposed in the KIP currently for member id. And "even if multiple
> consumers are using the same member id" (I think you meant registry id
> here?), we can fence against them.
>
> BTW I'm not married (but cannot speak for Jason :P ) to "registry id", so
> any other naming suggestions are welcome. John's suggestion of `member
> name` sounds okay to me. Another wild idea I had is just name it "consumer
> id".
>
> * About the metrics: note that during a consumer bouncing case, it will
> resume and come back with an empty member id and a pre-existed registry id.
> On the coordinator case, it is considered a "fenced" event, but it is
> really not an event that should be alerted. So I think we should not simply
> record a metric whenever the coordinator updates the registryId -> memberId
> map.
>
> We could, however, consider adding a metric for the other event: whenever a
> coordinator is rejecting a request because of incorrect member id. Note
> this event covers both a static membership related fencing event, plus
> today's normal cases where members are kicked out of the group. During a
> rebalance, we may see some spikes of this metrics but then during normal
> operations, this metric should stay as 0; but when we see this metric stays
> as non-zero consistently, it means there are some zombies lurking around.
>
> * About the code base: GroupCoordinator represents the broker-side
> coordinator logic, and ConsumerCoordinator is on the client side,
> representing the related logic for consumer to communicate with the
> broker-side coordinator. When we talk about the "coordinator" in this
> discussion, unless clearly specified, it should always be to the
> broker-side logic, i.e. GroupCoordinator.
>
>
> @John
>
> Note that a consumer can only know that it is already fenced when talking
> to the coordinator (again). More specifically, once a consumer gets
> assignment, it will keep fetching from the brokers without talking to
> coordinator until the next heartbeat request. Before that time, it is
> un-avoidable that two consumers may be fetching from the same partitions
> now, I used commit request just as an example of any requests sent to the
> coordinator, not saying that they will only be aware until then. In fact,
> when they sent Heartbeat request to the coordinator, they could also be
> notified that "you are fenced".
>
>
> Guozhang
>
>
> On Wed, Aug 1, 2018 at 10:41 PM, Boyang Chen  wrote:
>
> > Hey Guozhang,
> >
> >
> > thanks for this detailed proposal! Quickly I want to clarify, where this
> > registry id is generated? My understanding is that the registry id is a
> > unique id provided by user correct? This way even if multiple consumers
> are
> > using the same member id, we are fencing against polluting the commit
> > history. Very thoughtful approach! Furthermore, I

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Guozhang Wang]

Re: reuse of "client.id": good question. Although client.id could be unique
to guarantee fencing, by using it we are binding the user's behavior of
specifically setting client id to the indication of using static
membership. There are, however, cases where users want to specify client id
but NOT want to use static membership protocols we proposed here. This was
also why in transactional messaging we chose to not reuse client.id as
transactional.id since the latter also indicate that "I want to turn on
transactional messaging". If we agree that here setting this "id" has
similar semantics of enabling certain additional feature, then we should
not piggy-back that on the existing client.id.


Guozhang

On Sat, Aug 4, 2018 at 10:13 PM, Boyang Chen  wrote:

> Hey Guozhang,
>
>
> I think this makes sense. I'm wondering whether we could reuse the
> consumer client id as unique identifier instead of introducing new terms.
> From my experience, they should be unique in either Stream or Consumer use
> case.
>
>
> Thanks a lot for the clarification!
>
> 
> From: Guozhang Wang 
> Sent: Saturday, August 4, 2018 5:25 AM
> To: dev
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> @Boyang,
>
> * Yes registry id should be generated via a config, similar to what you
> proposed in the KIP currently for member id. And "even if multiple
> consumers are using the same member id" (I think you meant registry id
> here?), we can fence against them.
>
> BTW I'm not married (but cannot speak for Jason :P ) to "registry id", so
> any other naming suggestions are welcome. John's suggestion of `member
> name` sounds okay to me. Another wild idea I had is just name it "consumer
> id".
>
> * About the metrics: note that during a consumer bouncing case, it will
> resume and come back with an empty member id and a pre-existed registry id.
> On the coordinator case, it is considered a "fenced" event, but it is
> really not an event that should be alerted. So I think we should not simply
> record a metric whenever the coordinator updates the registryId -> memberId
> map.
>
> We could, however, consider adding a metric for the other event: whenever a
> coordinator is rejecting a request because of incorrect member id. Note
> this event covers both a static membership related fencing event, plus
> today's normal cases where members are kicked out of the group. During a
> rebalance, we may see some spikes of this metrics but then during normal
> operations, this metric should stay as 0; but when we see this metric stays
> as non-zero consistently, it means there are some zombies lurking around.
>
> * About the code base: GroupCoordinator represents the broker-side
> coordinator logic, and ConsumerCoordinator is on the client side,
> representing the related logic for consumer to communicate with the
> broker-side coordinator. When we talk about the "coordinator" in this
> discussion, unless clearly specified, it should always be to the
> broker-side logic, i.e. GroupCoordinator.
>
>
> @John
>
> Note that a consumer can only know that it is already fenced when talking
> to the coordinator (again). More specifically, once a consumer gets
> assignment, it will keep fetching from the brokers without talking to
> coordinator until the next heartbeat request. Before that time, it is
> un-avoidable that two consumers may be fetching from the same partitions
> now, I used commit request just as an example of any requests sent to the
> coordinator, not saying that they will only be aware until then. In fact,
> when they sent Heartbeat request to the coordinator, they could also be
> notified that "you are fenced".
>
>
> Guozhang
>
>
> On Wed, Aug 1, 2018 at 10:41 PM, Boyang Chen  wrote:
>
> > Hey Guozhang,
> >
> >
> > thanks for this detailed proposal! Quickly I want to clarify, where this
> > registry id is generated? My understanding is that the registry id is a
> > unique id provided by user correct? This way even if multiple consumers
> are
> > using the same member id, we are fencing against polluting the commit
> > history. Very thoughtful approach! Furthermore, I feel we should export
> > alerting metrics on catching duplicate consumer instances since multiple
> of
> > them could still be reading the same topic partition and executing
> > duplicate business logic.
> >
> >
> > Also a minor comment is that our current codebase has two different
> > classes: the GroupCoordinator and ConsumerCoordinator. When talking about
> > term "coordinator", it would

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

Hey Guozhang,


I think this makes sense. I'm wondering whether we could reuse the consumer 
client id as unique identifier instead of introducing new terms. From my 
experience, they should be unique in either Stream or Consumer use case.


Thanks a lot for the clarification!


From: Guozhang Wang 
Sent: Saturday, August 4, 2018 5:25 AM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

@Boyang,

* Yes registry id should be generated via a config, similar to what you
proposed in the KIP currently for member id. And "even if multiple
consumers are using the same member id" (I think you meant registry id
here?), we can fence against them.

BTW I'm not married (but cannot speak for Jason :P ) to "registry id", so
any other naming suggestions are welcome. John's suggestion of `member
name` sounds okay to me. Another wild idea I had is just name it "consumer
id".

* About the metrics: note that during a consumer bouncing case, it will
resume and come back with an empty member id and a pre-existed registry id.
On the coordinator case, it is considered a "fenced" event, but it is
really not an event that should be alerted. So I think we should not simply
record a metric whenever the coordinator updates the registryId -> memberId
map.

We could, however, consider adding a metric for the other event: whenever a
coordinator is rejecting a request because of incorrect member id. Note
this event covers both a static membership related fencing event, plus
today's normal cases where members are kicked out of the group. During a
rebalance, we may see some spikes of this metrics but then during normal
operations, this metric should stay as 0; but when we see this metric stays
as non-zero consistently, it means there are some zombies lurking around.

* About the code base: GroupCoordinator represents the broker-side
coordinator logic, and ConsumerCoordinator is on the client side,
representing the related logic for consumer to communicate with the
broker-side coordinator. When we talk about the "coordinator" in this
discussion, unless clearly specified, it should always be to the
broker-side logic, i.e. GroupCoordinator.


@John

Note that a consumer can only know that it is already fenced when talking
to the coordinator (again). More specifically, once a consumer gets
assignment, it will keep fetching from the brokers without talking to
coordinator until the next heartbeat request. Before that time, it is
un-avoidable that two consumers may be fetching from the same partitions
now, I used commit request just as an example of any requests sent to the
coordinator, not saying that they will only be aware until then. In fact,
when they sent Heartbeat request to the coordinator, they could also be
notified that "you are fenced".


Guozhang


On Wed, Aug 1, 2018 at 10:41 PM, Boyang Chen  wrote:

> Hey Guozhang,
>
>
> thanks for this detailed proposal! Quickly I want to clarify, where this
> registry id is generated? My understanding is that the registry id is a
> unique id provided by user correct? This way even if multiple consumers are
> using the same member id, we are fencing against polluting the commit
> history. Very thoughtful approach! Furthermore, I feel we should export
> alerting metrics on catching duplicate consumer instances since multiple of
> them could still be reading the same topic partition and executing
> duplicate business logic.
>
>
> Also a minor comment is that our current codebase has two different
> classes: the GroupCoordinator and ConsumerCoordinator. When talking about
> term "coordinator", it would be great to distinguish between the two, or
> explicitly nominate broker coordinator as default "coordinator".
>
>
> Best,
>
> Boyang
>
> ________
> From: Guozhang Wang 
> Sent: Thursday, August 2, 2018 1:59 AM
> To: dev
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> Hello Boyang,
>
> Thanks for the great summary.
>
> Personally I think 1) is still doable as we are not necessarily have to
> rely on source of truth to always fence "the right zombie"; in fact, my
> point is that as long as we can fence "someone" when there are zombies
> sharing the same member id so that users are notified of such issues and
> then react to it, it is good enough. Letting first or last comer wins falls
> in that consideration.
>
> I've thought about Jason's "registry id" approach, and I think it may
> actually be suitable for providing fencing while approach the goal of avoid
> unnecessary rebalances. Here are some more concrete details (@Jason, if it
> is not the same as what you had originally in mind, let

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Guozhang Wang]

@Boyang,

* Yes registry id should be generated via a config, similar to what you
proposed in the KIP currently for member id. And "even if multiple
consumers are using the same member id" (I think you meant registry id
here?), we can fence against them.

BTW I'm not married (but cannot speak for Jason :P ) to "registry id", so
any other naming suggestions are welcome. John's suggestion of `member
name` sounds okay to me. Another wild idea I had is just name it "consumer
id".

* About the metrics: note that during a consumer bouncing case, it will
resume and come back with an empty member id and a pre-existed registry id.
On the coordinator case, it is considered a "fenced" event, but it is
really not an event that should be alerted. So I think we should not simply
record a metric whenever the coordinator updates the registryId -> memberId
map.

We could, however, consider adding a metric for the other event: whenever a
coordinator is rejecting a request because of incorrect member id. Note
this event covers both a static membership related fencing event, plus
today's normal cases where members are kicked out of the group. During a
rebalance, we may see some spikes of this metrics but then during normal
operations, this metric should stay as 0; but when we see this metric stays
as non-zero consistently, it means there are some zombies lurking around.

* About the code base: GroupCoordinator represents the broker-side
coordinator logic, and ConsumerCoordinator is on the client side,
representing the related logic for consumer to communicate with the
broker-side coordinator. When we talk about the "coordinator" in this
discussion, unless clearly specified, it should always be to the
broker-side logic, i.e. GroupCoordinator.


@John

Note that a consumer can only know that it is already fenced when talking
to the coordinator (again). More specifically, once a consumer gets
assignment, it will keep fetching from the brokers without talking to
coordinator until the next heartbeat request. Before that time, it is
un-avoidable that two consumers may be fetching from the same partitions
now, I used commit request just as an example of any requests sent to the
coordinator, not saying that they will only be aware until then. In fact,
when they sent Heartbeat request to the coordinator, they could also be
notified that "you are fenced".


Guozhang


On Wed, Aug 1, 2018 at 10:41 PM, Boyang Chen  wrote:

> Hey Guozhang,
>
>
> thanks for this detailed proposal! Quickly I want to clarify, where this
> registry id is generated? My understanding is that the registry id is a
> unique id provided by user correct? This way even if multiple consumers are
> using the same member id, we are fencing against polluting the commit
> history. Very thoughtful approach! Furthermore, I feel we should export
> alerting metrics on catching duplicate consumer instances since multiple of
> them could still be reading the same topic partition and executing
> duplicate business logic.
>
>
> Also a minor comment is that our current codebase has two different
> classes: the GroupCoordinator and ConsumerCoordinator. When talking about
> term "coordinator", it would be great to distinguish between the two, or
> explicitly nominate broker coordinator as default "coordinator".
>
>
> Best,
>
> Boyang
>
> ________
> From: Guozhang Wang 
> Sent: Thursday, August 2, 2018 1:59 AM
> To: dev
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> Hello Boyang,
>
> Thanks for the great summary.
>
> Personally I think 1) is still doable as we are not necessarily have to
> rely on source of truth to always fence "the right zombie"; in fact, my
> point is that as long as we can fence "someone" when there are zombies
> sharing the same member id so that users are notified of such issues and
> then react to it, it is good enough. Letting first or last comer wins falls
> in that consideration.
>
> I've thought about Jason's "registry id" approach, and I think it may
> actually be suitable for providing fencing while approach the goal of avoid
> unnecessary rebalances. Here are some more concrete details (@Jason, if it
> is not the same as what you had originally in mind, let me know):
>
>
> a. In JoinGroupRequest, include a "registry id" in addition to the "member
> id" to the coordinator. In all other requests like CommitOffset, etc, still
> only include the "member id".
>
> b. On the coordinator side, when receiving a JoinGroupRequest with a
> registry id with empty member id, and this registry id is indeed in the
> cache, it still generates a new member id and return it in the res

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[John Roesler]

Hi Boyang,

It seems like there's already some confusion about the roles of registry id
vs member id.

My understanding is that the protocol is currently that the broker assigns
a unique "member id" to each consumer in a group, and those consumers must
include this id in each subsequent request. It seems like the latest
proposal is actually not to change this at all.
It sounds like the latest idea is to add a "registry id", which is what you
originally proposed to replace member id. That is, it's provided in config
and it's a stable identifier that semantically identifies each member of
the group, which persists across restarts, etc.

Maybe "member name" would be a more self-explanatory name for this property?

It would be a good idea to have metrics tracking fence events etc, for
operational visibility. It does seem like when the broker fences a consumer
it should reply with a "you've been fenced" message, and the consumer
should stop executing. Your last message sounded like the duplicate
consumers would keep executing.

I hope this helps,
-John

On Fri, Aug 3, 2018 at 1:56 PM Boyang Chen  wrote:

> Hey Guozhang, do you think my understanding makes sense? I would also like
> to hear from Jason, Mike, John and everyone's opinion on this new approach.
>
>
> Thanks!
>
> 
> From: Boyang Chen 
> Sent: Thursday, August 2, 2018 1:41 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> Hey Guozhang,
>
>
> thanks for this detailed proposal! Quickly I want to clarify, where this
> registry id is generated? My understanding is that the registry id is a
> unique id provided by user correct? This way even if multiple consumers are
> using the same member id, we are fencing against polluting the commit
> history. Very thoughtful approach! Furthermore, I feel we should export
> alerting metrics on catching duplicate consumer instances since multiple of
> them could still be reading the same topic partition and executing
> duplicate business logic.
>
>
> Also a minor comment is that our current codebase has two different
> classes: the GroupCoordinator and ConsumerCoordinator. When talking about
> term "coordinator", it would be great to distinguish between the two, or
> explicitly nominate broker coordinator as default "coordinator".
>
>
> Best,
>
> Boyang
>
> ____________
> From: Guozhang Wang 
> Sent: Thursday, August 2, 2018 1:59 AM
> To: dev
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> Hello Boyang,
>
> Thanks for the great summary.
>
> Personally I think 1) is still doable as we are not necessarily have to
> rely on source of truth to always fence "the right zombie"; in fact, my
> point is that as long as we can fence "someone" when there are zombies
> sharing the same member id so that users are notified of such issues and
> then react to it, it is good enough. Letting first or last comer wins falls
> in that consideration.
>
> I've thought about Jason's "registry id" approach, and I think it may
> actually be suitable for providing fencing while approach the goal of avoid
> unnecessary rebalances. Here are some more concrete details (@Jason, if it
> is not the same as what you had originally in mind, let me know):
>
>
> a. In JoinGroupRequest, include a "registry id" in addition to the "member
> id" to the coordinator. In all other requests like CommitOffset, etc, still
> only include the "member id".
>
> b. On the coordinator side, when receiving a JoinGroupRequest with a
> registry id with empty member id, and this registry id is indeed in the
> cache, it still generates a new member id and return it in the response;
> additionally, and update the [registry-id, member-id] mapping in its member
> metadata cache (i.e. blind-write to the cache regardless whether it already
> contains this registry id); if both registry-id and member-id are
> specified, check if it matches in the cache, if not or if the cache does
> not include this mapping at all, return an ILLEGAL_MEMBER_ID error to let
> the client reset its member id and reset again.
>
> ** NOTE ** this logic is actually "letting the last comer win", i.e.
> whenever a new client comes with a registry id and empty member id, always
> treat it as a valid consumer and replace with whatever existed with this
> registry id, and the previous member id will be effectively removed from
> the group.
>
> c. On the coordinator side, when receiving any other group requests like
> CommitOffset, make sure the member 

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

Hey Guozhang, do you think my understanding makes sense? I would also like to 
hear from Jason, Mike, John and everyone's opinion on this new approach.


Thanks!


From: Boyang Chen 
Sent: Thursday, August 2, 2018 1:41 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hey Guozhang,


thanks for this detailed proposal! Quickly I want to clarify, where this 
registry id is generated? My understanding is that the registry id is a unique 
id provided by user correct? This way even if multiple consumers are using the 
same member id, we are fencing against polluting the commit history. Very 
thoughtful approach! Furthermore, I feel we should export alerting metrics on 
catching duplicate consumer instances since multiple of them could still be 
reading the same topic partition and executing duplicate business logic.


Also a minor comment is that our current codebase has two different classes: 
the GroupCoordinator and ConsumerCoordinator. When talking about term 
"coordinator", it would be great to distinguish between the two, or explicitly 
nominate broker coordinator as default "coordinator".


Best,

Boyang


From: Guozhang Wang 
Sent: Thursday, August 2, 2018 1:59 AM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hello Boyang,

Thanks for the great summary.

Personally I think 1) is still doable as we are not necessarily have to
rely on source of truth to always fence "the right zombie"; in fact, my
point is that as long as we can fence "someone" when there are zombies
sharing the same member id so that users are notified of such issues and
then react to it, it is good enough. Letting first or last comer wins falls
in that consideration.

I've thought about Jason's "registry id" approach, and I think it may
actually be suitable for providing fencing while approach the goal of avoid
unnecessary rebalances. Here are some more concrete details (@Jason, if it
is not the same as what you had originally in mind, let me know):


a. In JoinGroupRequest, include a "registry id" in addition to the "member
id" to the coordinator. In all other requests like CommitOffset, etc, still
only include the "member id".

b. On the coordinator side, when receiving a JoinGroupRequest with a
registry id with empty member id, and this registry id is indeed in the
cache, it still generates a new member id and return it in the response;
additionally, and update the [registry-id, member-id] mapping in its member
metadata cache (i.e. blind-write to the cache regardless whether it already
contains this registry id); if both registry-id and member-id are
specified, check if it matches in the cache, if not or if the cache does
not include this mapping at all, return an ILLEGAL_MEMBER_ID error to let
the client reset its member id and reset again.

** NOTE ** this logic is actually "letting the last comer win", i.e.
whenever a new client comes with a registry id and empty member id, always
treat it as a valid consumer and replace with whatever existed with this
registry id, and the previous member id will be effectively removed from
the group.

c. On the coordinator side, when receiving any other group requests like
CommitOffset, make sure the member id is indeed in this cache (i.e. it is
contained in the [registry-id. member-id] mapping), otherwise reject it as
we do today.


--

So to give an example, if clientA joins group with registry id 1, it will
be assigned a member id X and coordinator remembers current mapping [1, X]
and "X" is the valid member belonging to this group; if another clientB
joins group but with the same registry id 1 with empty member id,
coordinator assigns it with a new member id Y and update the mapping [1,
Y]. Hence the coordinator always blindly trust the new comer as the valid
member and kicks out the previous one. At this time both clientA and
clientB may be fetching from the same assigned partitions, BUT when clientA
tries to commit the offset, by sending with member id X, it will be
rejected.

--

And then back to the original goal of 5/6, when a consumer shuts down and
restarts, it will send a JoinGroupRequest with the registry id with empty
member id, as long as coordinator still remembers this registry id, it will
sends back the original assigned partitions to the consumer, while updating
its cache with the newly generated member id. No rebalance will be
triggered, and if there are other zombies with the same registry ids and
already get the assignment before, although they can still fetch data, they
will not be able to commit.

--



Guozhang



On Wed, Aug 1, 2018 at 8:07 AM, Boyang Chen  wrote:

> Thanks for everyone's (Guozhang, Jason, Mike, John and so

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

Hey Guozhang,


thanks for this detailed proposal! Quickly I want to clarify, where this 
registry id is generated? My understanding is that the registry id is a unique 
id provided by user correct? This way even if multiple consumers are using the 
same member id, we are fencing against polluting the commit history. Very 
thoughtful approach! Furthermore, I feel we should export alerting metrics on 
catching duplicate consumer instances since multiple of them could still be 
reading the same topic partition and executing duplicate business logic.


Also a minor comment is that our current codebase has two different classes: 
the GroupCoordinator and ConsumerCoordinator. When talking about term 
"coordinator", it would be great to distinguish between the two, or explicitly 
nominate broker coordinator as default "coordinator".


Best,

Boyang


From: Guozhang Wang 
Sent: Thursday, August 2, 2018 1:59 AM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hello Boyang,

Thanks for the great summary.

Personally I think 1) is still doable as we are not necessarily have to
rely on source of truth to always fence "the right zombie"; in fact, my
point is that as long as we can fence "someone" when there are zombies
sharing the same member id so that users are notified of such issues and
then react to it, it is good enough. Letting first or last comer wins falls
in that consideration.

I've thought about Jason's "registry id" approach, and I think it may
actually be suitable for providing fencing while approach the goal of avoid
unnecessary rebalances. Here are some more concrete details (@Jason, if it
is not the same as what you had originally in mind, let me know):


a. In JoinGroupRequest, include a "registry id" in addition to the "member
id" to the coordinator. In all other requests like CommitOffset, etc, still
only include the "member id".

b. On the coordinator side, when receiving a JoinGroupRequest with a
registry id with empty member id, and this registry id is indeed in the
cache, it still generates a new member id and return it in the response;
additionally, and update the [registry-id, member-id] mapping in its member
metadata cache (i.e. blind-write to the cache regardless whether it already
contains this registry id); if both registry-id and member-id are
specified, check if it matches in the cache, if not or if the cache does
not include this mapping at all, return an ILLEGAL_MEMBER_ID error to let
the client reset its member id and reset again.

** NOTE ** this logic is actually "letting the last comer win", i.e.
whenever a new client comes with a registry id and empty member id, always
treat it as a valid consumer and replace with whatever existed with this
registry id, and the previous member id will be effectively removed from
the group.

c. On the coordinator side, when receiving any other group requests like
CommitOffset, make sure the member id is indeed in this cache (i.e. it is
contained in the [registry-id. member-id] mapping), otherwise reject it as
we do today.


--

So to give an example, if clientA joins group with registry id 1, it will
be assigned a member id X and coordinator remembers current mapping [1, X]
and "X" is the valid member belonging to this group; if another clientB
joins group but with the same registry id 1 with empty member id,
coordinator assigns it with a new member id Y and update the mapping [1,
Y]. Hence the coordinator always blindly trust the new comer as the valid
member and kicks out the previous one. At this time both clientA and
clientB may be fetching from the same assigned partitions, BUT when clientA
tries to commit the offset, by sending with member id X, it will be
rejected.

--

And then back to the original goal of 5/6, when a consumer shuts down and
restarts, it will send a JoinGroupRequest with the registry id with empty
member id, as long as coordinator still remembers this registry id, it will
sends back the original assigned partitions to the consumer, while updating
its cache with the newly generated member id. No rebalance will be
triggered, and if there are other zombies with the same registry ids and
already get the assignment before, although they can still fetch data, they
will not be able to commit.

--



Guozhang



On Wed, Aug 1, 2018 at 8:07 AM, Boyang Chen  wrote:

> Thanks for everyone's (Guozhang, Jason, Mike, John and someone in case I'm
> missing) input here! Looks like we have got more interesting thoughts on
> this KIP.
>
>
> When I draft KIP-345, the original proposal is targeted at solving "static
> membership", which is trying to address 1/2/5/6  proposed by Guozhang. In
> my mindset, the most important detail I'm trying to follow is the owner of
> me

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Guozhang Wang]

Hello Boyang,

Thanks for the great summary.

Personally I think 1) is still doable as we are not necessarily have to
rely on source of truth to always fence "the right zombie"; in fact, my
point is that as long as we can fence "someone" when there are zombies
sharing the same member id so that users are notified of such issues and
then react to it, it is good enough. Letting first or last comer wins falls
in that consideration.

I've thought about Jason's "registry id" approach, and I think it may
actually be suitable for providing fencing while approach the goal of avoid
unnecessary rebalances. Here are some more concrete details (@Jason, if it
is not the same as what you had originally in mind, let me know):


a. In JoinGroupRequest, include a "registry id" in addition to the "member
id" to the coordinator. In all other requests like CommitOffset, etc, still
only include the "member id".

b. On the coordinator side, when receiving a JoinGroupRequest with a
registry id with empty member id, and this registry id is indeed in the
cache, it still generates a new member id and return it in the response;
additionally, and update the [registry-id, member-id] mapping in its member
metadata cache (i.e. blind-write to the cache regardless whether it already
contains this registry id); if both registry-id and member-id are
specified, check if it matches in the cache, if not or if the cache does
not include this mapping at all, return an ILLEGAL_MEMBER_ID error to let
the client reset its member id and reset again.

** NOTE ** this logic is actually "letting the last comer win", i.e.
whenever a new client comes with a registry id and empty member id, always
treat it as a valid consumer and replace with whatever existed with this
registry id, and the previous member id will be effectively removed from
the group.

c. On the coordinator side, when receiving any other group requests like
CommitOffset, make sure the member id is indeed in this cache (i.e. it is
contained in the [registry-id. member-id] mapping), otherwise reject it as
we do today.


--

So to give an example, if clientA joins group with registry id 1, it will
be assigned a member id X and coordinator remembers current mapping [1, X]
and "X" is the valid member belonging to this group; if another clientB
joins group but with the same registry id 1 with empty member id,
coordinator assigns it with a new member id Y and update the mapping [1,
Y]. Hence the coordinator always blindly trust the new comer as the valid
member and kicks out the previous one. At this time both clientA and
clientB may be fetching from the same assigned partitions, BUT when clientA
tries to commit the offset, by sending with member id X, it will be
rejected.

--

And then back to the original goal of 5/6, when a consumer shuts down and
restarts, it will send a JoinGroupRequest with the registry id with empty
member id, as long as coordinator still remembers this registry id, it will
sends back the original assigned partitions to the consumer, while updating
its cache with the newly generated member id. No rebalance will be
triggered, and if there are other zombies with the same registry ids and
already get the assignment before, although they can still fetch data, they
will not be able to commit.

--



Guozhang



On Wed, Aug 1, 2018 at 8:07 AM, Boyang Chen  wrote:

> Thanks for everyone's (Guozhang, Jason, Mike, John and someone in case I'm
> missing) input here! Looks like we have got more interesting thoughts on
> this KIP.
>
>
> When I draft KIP-345, the original proposal is targeted at solving "static
> membership", which is trying to address 1/2/5/6  proposed by Guozhang. In
> my mindset, the most important detail I'm trying to follow is the owner of
> member id generation or the membership metadata (proposed by John).
>
>
> Guozhang's static membership approach overall LGTM. It would make the
> whole protocol much clear with group member come and go explicitly in a
> static setting. The static approach appoints Consumer/Stream app as the
> owner of member id generation, which brought up multiple validation
> concerns. As the host:port approach is vetoed, and I have thought for a
> while for other validation strategies but failed, I think it's time to
> decide whether we want to focus our next step discussion on
>
>
>   1.  how to let server validate id uniqueness generated by client, or
>   2.  how to let client materialize the id provided by the server
>
>
> Let me know your favor of the two topics here. Personally I'm inclined to
> 2 as 1 is intrinsically complex (no source of truth model). Hope we are on
> the same page now.
>
>
> Best,
>
> Boyang
>
> ____________
> From: John Roesler 
> Sent: Wednesday

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

Thanks for everyone's (Guozhang, Jason, Mike, John and someone in case I'm 
missing) input here! Looks like we have got more interesting thoughts on this 
KIP.


When I draft KIP-345, the original proposal is targeted at solving "static 
membership", which is trying to address 1/2/5/6  proposed by Guozhang. In my 
mindset, the most important detail I'm trying to follow is the owner of member 
id generation or the membership metadata (proposed by John).


Guozhang's static membership approach overall LGTM. It would make the whole 
protocol much clear with group member come and go explicitly in a static 
setting. The static approach appoints Consumer/Stream app as the owner of 
member id generation, which brought up multiple validation concerns. As the 
host:port approach is vetoed, and I have thought for a while for other 
validation strategies but failed, I think it's time to decide whether we want 
to focus our next step discussion on


  1.  how to let server validate id uniqueness generated by client, or
  2.  how to let client materialize the id provided by the server


Let me know your favor of the two topics here. Personally I'm inclined to 2 as 
1 is intrinsically complex (no source of truth model). Hope we are on the same 
page now.


Best,

Boyang


From: John Roesler 
Sent: Wednesday, August 1, 2018 5:28 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hi Boyang,

Overall, this seems like a good addition to the consumer.

I agree with the others that we should attempt to validate the uniqueness
of member.id usage. FWIW, Jason's idea of using a configured logical id +
assigned unique id seems to be suitably flexible and free of assumptions,
as opposed to picking machine characteristics like hostname.

It seems like a survey of solutions to the same problem in other software
might help to guide this discussion. Kafka Consumer isn't the first
stateful distributed system to encounter the need for stable shard
assignment. I would think that any mature distributed database and any
stateful stream processing framework would have some algorithm to solve
this problem, some of which might offer unique advantages over what we've
discussed so far.

Hopefully it wouldn't pollute the discussion if I threw out an alternative
idea: It seems like the purpose behind this proposal is to allow stateful
consumers to re-acquire their previous partitions when they return to the
group. In this scenario, it seems reliable that the consumers would know
what partitions they previously had assigned (since they are stateful).
Instead of reporting a member.id, they could just include their prior list
of partitions in the join-group request. This would solve the problem just
as well as member.id, while allowing more advanced techniques like
splitting state between two instances. For example, an instance I1 with TP1
and TP2 state shuts down, and we copy the state for TP2 to I2 (and delete
it from I1); then we start I1 and I2 up, and I1 only reports that it wants
TP1 while I2 only reports that it wants TP2. Thus, we can scale out or in
without incurring costly state reconstruction.


On a separate note, what is the impact of changing the default for
LEAVE_GROUP_ON_CLOSE_CONFIG? It seems like could potentially harm
applications not using member.id. Can we circumvent the issue without
changing that default globally? Such as ignoring that config when member.id
is set?

Thanks for the proposal!
-John

On Mon, Jul 30, 2018 at 9:57 PM Guozhang Wang  wrote:

> @Jason,
>
> Good point about disconnects. And with that I think I agree that a registry
> id maybe a better idea to enable fencing than validating on host / port.
>
>
> Guozhang
>
>
> On Mon, Jul 30, 2018 at 5:40 PM, Jason Gustafson 
> wrote:
>
> > Hey Guozhang,
> >
> > Thanks for the detailed response. Really quick about the fencing issue, I
> > think host/port will not be sufficient because it cannot handle
> > disconnects. For example, if the coordinator moves to another broker,
> then
> > there is no way we'd be able to guarantee the same host/port. Currently
> we
> > try to avoid rebalancing when the coordinator moves. That said, I agree
> in
> > principle with the "first comer wins" approach you've suggested.
> Basically
> > a member is only removed from the group if its session expires or it
> leaves
> > the group explicitly.
> >
> > -Jason
> >
> > On Mon, Jul 30, 2018 at 4:24 PM, Mike Freyberger <
> mfreyber...@appnexus.com
> > >
> > wrote:
> >
> > > Guozhang,
> > >
> > > Thanks for giving us a great starting point.
> > >
> > > A few questions that come to mind right away:
> > >
> > > 1) What do you think a reasonable grou

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[John Roesler]

 > the client side, The correctness will depend on the user
> > logic,
> > > > however I'm
> > > > > pessimistic.
> > > > >
> > > > >
> > > > > Although the impact is very high, the symptom is not hard
> to
> > > triage,
> > > > > because user could visualize consumer identity overlapping
> > > fairly
> > > > easily by
> > > > > exported consumer metrics. On the user standpoint, they
> would
> > > be
> > > > fully
> > > > > aware of the potential erratic status before enabling "
> > > member.id"
> > >     >     > configuration IMO. Let me know your thoughts James!
> > > > >
> > > > >
> > > > > Next is Jason's suggestion. Jason shared a higher viewpoint
> > and
> > > > pointed
> > > > > out the problem that we need to solve is to maintain "a
> > strong
> > > bias
> > > > towards
> > > > > being able to reuse previous state". The proposed approach
> is
> > > to
> > > > separate
> > > > > the notion of consumer membership and consumer identity.
> > > > >
> > > > >
> > > > > The original idea of this KIP was on the Stream
> application,
> > > so I
> > > > > understand that the identities of multiple consumers belong
> > to
> > > one
> > > > > instance, where each Stream thread will be using one
> > dedicated
> > > main
> > > > > consumer. So in a Stream use case, we could internally
> > generate
> > > > member id
> > > > > with USER_DEFINED_ID + STREAM_THREAD_ID.
> > > > >
> > > > >
> > > > > In pure consumer use case, this could be a little bit
> > > challenging
> > > > since
> > > > > user could arbitrarily initiate multiple consumers on the
> > same
> > > > instance
> > > > > which is out of our library control. This could add up the
> > > > possibility of
> > > > > member id collision. So instead of making developers life
> > > easier,
> > > > > introducing member id config could break the existing code
> > > logic and
> > > > take
> > > > > long time to understand and fix. Although I still assume
> this
> > > is an
> > > > > advanced config, user may use member id config even before
> > > they fully
> > > > > understand the problem, and use the same set of
> > initialization
> > > logic
> > > > cross
> > > > > multiple consumers on the same instance.
> > > > >
> > > > >
> > > > > I hope I have explained my understanding of the pros and
> cons
> > > of
> > > > this KIP
> > > > > better. Remember the core argument of this KIP: If the
> broker
> > > > recognize
> > > > > this consumer as an existing member, it shouldn't trigger
> > > rebalance.
> > > > If we
> > > > > build our discussion on top of this argument, the client
> > > management
> > > > of
> > > > > group membership could be tricky at first, but considering
> > our
> > > > original
> > > > > motivation to leader-follower rebalance model, I feel that
> > > having
> > > > broker to
> > > > > create membership info and let client maintain would be
> less
> > > > appealing and
> > > > > fragile. Having client generate membership data could build
> > up
> > > > > source-of-truth model and streamline the current
> > architecture.
> > > We
> > > > need also
> > > > > consider flexibility introduced by this KIP for cloud users
> > to
> > > > coordinate
> > > > > consumer/stream instances more freely. Honestly, I'm
> > > interested in
> > > > Jason's
> > > > > registration id

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Guozhang Wang]

 > >
> > > >
> > > > I think James' question is primarily the severity of user
> > using this
> > > > config wrongly. The impact would be that the same member id
> > being
> > > used by
> > > > multiple or even all of the consumers. The assignment
> protocol
> > > couldn't
> > > > distinguish any of the overlapping consumers, thus assigning
> > the
> > > exact same
> > > > partitions multiple times to different consumers. I would say
> > the
> > > processed
> > > > result would be including a lot of duplicates and unnecessary
> > heavy
> > > load on
> > > > the client side, The correctness will depend on the user
> logic,
> > > however I'm
> > > > pessimistic.
> > > >
> > > >
> > > > Although the impact is very high, the symptom is not hard to
> > triage,
> > > > because user could visualize consumer identity overlapping
> > fairly
> > > easily by
> > > > exported consumer metrics. On the user standpoint, they would
> > be
> > > fully
> > > > aware of the potential erratic status before enabling "
> > member.id"
> > > > configuration IMO. Let me know your thoughts James!
> > > >
> > > >
> > > > Next is Jason's suggestion. Jason shared a higher viewpoint
> and
> > > pointed
> > > > out the problem that we need to solve is to maintain "a
> strong
> > bias
> > > towards
> > > > being able to reuse previous state". The proposed approach is
> > to
> > > separate
> > > > the notion of consumer membership and consumer identity.
> > > >
> > > >
> > > > The original idea of this KIP was on the Stream application,
> > so I
> > > > understand that the identities of multiple consumers belong
> to
> > one
> > > > instance, where each Stream thread will be using one
> dedicated
> > main
> > > > consumer. So in a Stream use case, we could internally
> generate
> > > member id
> > > > with USER_DEFINED_ID + STREAM_THREAD_ID.
> > > >
> > > >
> > > > In pure consumer use case, this could be a little bit
> > challenging
> > > since
> > > > user could arbitrarily initiate multiple consumers on the
> same
> > > instance
> > > > which is out of our library control. This could add up the
> > > possibility of
> > > > member id collision. So instead of making developers life
> > easier,
> > > > introducing member id config could break the existing code
> > logic and
> > > take
> > > > long time to understand and fix. Although I still assume this
> > is an
> > > > advanced config, user may use member id config even before
> > they fully
> > > > understand the problem, and use the same set of
> initialization
> > logic
> > > cross
> > > > multiple consumers on the same instance.
> > > >
> > > >
> > > > I hope I have explained my understanding of the pros and cons
> > of
> > > this KIP
> > > > better. Remember the core argument of this KIP: If the broker
> > > recognize
> > > > this consumer as an existing member, it shouldn't trigger
> > rebalance.
> > > If we
> > > > build our discussion on top of this argument, the client
> > management
> > > of
> > > > group membership could be tricky at first, but considering
> our
> > > original
> > > > motivation to leader-follower rebalance model, I feel that
> > having
> > > broker to
> > > > create membership info and let client maintain would be less
> > > appealing and
> > > > fragile. Having client generate membership data could build
> up
> > > > source-of-truth model and streamline the current
> architecture.
> > We
> > > need also
> > > 

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Guozhang Wang]

me know your thoughts James!
> > >
> > >
> > > Next is Jason's suggestion. Jason shared a higher viewpoint and
> > pointed
> > > out the problem that we need to solve is to maintain "a strong
> bias
> > towards
> > > being able to reuse previous state". The proposed approach is
> to
> > separate
> > > the notion of consumer membership and consumer identity.
> > >
> > >
> > > The original idea of this KIP was on the Stream application,
> so I
> > > understand that the identities of multiple consumers belong to
> one
> > > instance, where each Stream thread will be using one dedicated
> main
> > > consumer. So in a Stream use case, we could internally generate
> > member id
> > > with USER_DEFINED_ID + STREAM_THREAD_ID.
> > >
> > >
> > > In pure consumer use case, this could be a little bit
> challenging
> > since
> > > user could arbitrarily initiate multiple consumers on the same
> > instance
> > > which is out of our library control. This could add up the
> > possibility of
> > > member id collision. So instead of making developers life
> easier,
> > > introducing member id config could break the existing code
> logic and
> > take
> > > long time to understand and fix. Although I still assume this
> is an
> > > advanced config, user may use member id config even before
> they fully
> > > understand the problem, and use the same set of initialization
> logic
> > cross
> > > multiple consumers on the same instance.
> > >
> > >
> > > I hope I have explained my understanding of the pros and cons
> of
> > this KIP
> > > better. Remember the core argument of this KIP: If the broker
> > recognize
> > > this consumer as an existing member, it shouldn't trigger
> rebalance.
> > If we
> > > build our discussion on top of this argument, the client
> management
> > of
> > > group membership could be tricky at first, but considering our
> > original
> > > motivation to leader-follower rebalance model, I feel that
> having
> > broker to
> > > create membership info and let client maintain would be less
> > appealing and
> > > fragile. Having client generate membership data could build up
> > > source-of-truth model and streamline the current architecture.
> We
> > need also
> > > consider flexibility introduced by this KIP for cloud users to
> > coordinate
> > > consumer/stream instances more freely. Honestly, I'm
> interested in
> > Jason's
> > > registration id proposal and open to more voices, but I feel it
> > would be
> > > more complex than the current KIP for user to understand. Hope
> this
> > makes
> > > sense, Jason.
> > >
> > >
> > > Thank you again for the feedback!
> > >
> > >
> > > Best,
> > >
> > > Boyang
> > >
> > >
> > > 
> > > From: Jason Gustafson 
> > > Sent: Saturday, July 28, 2018 6:50 AM
> > > To: dev
> > > Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer
> rebalances
> > by
> > > specifying member id
> > >
> > > Hey Boyang,
> > >
> > > Thanks for the KIP. I think my main question is in the same
> vein as
> > James'.
> > > The problem is that the coordinator needs to be able to
> identify
> > which
> > > instance of a particular memberId is the active one. For EOS,
> each
> > > transactionalId gets an epoch. When a new producer is started,
> it
> > bumps the
> > > epoch which allows the transaction coordinator to fence off any
> > zombie
> > > instances which may try to continue doing work with the old
> epoch.
> > It seems
> > > like we need a similar protection for consumer members.
> > >
> >

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Jason Gustafson]

>
> > >
> > > The original idea of this KIP was on the Stream application,
> so I
> > > understand that the identities of multiple consumers belong to
> one
> > > instance, where each Stream thread will be using one dedicated
> main
> > > consumer. So in a Stream use case, we could internally generate
> > member id
> > > with USER_DEFINED_ID + STREAM_THREAD_ID.
> > >
> > >
> > > In pure consumer use case, this could be a little bit
> challenging
> > since
> > > user could arbitrarily initiate multiple consumers on the same
> > instance
> > > which is out of our library control. This could add up the
> > possibility of
> > > member id collision. So instead of making developers life
> easier,
> > > introducing member id config could break the existing code
> logic and
> > take
> > > long time to understand and fix. Although I still assume this
> is an
> > > advanced config, user may use member id config even before
> they fully
> > > understand the problem, and use the same set of initialization
> logic
> > cross
> > > multiple consumers on the same instance.
> > >
> > >
> > > I hope I have explained my understanding of the pros and cons
> of
> > this KIP
> > > better. Remember the core argument of this KIP: If the broker
> > recognize
> > > this consumer as an existing member, it shouldn't trigger
> rebalance.
> > If we
> > > build our discussion on top of this argument, the client
> management
> > of
> > > group membership could be tricky at first, but considering our
> > original
> > > motivation to leader-follower rebalance model, I feel that
> having
> > broker to
> > > create membership info and let client maintain would be less
> > appealing and
> > > fragile. Having client generate membership data could build up
> > > source-of-truth model and streamline the current architecture.
> We
> > need also
> > > consider flexibility introduced by this KIP for cloud users to
> > coordinate
> > > consumer/stream instances more freely. Honestly, I'm
> interested in
> > Jason's
> > > registration id proposal and open to more voices, but I feel it
> > would be
> > > more complex than the current KIP for user to understand. Hope
> this
> > makes
> > > sense, Jason.
> > >
> > >
> > > Thank you again for the feedback!
> > >
> > >
> > > Best,
> > >
> > > Boyang
> > >
> > >
> > > 
> > > From: Jason Gustafson 
> > > Sent: Saturday, July 28, 2018 6:50 AM
> > > To: dev
> > > Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer
> rebalances
> > by
> > > specifying member id
> > >
> > > Hey Boyang,
> > >
> > > Thanks for the KIP. I think my main question is in the same
> vein as
> > James'.
> > > The problem is that the coordinator needs to be able to
> identify
> > which
> > > instance of a particular memberId is the active one. For EOS,
> each
> > > transactionalId gets an epoch. When a new producer is started,
> it
> > bumps the
> > > epoch which allows the transaction coordinator to fence off any
> > zombie
> > > instances which may try to continue doing work with the old
> epoch.
> > It seems
> > > like we need a similar protection for consumer members.
> > >
> > > Suppose for example that we distinguish between a registration
> id
> > which is
> > > provided by the user and a member id which is assigned
> uniquely by
> > the
> > > coordinator. In the JoinGroup request, both the registration
> id and
> > the
> > > member id are provided. When a consumer is first started, it
> doesn't
> > know
> > > the memberId, s

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Mike Freyberger]

 need also
>     > consider flexibility introduced by this KIP for cloud users to
> coordinate
> > consumer/stream instances more freely. Honestly, I'm interested in
> Jason's
> > registration id proposal and open to more voices, but I feel it
> would be
> > more complex than the current KIP for user to understand. Hope this
> makes
> > sense, Jason.
> >
> >
> > Thank you again for the feedback!
> >
> >
> > Best,
> >
> > Boyang
> >
> >
> > 
> > From: Jason Gustafson 
> > Sent: Saturday, July 28, 2018 6:50 AM
> > To: dev
> > Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances
> by
> > specifying member id
> >
> > Hey Boyang,
> >
> > Thanks for the KIP. I think my main question is in the same vein as
> James'.
> > The problem is that the coordinator needs to be able to identify
> which
> > instance of a particular memberId is the active one. For EOS, each
> > transactionalId gets an epoch. When a new producer is started, it
> bumps the
> > epoch which allows the transaction coordinator to fence off any
> zombie
> > instances which may try to continue doing work with the old epoch.
> It seems
> > like we need a similar protection for consumer members.
> >
> > Suppose for example that we distinguish between a registration id
> which is
> > provided by the user and a member id which is assigned uniquely by
> the
> > coordinator. In the JoinGroup request, both the registration id and
> the
> > member id are provided. When a consumer is first started, it doesn't
> know
> > the memberId, so it it provides only the registration id. The
> coordinator
> > can then assign a new memberId and invalidate the previous one that
> was
> > associated with the registration id. This would then fence off the
> previous
> > instance which was still trying to use the member id.
> >
> > Taking a little bit of a step back, I think the main observation in
> this
> > KIP is that applications with heavy local state need to have a
> strong bias
> > toward being able to reuse that state. It is a bit like Kafka itself
> in the
> > sense that a replica is not moved just because the broker is
> shutdown as
> > the cost of moving the log is extremely high. I'm wondering if we
> need to
> > think about streams applications in a similar way. Should there be a
> static
> > notion of the members of the group so that streams can make
> rebalancing
> > decisions more easily without depending so heavily on transient
> membership?
> > I feel the hacks we've put in place in some cases to avoid
> rebalances are a
> > bit brittle. Delaying group joining for example is an example of
> this. If
> > you knew ahead of time who the stable members of the group were,
> then this
> > would not be needed. Anyway, just a thought.
> >
> > Thanks,
> > Jason
> >
> >
> >
> > On Fri, Jul 27, 2018 at 1:58 PM, James Cheng 
> wrote:
> >
> > > When you say that it will "break", what does this breakage look
> like?
> > Will
> > > the consumer-group be non-functional? Will just those instances be
> > > non-functional? Or will the group be functional, but the
> rebalancing be
> > > non-optimal and require more round-trips/data-transfer? (similar
> to the
>     > > current algorithm)
> > >
> > > I'm trying to assess the potential for user-error and the impact 
of
> > > user-error.
> > >
> > > -James
> > >
> > > > On Jul 27, 2018, at 11:25 AM, Boyang Chen 
> wrote:
> > > >
> > > > Hey James,
> > > >
> > > >
> > > > the algorithm is relying on client side to provide unique
> consumer
> > > member id. It will break unless we enfor

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Guozhang Wang]

dpoint, they would be
> fully
> > aware of the potential erratic status before enabling "member.id"
> > configuration IMO. Let me know your thoughts James!
> >
> >
> > Next is Jason's suggestion. Jason shared a higher viewpoint and
> pointed
> > out the problem that we need to solve is to maintain "a strong bias
> towards
> > being able to reuse previous state". The proposed approach is to
> separate
> > the notion of consumer membership and consumer identity.
> >
> >
> > The original idea of this KIP was on the Stream application, so I
> > understand that the identities of multiple consumers belong to one
> > instance, where each Stream thread will be using one dedicated main
> > consumer. So in a Stream use case, we could internally generate
> member id
> > with USER_DEFINED_ID + STREAM_THREAD_ID.
> >
> >
> > In pure consumer use case, this could be a little bit challenging
> since
> > user could arbitrarily initiate multiple consumers on the same
> instance
> > which is out of our library control. This could add up the
> possibility of
> > member id collision. So instead of making developers life easier,
> > introducing member id config could break the existing code logic and
> take
> > long time to understand and fix. Although I still assume this is an
> > advanced config, user may use member id config even before they fully
> > understand the problem, and use the same set of initialization logic
> cross
> > multiple consumers on the same instance.
> >
> >
> > I hope I have explained my understanding of the pros and cons of
> this KIP
> > better. Remember the core argument of this KIP: If the broker
> recognize
> > this consumer as an existing member, it shouldn't trigger rebalance.
> If we
> > build our discussion on top of this argument, the client management
> of
> > group membership could be tricky at first, but considering our
> original
> > motivation to leader-follower rebalance model, I feel that having
> broker to
> > create membership info and let client maintain would be less
> appealing and
> > fragile. Having client generate membership data could build up
> > source-of-truth model and streamline the current architecture. We
> need also
> > consider flexibility introduced by this KIP for cloud users to
> coordinate
> > consumer/stream instances more freely. Honestly, I'm interested in
> Jason's
> > registration id proposal and open to more voices, but I feel it
> would be
> > more complex than the current KIP for user to understand. Hope this
> makes
> > sense, Jason.
> >
> >
> > Thank you again for the feedback!
> >
> >
> > Best,
> >
> > Boyang
> >
> >
> > 
> > From: Jason Gustafson 
> > Sent: Saturday, July 28, 2018 6:50 AM
> > To: dev
> > Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances
> by
> > specifying member id
> >
> > Hey Boyang,
> >
> > Thanks for the KIP. I think my main question is in the same vein as
> James'.
> > The problem is that the coordinator needs to be able to identify
> which
> > instance of a particular memberId is the active one. For EOS, each
> > transactionalId gets an epoch. When a new producer is started, it
> bumps the
> > epoch which allows the transaction coordinator to fence off any
> zombie
> > instances which may try to continue doing work with the old epoch.
> It seems
> > like we need a similar protection for consumer members.
> >
> > Suppose for example that we distinguish between a registration id
> which is
> > provided by the user and a member id which is assigned uniquely by
> the
> > coordinator. In the JoinGroup request, both the registration id and
> the
> > member id are provided. When a consumer is first started, it doesn't
> know
> > the memberId, so it it provides only the registration id. The
> coordinator
> > can then assign a new memberId and invalidate the previous one that
> was
> > associated with the registration id. This would then fence off the
> previous
> > instance which was still trying to use the member id.
> >
> > Taking a little bit of a step back, I think the main observation in

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Mike Freyberger]

ng developers life easier,
> introducing member id config could break the existing code logic and take
> long time to understand and fix. Although I still assume this is an
> advanced config, user may use member id config even before they fully
> understand the problem, and use the same set of initialization logic cross
> multiple consumers on the same instance.
>
>
> I hope I have explained my understanding of the pros and cons of this KIP
> better. Remember the core argument of this KIP: If the broker recognize
> this consumer as an existing member, it shouldn't trigger rebalance. If we
> build our discussion on top of this argument, the client management of
> group membership could be tricky at first, but considering our original
> motivation to leader-follower rebalance model, I feel that having broker 
to
> create membership info and let client maintain would be less appealing and
> fragile. Having client generate membership data could build up
> source-of-truth model and streamline the current architecture. We need 
also
> consider flexibility introduced by this KIP for cloud users to coordinate
> consumer/stream instances more freely. Honestly, I'm interested in Jason's
> registration id proposal and open to more voices, but I feel it would be
> more complex than the current KIP for user to understand. Hope this makes
> sense, Jason.
>
>
> Thank you again for the feedback!
    >
    >
    > Best,
>
> Boyang
    >
    >
> 
> From: Jason Gustafson 
> Sent: Saturday, July 28, 2018 6:50 AM
> To: dev
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> Hey Boyang,
>
> Thanks for the KIP. I think my main question is in the same vein as 
James'.
> The problem is that the coordinator needs to be able to identify which
> instance of a particular memberId is the active one. For EOS, each
> transactionalId gets an epoch. When a new producer is started, it bumps 
the
> epoch which allows the transaction coordinator to fence off any zombie
> instances which may try to continue doing work with the old epoch. It 
seems
> like we need a similar protection for consumer members.
>
> Suppose for example that we distinguish between a registration id which is
> provided by the user and a member id which is assigned uniquely by the
> coordinator. In the JoinGroup request, both the registration id and the
> member id are provided. When a consumer is first started, it doesn't know
> the memberId, so it it provides only the registration id. The coordinator
> can then assign a new memberId and invalidate the previous one that was
> associated with the registration id. This would then fence off the 
previous
> instance which was still trying to use the member id.
>
> Taking a little bit of a step back, I think the main observation in this
> KIP is that applications with heavy local state need to have a strong bias
> toward being able to reuse that state. It is a bit like Kafka itself in 
the
> sense that a replica is not moved just because the broker is shutdown as
> the cost of moving the log is extremely high. I'm wondering if we need to
> think about streams applications in a similar way. Should there be a 
static
> notion of the members of the group so that streams can make rebalancing
> decisions more easily without depending so heavily on transient 
membership?
> I feel the hacks we've put in place in some cases to avoid rebalances are 
a
> bit brittle. Delaying group joining for example is an example of this. If
> you knew ahead of time who the stable members of the group were, then this
> would not be needed. Anyway, just a thought.
>
> Thanks,
> Jason
>
>
>
> On Fri, Jul 27, 2018 at 1:58 PM, James Cheng  wrote:
>
> > When you say that it will "break", what does this breakage look like?
> Will
> > the consumer-group be non-functional? Will just those instances be
> > non-functional? Or will the group be functional, but the rebalancing be
> > non-optimal and require more round-trips/data-transfer? (similar to the
> > current algorithm)
> >
> > I'm trying to assess the potential for user-error and the impact of
> > user-error.
> >
> > -James
> >
> > > On Jul 27, 2018, at 11:25 AM, Boyang Chen  wrote:
> > >
> > > Hey James,
> > >
> >

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Jason Gustafson]

e need also
> consider flexibility introduced by this KIP for cloud users to coordinate
> consumer/stream instances more freely. Honestly, I'm interested in Jason's
> registration id proposal and open to more voices, but I feel it would be
> more complex than the current KIP for user to understand. Hope this makes
> sense, Jason.
>
>
> Thank you again for the feedback!
>
>
> Best,
>
> Boyang
>
>
> 
> From: Jason Gustafson 
> Sent: Saturday, July 28, 2018 6:50 AM
> To: dev
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> Hey Boyang,
>
> Thanks for the KIP. I think my main question is in the same vein as James'.
> The problem is that the coordinator needs to be able to identify which
> instance of a particular memberId is the active one. For EOS, each
> transactionalId gets an epoch. When a new producer is started, it bumps the
> epoch which allows the transaction coordinator to fence off any zombie
> instances which may try to continue doing work with the old epoch. It seems
> like we need a similar protection for consumer members.
>
> Suppose for example that we distinguish between a registration id which is
> provided by the user and a member id which is assigned uniquely by the
> coordinator. In the JoinGroup request, both the registration id and the
> member id are provided. When a consumer is first started, it doesn't know
> the memberId, so it it provides only the registration id. The coordinator
> can then assign a new memberId and invalidate the previous one that was
> associated with the registration id. This would then fence off the previous
> instance which was still trying to use the member id.
>
> Taking a little bit of a step back, I think the main observation in this
> KIP is that applications with heavy local state need to have a strong bias
> toward being able to reuse that state. It is a bit like Kafka itself in the
> sense that a replica is not moved just because the broker is shutdown as
> the cost of moving the log is extremely high. I'm wondering if we need to
> think about streams applications in a similar way. Should there be a static
> notion of the members of the group so that streams can make rebalancing
> decisions more easily without depending so heavily on transient membership?
> I feel the hacks we've put in place in some cases to avoid rebalances are a
> bit brittle. Delaying group joining for example is an example of this. If
> you knew ahead of time who the stable members of the group were, then this
> would not be needed. Anyway, just a thought.
>
> Thanks,
> Jason
>
>
>
> On Fri, Jul 27, 2018 at 1:58 PM, James Cheng  wrote:
>
> > When you say that it will "break", what does this breakage look like?
> Will
> > the consumer-group be non-functional? Will just those instances be
> > non-functional? Or will the group be functional, but the rebalancing be
> > non-optimal and require more round-trips/data-transfer? (similar to the
> > current algorithm)
> >
> > I'm trying to assess the potential for user-error and the impact of
> > user-error.
> >
> > -James
> >
> > > On Jul 27, 2018, at 11:25 AM, Boyang Chen  wrote:
> > >
> > > Hey James,
> > >
> > >
> > > the algorithm is relying on client side to provide unique consumer
> > member id. It will break unless we enforce some sort of validation (host
> +
> > port) on the server side. To simplify the first version, we do not plan
> to
> > enforce validation. A good comparison would be the EOS producer which is
> in
> > charge of generating unique transaction id sequence. IMO for broker
> logic,
> > the tolerance of client side error is not unlimited.
> > >
> > >
> > > Thank you!
> > >
> > >
> > > 
> > > From: James Cheng 
> > > Sent: Saturday, July 28, 2018 1:26 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> > specifying member id
> > >
> > >
> > >> On Jul 26, 2018, at 11:09 PM, Guozhang Wang 
> wrote:
> > >>
> > >> Hi Boyang,
> > >>
> > >> Thanks for the proposed KIP. I made a pass over the wiki and here are
> > some
> > >> comments / questions:
> > >>
> > >> 1. In order to preserve broker compatibility, we need to make sure the
> > >> broker version discovery logic can be integrated with this new logic.
> > I.e.
> > >> if a newer versioned consumer is talki

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

Thanks for the replies, James and Jason. Let me try to summarize your concerns.


I think James' question is primarily the severity of user using this config 
wrongly. The impact would be that the same member id being used by multiple or 
even all of the consumers. The assignment protocol couldn't distinguish any of 
the overlapping consumers, thus assigning the exact same partitions multiple 
times to different consumers. I would say the processed result would be 
including a lot of duplicates and unnecessary heavy load on the client side, 
The correctness will depend on the user logic, however I'm pessimistic.


Although the impact is very high, the symptom is not hard to triage, because 
user could visualize consumer identity overlapping fairly easily by exported 
consumer metrics. On the user standpoint, they would be fully aware of the 
potential erratic status before enabling "member.id" configuration IMO. Let me 
know your thoughts James!


Next is Jason's suggestion. Jason shared a higher viewpoint and pointed out the 
problem that we need to solve is to maintain "a strong bias towards being able 
to reuse previous state". The proposed approach is to separate the notion of 
consumer membership and consumer identity.


The original idea of this KIP was on the Stream application, so I understand 
that the identities of multiple consumers belong to one instance, where each 
Stream thread will be using one dedicated main consumer. So in a Stream use 
case, we could internally generate member id with USER_DEFINED_ID + 
STREAM_THREAD_ID.


In pure consumer use case, this could be a little bit challenging since user 
could arbitrarily initiate multiple consumers on the same instance which is out 
of our library control. This could add up the possibility of member id 
collision. So instead of making developers life easier, introducing member id 
config could break the existing code logic and take long time to understand and 
fix. Although I still assume this is an advanced config, user may use member id 
config even before they fully understand the problem, and use the same set of 
initialization logic cross multiple consumers on the same instance.


I hope I have explained my understanding of the pros and cons of this KIP 
better. Remember the core argument of this KIP: If the broker recognize this 
consumer as an existing member, it shouldn't trigger rebalance. If we build our 
discussion on top of this argument, the client management of group membership 
could be tricky at first, but considering our original motivation to 
leader-follower rebalance model, I feel that having broker to create membership 
info and let client maintain would be less appealing and fragile. Having client 
generate membership data could build up source-of-truth model and streamline 
the current architecture. We need also consider flexibility introduced by this 
KIP for cloud users to coordinate consumer/stream instances more freely. 
Honestly, I'm interested in Jason's registration id proposal and open to more 
voices, but I feel it would be more complex than the current KIP for user to 
understand. Hope this makes sense, Jason.


Thank you again for the feedback!


Best,

Boyang



From: Jason Gustafson 
Sent: Saturday, July 28, 2018 6:50 AM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hey Boyang,

Thanks for the KIP. I think my main question is in the same vein as James'.
The problem is that the coordinator needs to be able to identify which
instance of a particular memberId is the active one. For EOS, each
transactionalId gets an epoch. When a new producer is started, it bumps the
epoch which allows the transaction coordinator to fence off any zombie
instances which may try to continue doing work with the old epoch. It seems
like we need a similar protection for consumer members.

Suppose for example that we distinguish between a registration id which is
provided by the user and a member id which is assigned uniquely by the
coordinator. In the JoinGroup request, both the registration id and the
member id are provided. When a consumer is first started, it doesn't know
the memberId, so it it provides only the registration id. The coordinator
can then assign a new memberId and invalidate the previous one that was
associated with the registration id. This would then fence off the previous
instance which was still trying to use the member id.

Taking a little bit of a step back, I think the main observation in this
KIP is that applications with heavy local state need to have a strong bias
toward being able to reuse that state. It is a bit like Kafka itself in the
sense that a replica is not moved just because the broker is shutdown as
the cost of moving the log is extremely high. I'm wondering if we need to
think about streams applications in a similar way. Should there be a static
notion of the members of the group so t

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Jason Gustafson]

Hey Boyang,

Thanks for the KIP. I think my main question is in the same vein as James'.
The problem is that the coordinator needs to be able to identify which
instance of a particular memberId is the active one. For EOS, each
transactionalId gets an epoch. When a new producer is started, it bumps the
epoch which allows the transaction coordinator to fence off any zombie
instances which may try to continue doing work with the old epoch. It seems
like we need a similar protection for consumer members.

Suppose for example that we distinguish between a registration id which is
provided by the user and a member id which is assigned uniquely by the
coordinator. In the JoinGroup request, both the registration id and the
member id are provided. When a consumer is first started, it doesn't know
the memberId, so it it provides only the registration id. The coordinator
can then assign a new memberId and invalidate the previous one that was
associated with the registration id. This would then fence off the previous
instance which was still trying to use the member id.

Taking a little bit of a step back, I think the main observation in this
KIP is that applications with heavy local state need to have a strong bias
toward being able to reuse that state. It is a bit like Kafka itself in the
sense that a replica is not moved just because the broker is shutdown as
the cost of moving the log is extremely high. I'm wondering if we need to
think about streams applications in a similar way. Should there be a static
notion of the members of the group so that streams can make rebalancing
decisions more easily without depending so heavily on transient membership?
I feel the hacks we've put in place in some cases to avoid rebalances are a
bit brittle. Delaying group joining for example is an example of this. If
you knew ahead of time who the stable members of the group were, then this
would not be needed. Anyway, just a thought.

Thanks,
Jason



On Fri, Jul 27, 2018 at 1:58 PM, James Cheng  wrote:

> When you say that it will "break", what does this breakage look like? Will
> the consumer-group be non-functional? Will just those instances be
> non-functional? Or will the group be functional, but the rebalancing be
> non-optimal and require more round-trips/data-transfer? (similar to the
> current algorithm)
>
> I'm trying to assess the potential for user-error and the impact of
> user-error.
>
> -James
>
> > On Jul 27, 2018, at 11:25 AM, Boyang Chen  wrote:
> >
> > Hey James,
> >
> >
> > the algorithm is relying on client side to provide unique consumer
> member id. It will break unless we enforce some sort of validation (host +
> port) on the server side. To simplify the first version, we do not plan to
> enforce validation. A good comparison would be the EOS producer which is in
> charge of generating unique transaction id sequence. IMO for broker logic,
> the tolerance of client side error is not unlimited.
> >
> >
> > Thank you!
> >
> >
> > ________________
> > From: James Cheng 
> > Sent: Saturday, July 28, 2018 1:26 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
> >
> >
> >> On Jul 26, 2018, at 11:09 PM, Guozhang Wang  wrote:
> >>
> >> Hi Boyang,
> >>
> >> Thanks for the proposed KIP. I made a pass over the wiki and here are
> some
> >> comments / questions:
> >>
> >> 1. In order to preserve broker compatibility, we need to make sure the
> >> broker version discovery logic can be integrated with this new logic.
> I.e.
> >> if a newer versioned consumer is talking to an older versioned broker
> who
> >> does not recognize V4, the client needs to downgrade its
> JoinGroupRequest
> >> version to V3 and not setting the member-id specifically. You can take a
> >> look at the ApiVersionsRequest and see how to work with it.
> >>
> >> 2. There may exist some manners to validate that two different clients
> do
> >> not send with the same member id, for example if we pass along the
> >> host:port information from KafkaApis to the GroupCoordinator interface.
> But
> >> I think this is overly complicated the logic and may not worthwhile than
> >> relying on users to specify unique member ids.
> >
> > Boyang,
> >
> > Thanks for the KIP! How will the algorithm behave if multiple consumers
> provide the same member id?
> >
> > -James
> >
> >> 3. Minor: you would need to bumping up the version of JoinGroupResponse
> to
> >> V4 as well.
> >>
> >> 4. Minor: in the wiki page, you need to specify the actu

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[James Cheng]

When you say that it will "break", what does this breakage look like? Will the 
consumer-group be non-functional? Will just those instances be non-functional? 
Or will the group be functional, but the rebalancing be non-optimal and require 
more round-trips/data-transfer? (similar to the current algorithm)

I'm trying to assess the potential for user-error and the impact of user-error.

-James

> On Jul 27, 2018, at 11:25 AM, Boyang Chen  wrote:
> 
> Hey James,
> 
> 
> the algorithm is relying on client side to provide unique consumer member id. 
> It will break unless we enforce some sort of validation (host + port) on the 
> server side. To simplify the first version, we do not plan to enforce 
> validation. A good comparison would be the EOS producer which is in charge of 
> generating unique transaction id sequence. IMO for broker logic, the 
> tolerance of client side error is not unlimited.
> 
> 
> Thank you!
> 
> 
> 
> From: James Cheng 
> Sent: Saturday, July 28, 2018 1:26 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
> specifying member id
> 
> 
>> On Jul 26, 2018, at 11:09 PM, Guozhang Wang  wrote:
>> 
>> Hi Boyang,
>> 
>> Thanks for the proposed KIP. I made a pass over the wiki and here are some
>> comments / questions:
>> 
>> 1. In order to preserve broker compatibility, we need to make sure the
>> broker version discovery logic can be integrated with this new logic. I.e.
>> if a newer versioned consumer is talking to an older versioned broker who
>> does not recognize V4, the client needs to downgrade its JoinGroupRequest
>> version to V3 and not setting the member-id specifically. You can take a
>> look at the ApiVersionsRequest and see how to work with it.
>> 
>> 2. There may exist some manners to validate that two different clients do
>> not send with the same member id, for example if we pass along the
>> host:port information from KafkaApis to the GroupCoordinator interface. But
>> I think this is overly complicated the logic and may not worthwhile than
>> relying on users to specify unique member ids.
> 
> Boyang,
> 
> Thanks for the KIP! How will the algorithm behave if multiple consumers 
> provide the same member id?
> 
> -James
> 
>> 3. Minor: you would need to bumping up the version of JoinGroupResponse to
>> V4 as well.
>> 
>> 4. Minor: in the wiki page, you need to specify the actual string value for
>> `MEMBER_ID`, for example "member.id".
>> 
>> 5. When this additional config it specified by users, we should consider
>> setting the default of internal `LEAVE_GROUP_ON_CLOSE_CONFIG` to false,
>> since otherwise its effectiveness would be less.
>> 
>> 
>> Guozhang
>> 
>> 
>> 
>>> On Thu, Jul 26, 2018 at 9:20 PM, Boyang Chen  wrote:
>>> 
>>> Hey friends,
>>> 
>>> 
>>> I would like to open a discussion thread on KIP-345:
>>> 
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A
>>> +Reduce+multiple+consumer+rebalances+by+specifying+member+id
>>> 
>>> 
>>> This KIP is trying to resolve multiple rebalances by maintaining the
>>> consumer member id across rebalance generations. I have verified the theory
>>> on our internal Stream application, and it could reduce rebalance time to a
>>> few seconds when service is rolling restart.
>>> 
>>> 
>>> Let me know your thoughts, thank you!
>>> 
>>> 
>>> Best,
>>> 
>>> Boyang
>>> 
>> 
>> 
>> 
>> --
>> -- Guozhang

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

Hey James,


the algorithm is relying on client side to provide unique consumer member id. 
It will break unless we enforce some sort of validation (host + port) on the 
server side. To simplify the first version, we do not plan to enforce 
validation. A good comparison would be the EOS producer which is in charge of 
generating unique transaction id sequence. IMO for broker logic, the tolerance 
of client side error is not unlimited.


Thank you!



From: James Cheng 
Sent: Saturday, July 28, 2018 1:26 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id


> On Jul 26, 2018, at 11:09 PM, Guozhang Wang  wrote:
>
> Hi Boyang,
>
> Thanks for the proposed KIP. I made a pass over the wiki and here are some
> comments / questions:
>
> 1. In order to preserve broker compatibility, we need to make sure the
> broker version discovery logic can be integrated with this new logic. I.e.
> if a newer versioned consumer is talking to an older versioned broker who
> does not recognize V4, the client needs to downgrade its JoinGroupRequest
> version to V3 and not setting the member-id specifically. You can take a
> look at the ApiVersionsRequest and see how to work with it.
>
> 2. There may exist some manners to validate that two different clients do
> not send with the same member id, for example if we pass along the
> host:port information from KafkaApis to the GroupCoordinator interface. But
> I think this is overly complicated the logic and may not worthwhile than
> relying on users to specify unique member ids.

Boyang,

Thanks for the KIP! How will the algorithm behave if multiple consumers provide 
the same member id?

-James

> 3. Minor: you would need to bumping up the version of JoinGroupResponse to
> V4 as well.
>
> 4. Minor: in the wiki page, you need to specify the actual string value for
> `MEMBER_ID`, for example "member.id".
>
> 5. When this additional config it specified by users, we should consider
> setting the default of internal `LEAVE_GROUP_ON_CLOSE_CONFIG` to false,
> since otherwise its effectiveness would be less.
>
>
> Guozhang
>
>
>
>> On Thu, Jul 26, 2018 at 9:20 PM, Boyang Chen  wrote:
>>
>> Hey friends,
>>
>>
>> I would like to open a discussion thread on KIP-345:
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A
>> +Reduce+multiple+consumer+rebalances+by+specifying+member+id
>>
>>
>> This KIP is trying to resolve multiple rebalances by maintaining the
>> consumer member id across rebalance generations. I have verified the theory
>> on our internal Stream application, and it could reduce rebalance time to a
>> few seconds when service is rolling restart.
>>
>>
>> Let me know your thoughts, thank you!
>>
>>
>> Best,
>>
>> Boyang
>>
>
>
>
> --
> -- Guozhang

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[James Cheng]

> On Jul 26, 2018, at 11:09 PM, Guozhang Wang  wrote:
> 
> Hi Boyang,
> 
> Thanks for the proposed KIP. I made a pass over the wiki and here are some
> comments / questions:
> 
> 1. In order to preserve broker compatibility, we need to make sure the
> broker version discovery logic can be integrated with this new logic. I.e.
> if a newer versioned consumer is talking to an older versioned broker who
> does not recognize V4, the client needs to downgrade its JoinGroupRequest
> version to V3 and not setting the member-id specifically. You can take a
> look at the ApiVersionsRequest and see how to work with it.
> 
> 2. There may exist some manners to validate that two different clients do
> not send with the same member id, for example if we pass along the
> host:port information from KafkaApis to the GroupCoordinator interface. But
> I think this is overly complicated the logic and may not worthwhile than
> relying on users to specify unique member ids.

Boyang,

Thanks for the KIP! How will the algorithm behave if multiple consumers provide 
the same member id?

-James

> 3. Minor: you would need to bumping up the version of JoinGroupResponse to
> V4 as well.
> 
> 4. Minor: in the wiki page, you need to specify the actual string value for
> `MEMBER_ID`, for example "member.id".
> 
> 5. When this additional config it specified by users, we should consider
> setting the default of internal `LEAVE_GROUP_ON_CLOSE_CONFIG` to false,
> since otherwise its effectiveness would be less.
> 
> 
> Guozhang
> 
> 
> 
>> On Thu, Jul 26, 2018 at 9:20 PM, Boyang Chen  wrote:
>> 
>> Hey friends,
>> 
>> 
>> I would like to open a discussion thread on KIP-345:
>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A
>> +Reduce+multiple+consumer+rebalances+by+specifying+member+id
>> 
>> 
>> This KIP is trying to resolve multiple rebalances by maintaining the
>> consumer member id across rebalance generations. I have verified the theory
>> on our internal Stream application, and it could reduce rebalance time to a
>> few seconds when service is rolling restart.
>> 
>> 
>> Let me know your thoughts, thank you!
>> 
>> 
>> Best,
>> 
>> Boyang
>> 
> 
> 
> 
> -- 
> -- Guozhang

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Guozhang Wang]

Hi Boyang,

Thanks for the proposed KIP. I made a pass over the wiki and here are some
comments / questions:

1. In order to preserve broker compatibility, we need to make sure the
broker version discovery logic can be integrated with this new logic. I.e.
if a newer versioned consumer is talking to an older versioned broker who
does not recognize V4, the client needs to downgrade its JoinGroupRequest
version to V3 and not setting the member-id specifically. You can take a
look at the ApiVersionsRequest and see how to work with it.

2. There may exist some manners to validate that two different clients do
not send with the same member id, for example if we pass along the
host:port information from KafkaApis to the GroupCoordinator interface. But
I think this is overly complicated the logic and may not worthwhile than
relying on users to specify unique member ids.

3. Minor: you would need to bumping up the version of JoinGroupResponse to
V4 as well.

4. Minor: in the wiki page, you need to specify the actual string value for
`MEMBER_ID`, for example "member.id".

5. When this additional config it specified by users, we should consider
setting the default of internal `LEAVE_GROUP_ON_CLOSE_CONFIG` to false,
since otherwise its effectiveness would be less.


Guozhang



On Thu, Jul 26, 2018 at 9:20 PM, Boyang Chen  wrote:

> Hey friends,
>
>
> I would like to open a discussion thread on KIP-345:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A
> +Reduce+multiple+consumer+rebalances+by+specifying+member+id
>
>
> This KIP is trying to resolve multiple rebalances by maintaining the
> consumer member id across rebalance generations. I have verified the theory
> on our internal Stream application, and it could reduce rebalance time to a
> few seconds when service is rolling restart.
>
>
> Let me know your thoughts, thank you!
>
>
> Best,
>
> Boyang
>



-- 
-- Guozhang

[DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

Hey friends,


I would like to open a discussion thread on KIP-345:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Reduce+multiple+consumer+rebalances+by+specifying+member+id


This KIP is trying to resolve multiple rebalances by maintaining the consumer 
member id across rebalance generations. I have verified the theory on our 
internal Stream application, and it could reduce rebalance time to a few 
seconds when service is rolling restart.


Let me know your thoughts, thank you!


Best,

Boyang