Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

[Boyang Chen]

Hey friends,

we will be closing this vote thread since we already got 3 binding votes 
(Guozhang, Harsha and Jason), and 2 non-binding votes (Stanislav, Mayuresh). 
Thanks again for everyone who have been actively contributing to this thread 
(Dong, Matthias, Colin, Mike, John, Konstantine), this has been a really 
exciting journey and I have learnt a lot through the whole process!

For next steps, we will further separate the KStream changes of exposing 
consumer info in a separate KIP, which will be led by Guozhang. I shall discuss 
with with Matthias for the rollout plan since he will be managing 2.2 release 
and hope to get some work done before the deadline. Feel free to keep raising 
comments or questions on the discussion thread, I will reply them at earliest 
convenience.

Best,
Boyang


From: Boyang Chen 
Sent: Saturday, January 5, 2019 8:58 AM
To: dev@kafka.apache.org; dev@kafka.apache.org
Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to reduce 
consumer rebalances

Thanks so much Harsha and Jason! Will address the comment and make it a tuple.

Get Outlook for iOS<https://aka.ms/o0ukef>


From: Jason Gustafson 
Sent: Friday, January 4, 2019 3:50 PM
To: dev
Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to reduce 
consumer rebalances

Hey Boyang,

I had just a follow-up to my comment above. I wanted to suggest an
alternative schema for LeaveGroup:

LeaveGroupRequest => GroupId [GroupInstanceId MemberId]

So we have a single array instead of two arrays. Each element identifies a
single member. For dynamic members, we would use GroupInstanceId="" and
provide the dynamic MemberId, which is consistent with JoinGroup. For
static members, GroupInstanceId must be provided and Member could be
considered optional. I think this makes the schema more coherent, but I'll
leave it to you if there is a good reason to keep them separate.

In any case, my vote is +1. Thanks for the hard work on this KIP!

Best,
Jason

On Fri, Jan 4, 2019 at 1:09 PM Boyang Chen  wrote:

> Thanks Guozhang for the proposal! The update is done.
>
> 
> From: Guozhang Wang 
> Sent: Saturday, January 5, 2019 3:33 AM
> To: dev
> Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> reduce consumer rebalances
>
> Hello Boyang,
>
> I've made another pass on the wiki page again. One minor comment is on the
> "Server Behavior Changes" section, we should have a paragraph on the
> logical changes on handling new versions of LeaveGroupRequest (e.g. how to
> handle dynamic member v.s. static member etc).
>
> Other than that, I do not have further comments. I think we can continue
> the voting process after that.
>
> Guozhang
>
> On Wed, Jan 2, 2019 at 10:00 AM Boyang Chen  wrote:
>
> > Thanks Jason for the comment! I answered it on the discuss thread.
> >
> > Folks, could we continue the vote for this KIP? This is a very critical
> > improvement for our streaming system
> > stability and we need to get things rolling right at the start of 2019.
> >
> > Thank you for your time!
> > Boyang
> >
> > ________
> > From: Jason Gustafson 
> > Sent: Tuesday, December 18, 2018 7:40 AM
> > To: dev
> > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> > reduce consumer rebalances
> >
> > Hi Boyang,
> >
> > Thanks, the KIP looks good. Just one comment.
> >
> > The new schema for the LeaveGroup request is slightly odd since it is
> > handling both the single consumer use case and the administrative use
> case.
> > I wonder we could make it consistent from a batching perspective.
> >
> > In other words, instead of this:
> > LeaveGroupRequest => GroupId MemberId [GroupInstanceId]
> >
> > Maybe we could do this:
> > LeaveGroupRequest => GroupId [GroupInstanceId MemberId]
> >
> > For dynamic members, GroupInstanceId could be empty, which is consistent
> > with JoinGroup. What do you think?
> >
> > Also, just for clarification, what is the expected behavior if the
> current
> > memberId of a static member is passed to LeaveGroup? Will the static
> member
> > be removed? I know the consumer will not do this, but we'll still have to
> > handle the case on the broker.
> >
> > Best,
> > Jason
> >
> >
> > On Mon, Dec 10, 2018 at 11:54 PM Boyang Chen 
> wrote:
> >
> > > Thanks Stanislav！
> > >
> > > Get Outlook for iOS<https://aka.ms/o0ukef>
> > >
> > > 
> > > From: Stanislav Kozlovski 
> > > Sent: Mon

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

[Boyang Chen]

Thanks so much Harsha and Jason! Will address the comment and make it a tuple.

Get Outlook for iOS<https://aka.ms/o0ukef>


From: Jason Gustafson 
Sent: Friday, January 4, 2019 3:50 PM
To: dev
Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to reduce 
consumer rebalances

Hey Boyang,

I had just a follow-up to my comment above. I wanted to suggest an
alternative schema for LeaveGroup:

LeaveGroupRequest => GroupId [GroupInstanceId MemberId]

So we have a single array instead of two arrays. Each element identifies a
single member. For dynamic members, we would use GroupInstanceId="" and
provide the dynamic MemberId, which is consistent with JoinGroup. For
static members, GroupInstanceId must be provided and Member could be
considered optional. I think this makes the schema more coherent, but I'll
leave it to you if there is a good reason to keep them separate.

In any case, my vote is +1. Thanks for the hard work on this KIP!

Best,
Jason

On Fri, Jan 4, 2019 at 1:09 PM Boyang Chen  wrote:

> Thanks Guozhang for the proposal! The update is done.
>
> 
> From: Guozhang Wang 
> Sent: Saturday, January 5, 2019 3:33 AM
> To: dev
> Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> reduce consumer rebalances
>
> Hello Boyang,
>
> I've made another pass on the wiki page again. One minor comment is on the
> "Server Behavior Changes" section, we should have a paragraph on the
> logical changes on handling new versions of LeaveGroupRequest (e.g. how to
> handle dynamic member v.s. static member etc).
>
> Other than that, I do not have further comments. I think we can continue
> the voting process after that.
>
> Guozhang
>
> On Wed, Jan 2, 2019 at 10:00 AM Boyang Chen  wrote:
>
> > Thanks Jason for the comment! I answered it on the discuss thread.
> >
> > Folks, could we continue the vote for this KIP? This is a very critical
> > improvement for our streaming system
> > stability and we need to get things rolling right at the start of 2019.
> >
> > Thank you for your time!
> > Boyang
> >
> > ________
> > From: Jason Gustafson 
> > Sent: Tuesday, December 18, 2018 7:40 AM
> > To: dev
> > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> > reduce consumer rebalances
> >
> > Hi Boyang,
> >
> > Thanks, the KIP looks good. Just one comment.
> >
> > The new schema for the LeaveGroup request is slightly odd since it is
> > handling both the single consumer use case and the administrative use
> case.
> > I wonder we could make it consistent from a batching perspective.
> >
> > In other words, instead of this:
> > LeaveGroupRequest => GroupId MemberId [GroupInstanceId]
> >
> > Maybe we could do this:
> > LeaveGroupRequest => GroupId [GroupInstanceId MemberId]
> >
> > For dynamic members, GroupInstanceId could be empty, which is consistent
> > with JoinGroup. What do you think?
> >
> > Also, just for clarification, what is the expected behavior if the
> current
> > memberId of a static member is passed to LeaveGroup? Will the static
> member
> > be removed? I know the consumer will not do this, but we'll still have to
> > handle the case on the broker.
> >
> > Best,
> > Jason
> >
> >
> > On Mon, Dec 10, 2018 at 11:54 PM Boyang Chen 
> wrote:
> >
> > > Thanks Stanislav！
> > >
> > > Get Outlook for iOS<https://aka.ms/o0ukef>
> > >
> > > 
> > > From: Stanislav Kozlovski 
> > > Sent: Monday, December 10, 2018 11:28 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> > > reduce consumer rebalances
> > >
> > > This is great work, Boyang. Thank you very much.
> > >
> > > +1 (non-binding)
> > >
> > > On Mon, Dec 10, 2018 at 6:09 PM Boyang Chen 
> wrote:
> > >
> > > > Hey there, could I get more votes on this thread?
> > > >
> > > > Thanks for the vote from Mayuresh and Mike :)
> > > >
> > > > Best,
> > > > Boyang
> > > > 
> > > > From: Mayuresh Gharat 
> > > > Sent: Thursday, December 6, 2018 10:53 AM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> > > > reduce consumer rebalances
> > > >
> > > > +1 

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

[Jason Gustafson]

Hey Boyang,

I had just a follow-up to my comment above. I wanted to suggest an
alternative schema for LeaveGroup:

LeaveGroupRequest => GroupId [GroupInstanceId MemberId]

So we have a single array instead of two arrays. Each element identifies a
single member. For dynamic members, we would use GroupInstanceId="" and
provide the dynamic MemberId, which is consistent with JoinGroup. For
static members, GroupInstanceId must be provided and Member could be
considered optional. I think this makes the schema more coherent, but I'll
leave it to you if there is a good reason to keep them separate.

In any case, my vote is +1. Thanks for the hard work on this KIP!

Best,
Jason

On Fri, Jan 4, 2019 at 1:09 PM Boyang Chen  wrote:

> Thanks Guozhang for the proposal! The update is done.
>
> 
> From: Guozhang Wang 
> Sent: Saturday, January 5, 2019 3:33 AM
> To: dev
> Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> reduce consumer rebalances
>
> Hello Boyang,
>
> I've made another pass on the wiki page again. One minor comment is on the
> "Server Behavior Changes" section, we should have a paragraph on the
> logical changes on handling new versions of LeaveGroupRequest (e.g. how to
> handle dynamic member v.s. static member etc).
>
> Other than that, I do not have further comments. I think we can continue
> the voting process after that.
>
> Guozhang
>
> On Wed, Jan 2, 2019 at 10:00 AM Boyang Chen  wrote:
>
> > Thanks Jason for the comment! I answered it on the discuss thread.
> >
> > Folks, could we continue the vote for this KIP? This is a very critical
> > improvement for our streaming system
> > stability and we need to get things rolling right at the start of 2019.
> >
> > Thank you for your time!
> > Boyang
> >
> > ____________
> > From: Jason Gustafson 
> > Sent: Tuesday, December 18, 2018 7:40 AM
> > To: dev
> > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> > reduce consumer rebalances
> >
> > Hi Boyang,
> >
> > Thanks, the KIP looks good. Just one comment.
> >
> > The new schema for the LeaveGroup request is slightly odd since it is
> > handling both the single consumer use case and the administrative use
> case.
> > I wonder we could make it consistent from a batching perspective.
> >
> > In other words, instead of this:
> > LeaveGroupRequest => GroupId MemberId [GroupInstanceId]
> >
> > Maybe we could do this:
> > LeaveGroupRequest => GroupId [GroupInstanceId MemberId]
> >
> > For dynamic members, GroupInstanceId could be empty, which is consistent
> > with JoinGroup. What do you think?
> >
> > Also, just for clarification, what is the expected behavior if the
> current
> > memberId of a static member is passed to LeaveGroup? Will the static
> member
> > be removed? I know the consumer will not do this, but we'll still have to
> > handle the case on the broker.
> >
> > Best,
> > Jason
> >
> >
> > On Mon, Dec 10, 2018 at 11:54 PM Boyang Chen 
> wrote:
> >
> > > Thanks Stanislav！
> > >
> > > Get Outlook for iOS<https://aka.ms/o0ukef>
> > >
> > > 
> > > From: Stanislav Kozlovski 
> > > Sent: Monday, December 10, 2018 11:28 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> > > reduce consumer rebalances
> > >
> > > This is great work, Boyang. Thank you very much.
> > >
> > > +1 (non-binding)
> > >
> > > On Mon, Dec 10, 2018 at 6:09 PM Boyang Chen 
> wrote:
> > >
> > > > Hey there, could I get more votes on this thread?
> > > >
> > > > Thanks for the vote from Mayuresh and Mike :)
> > > >
> > > > Best,
> > > > Boyang
> > > > 
> > > > From: Mayuresh Gharat 
> > > > Sent: Thursday, December 6, 2018 10:53 AM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> > > > reduce consumer rebalances
> > > >
> > > > +1 (non-binding)
> > > >
> > > > Thanks,
> > > >
> > > > Mayuresh
> > > >
> > > > On Tue, Dec 4, 2018 at 6:58 AM Mike Freyberger <
> > > mike.freyber...@xandr.com>
> > > > wrote:
> > > >
> > > > > +1 (

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

[Boyang Chen]

Thanks Guozhang for the proposal! The update is done.


From: Guozhang Wang 
Sent: Saturday, January 5, 2019 3:33 AM
To: dev
Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to reduce 
consumer rebalances

Hello Boyang,

I've made another pass on the wiki page again. One minor comment is on the
"Server Behavior Changes" section, we should have a paragraph on the
logical changes on handling new versions of LeaveGroupRequest (e.g. how to
handle dynamic member v.s. static member etc).

Other than that, I do not have further comments. I think we can continue
the voting process after that.

Guozhang

On Wed, Jan 2, 2019 at 10:00 AM Boyang Chen  wrote:

> Thanks Jason for the comment! I answered it on the discuss thread.
>
> Folks, could we continue the vote for this KIP? This is a very critical
> improvement for our streaming system
> stability and we need to get things rolling right at the start of 2019.
>
> Thank you for your time!
> Boyang
>
> 
> From: Jason Gustafson 
> Sent: Tuesday, December 18, 2018 7:40 AM
> To: dev
> Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> reduce consumer rebalances
>
> Hi Boyang,
>
> Thanks, the KIP looks good. Just one comment.
>
> The new schema for the LeaveGroup request is slightly odd since it is
> handling both the single consumer use case and the administrative use case.
> I wonder we could make it consistent from a batching perspective.
>
> In other words, instead of this:
> LeaveGroupRequest => GroupId MemberId [GroupInstanceId]
>
> Maybe we could do this:
> LeaveGroupRequest => GroupId [GroupInstanceId MemberId]
>
> For dynamic members, GroupInstanceId could be empty, which is consistent
> with JoinGroup. What do you think?
>
> Also, just for clarification, what is the expected behavior if the current
> memberId of a static member is passed to LeaveGroup? Will the static member
> be removed? I know the consumer will not do this, but we'll still have to
> handle the case on the broker.
>
> Best,
> Jason
>
>
> On Mon, Dec 10, 2018 at 11:54 PM Boyang Chen  wrote:
>
> > Thanks Stanislav！
> >
> > Get Outlook for iOS<https://aka.ms/o0ukef>
> >
> > ________
> > From: Stanislav Kozlovski 
> > Sent: Monday, December 10, 2018 11:28 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> > reduce consumer rebalances
> >
> > This is great work, Boyang. Thank you very much.
> >
> > +1 (non-binding)
> >
> > On Mon, Dec 10, 2018 at 6:09 PM Boyang Chen  wrote:
> >
> > > Hey there, could I get more votes on this thread?
> > >
> > > Thanks for the vote from Mayuresh and Mike :)
> > >
> > > Best,
> > > Boyang
> > > 
> > > From: Mayuresh Gharat 
> > > Sent: Thursday, December 6, 2018 10:53 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> > > reduce consumer rebalances
> > >
> > > +1 (non-binding)
> > >
> > > Thanks,
> > >
> > > Mayuresh
> > >
> > > On Tue, Dec 4, 2018 at 6:58 AM Mike Freyberger <
> > mike.freyber...@xandr.com>
> > > wrote:
> > >
> > > > +1 (non binding)
> > > >
> > > > On 12/4/18, 9:43 AM, "Patrick Williams" <
> > patrick.willi...@storageos.com
> > > >
> > > > wrote:
> > > >
> > > > Pls take me off this VOTE list
> > > >
> > > > Best,
> > > >
> > > > Patrick Williams
> > > >
> > > > Sales Manager, UK & Ireland, Nordics & Israel
> > > > StorageOS
> > > > +44 (0)7549 676279
> > > > patrick.willi...@storageos.com
> > > >
> > > > 20 Midtown
> > > > 20 Proctor Street
> > > > Holborn
> > > > London WC1V 6NX
> > > >
> > > > Twitter: @patch37
> > > > LinkedIn: linkedin.com/in/patrickwilliams4 <
> > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flinkedin.com%2Fin%2Fpatrickwilliams4data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435%7C1%7C0%7C636801101252994092sdata=ipDTX%2FGARrFkwZfRuOY0M5m3iJ%2Bnkxovv6u9bBDaTyc%3Dreserved=0
> > > >
> > > >
> > > >
> > >
> >
> https://nam02.safelinks.pr

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

[Guozhang Wang]

Hello Boyang,

I've made another pass on the wiki page again. One minor comment is on the
"Server Behavior Changes" section, we should have a paragraph on the
logical changes on handling new versions of LeaveGroupRequest (e.g. how to
handle dynamic member v.s. static member etc).

Other than that, I do not have further comments. I think we can continue
the voting process after that.

Guozhang

On Wed, Jan 2, 2019 at 10:00 AM Boyang Chen  wrote:

> Thanks Jason for the comment! I answered it on the discuss thread.
>
> Folks, could we continue the vote for this KIP? This is a very critical
> improvement for our streaming system
> stability and we need to get things rolling right at the start of 2019.
>
> Thank you for your time!
> Boyang
>
> 
> From: Jason Gustafson 
> Sent: Tuesday, December 18, 2018 7:40 AM
> To: dev
> Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> reduce consumer rebalances
>
> Hi Boyang,
>
> Thanks, the KIP looks good. Just one comment.
>
> The new schema for the LeaveGroup request is slightly odd since it is
> handling both the single consumer use case and the administrative use case.
> I wonder we could make it consistent from a batching perspective.
>
> In other words, instead of this:
> LeaveGroupRequest => GroupId MemberId [GroupInstanceId]
>
> Maybe we could do this:
> LeaveGroupRequest => GroupId [GroupInstanceId MemberId]
>
> For dynamic members, GroupInstanceId could be empty, which is consistent
> with JoinGroup. What do you think?
>
> Also, just for clarification, what is the expected behavior if the current
> memberId of a static member is passed to LeaveGroup? Will the static member
> be removed? I know the consumer will not do this, but we'll still have to
> handle the case on the broker.
>
> Best,
> Jason
>
>
> On Mon, Dec 10, 2018 at 11:54 PM Boyang Chen  wrote:
>
> > Thanks Stanislav！
> >
> > Get Outlook for iOS<https://aka.ms/o0ukef>
> >
> > ________
> > From: Stanislav Kozlovski 
> > Sent: Monday, December 10, 2018 11:28 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> > reduce consumer rebalances
> >
> > This is great work, Boyang. Thank you very much.
> >
> > +1 (non-binding)
> >
> > On Mon, Dec 10, 2018 at 6:09 PM Boyang Chen  wrote:
> >
> > > Hey there, could I get more votes on this thread?
> > >
> > > Thanks for the vote from Mayuresh and Mike :)
> > >
> > > Best,
> > > Boyang
> > > 
> > > From: Mayuresh Gharat 
> > > Sent: Thursday, December 6, 2018 10:53 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> > > reduce consumer rebalances
> > >
> > > +1 (non-binding)
> > >
> > > Thanks,
> > >
> > > Mayuresh
> > >
> > > On Tue, Dec 4, 2018 at 6:58 AM Mike Freyberger <
> > mike.freyber...@xandr.com>
> > > wrote:
> > >
> > > > +1 (non binding)
> > > >
> > > > On 12/4/18, 9:43 AM, "Patrick Williams" <
> > patrick.willi...@storageos.com
> > > >
> > > > wrote:
> > > >
> > > > Pls take me off this VOTE list
> > > >
> > > > Best,
> > > >
> > > > Patrick Williams
> > > >
> > > > Sales Manager, UK & Ireland, Nordics & Israel
> > > > StorageOS
> > > > +44 (0)7549 676279
> > > > patrick.willi...@storageos.com
> > > >
> > > > 20 Midtown
> > > > 20 Proctor Street
> > > > Holborn
> > > > London WC1V 6NX
> > > >
> > > > Twitter: @patch37
> > > > LinkedIn: linkedin.com/in/patrickwilliams4 <
> > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flinkedin.com%2Fin%2Fpatrickwilliams4data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435%7C1%7C0%7C636801101252994092sdata=ipDTX%2FGARrFkwZfRuOY0M5m3iJ%2Bnkxovv6u9bBDaTyc%3Dreserved=0
> > > >
> > > >
> > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslack.storageos.com%2Fdata=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435%7C1%7C0%7C636801101252994092sdata=hxuKU6aZdQU%2FpxpqaaThR6IjpEmwIP5%2F3NhYzMYijkw%3Dreserved=0
> > >

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

[Harsha]

+1 (binding).

Thanks,
Harsha

On Wed, Jan 2, 2019, at 9:59 AM, Boyang Chen wrote:
> Thanks Jason for the comment! I answered it on the discuss thread.
> 
> Folks, could we continue the vote for this KIP? This is a very critical 
> improvement for our streaming system
> stability and we need to get things rolling right at the start of 2019.
> 
> Thank you for your time!
> Boyang
> 
> 
> From: Jason Gustafson 
> Sent: Tuesday, December 18, 2018 7:40 AM
> To: dev
> Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to 
> reduce consumer rebalances
> 
> Hi Boyang,
> 
> Thanks, the KIP looks good. Just one comment.
> 
> The new schema for the LeaveGroup request is slightly odd since it is
> handling both the single consumer use case and the administrative use case.
> I wonder we could make it consistent from a batching perspective.
> 
> In other words, instead of this:
> LeaveGroupRequest => GroupId MemberId [GroupInstanceId]
> 
> Maybe we could do this:
> LeaveGroupRequest => GroupId [GroupInstanceId MemberId]
> 
> For dynamic members, GroupInstanceId could be empty, which is consistent
> with JoinGroup. What do you think?
> 
> Also, just for clarification, what is the expected behavior if the current
> memberId of a static member is passed to LeaveGroup? Will the static member
> be removed? I know the consumer will not do this, but we'll still have to
> handle the case on the broker.
> 
> Best,
> Jason
> 
> 
> On Mon, Dec 10, 2018 at 11:54 PM Boyang Chen  wrote:
> 
> > Thanks Stanislav！
> >
> > Get Outlook for iOS<https://aka.ms/o0ukef>
> >
> > ____________
> > From: Stanislav Kozlovski 
> > Sent: Monday, December 10, 2018 11:28 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> > reduce consumer rebalances
> >
> > This is great work, Boyang. Thank you very much.
> >
> > +1 (non-binding)
> >
> > On Mon, Dec 10, 2018 at 6:09 PM Boyang Chen  wrote:
> >
> > > Hey there, could I get more votes on this thread?
> > >
> > > Thanks for the vote from Mayuresh and Mike :)
> > >
> > > Best,
> > > Boyang
> > > 
> > > From: Mayuresh Gharat 
> > > Sent: Thursday, December 6, 2018 10:53 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> > > reduce consumer rebalances
> > >
> > > +1 (non-binding)
> > >
> > > Thanks,
> > >
> > > Mayuresh
> > >
> > > On Tue, Dec 4, 2018 at 6:58 AM Mike Freyberger <
> > mike.freyber...@xandr.com>
> > > wrote:
> > >
> > > > +1 (non binding)
> > > >
> > > > On 12/4/18, 9:43 AM, "Patrick Williams" <
> > patrick.willi...@storageos.com
> > > >
> > > > wrote:
> > > >
> > > > Pls take me off this VOTE list
> > > >
> > > > Best,
> > > >
> > > > Patrick Williams
> > > >
> > > > Sales Manager, UK & Ireland, Nordics & Israel
> > > > StorageOS
> > > > +44 (0)7549 676279
> > > > patrick.willi...@storageos.com
> > > >
> > > > 20 Midtown
> > > > 20 Proctor Street
> > > > Holborn
> > > > London WC1V 6NX
> > > >
> > > > Twitter: @patch37
> > > > LinkedIn: linkedin.com/in/patrickwilliams4 <
> > > >
> > >
> > https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flinkedin.com%2Fin%2Fpatrickwilliams4data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435%7C1%7C0%7C636801101252994092sdata=ipDTX%2FGARrFkwZfRuOY0M5m3iJ%2Bnkxovv6u9bBDaTyc%3Dreserved=0
> > > >
> > > >
> > > >
> > >
> > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslack.storageos.com%2Fdata=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435%7C1%7C0%7C636801101252994092sdata=hxuKU6aZdQU%2FpxpqaaThR6IjpEmwIP5%2F3NhYzMYijkw%3Dreserved=0
> > > >
> > > >
> > > >
> > > > On 03/12/2018, 17:34, "Guozhang Wang"  wrote:
> > > >
> > > > Hello Boyang,
> > > >
> > > > I've browsed through the new wiki and there are still a couple of
> > > > minor
> > > > things to notice:
> > > >
>

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

[Boyang Chen]

Thanks Jason for the comment! I answered it on the discuss thread.

Folks, could we continue the vote for this KIP? This is a very critical 
improvement for our streaming system
stability and we need to get things rolling right at the start of 2019.

Thank you for your time!
Boyang


From: Jason Gustafson 
Sent: Tuesday, December 18, 2018 7:40 AM
To: dev
Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to reduce 
consumer rebalances

Hi Boyang,

Thanks, the KIP looks good. Just one comment.

The new schema for the LeaveGroup request is slightly odd since it is
handling both the single consumer use case and the administrative use case.
I wonder we could make it consistent from a batching perspective.

In other words, instead of this:
LeaveGroupRequest => GroupId MemberId [GroupInstanceId]

Maybe we could do this:
LeaveGroupRequest => GroupId [GroupInstanceId MemberId]

For dynamic members, GroupInstanceId could be empty, which is consistent
with JoinGroup. What do you think?

Also, just for clarification, what is the expected behavior if the current
memberId of a static member is passed to LeaveGroup? Will the static member
be removed? I know the consumer will not do this, but we'll still have to
handle the case on the broker.

Best,
Jason


On Mon, Dec 10, 2018 at 11:54 PM Boyang Chen  wrote:

> Thanks Stanislav！
>
> Get Outlook for iOS<https://aka.ms/o0ukef>
>
> 
> From: Stanislav Kozlovski 
> Sent: Monday, December 10, 2018 11:28 PM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> reduce consumer rebalances
>
> This is great work, Boyang. Thank you very much.
>
> +1 (non-binding)
>
> On Mon, Dec 10, 2018 at 6:09 PM Boyang Chen  wrote:
>
> > Hey there, could I get more votes on this thread?
> >
> > Thanks for the vote from Mayuresh and Mike :)
> >
> > Best,
> > Boyang
> > 
> > From: Mayuresh Gharat 
> > Sent: Thursday, December 6, 2018 10:53 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> > reduce consumer rebalances
> >
> > +1 (non-binding)
> >
> > Thanks,
> >
> > Mayuresh
> >
> > On Tue, Dec 4, 2018 at 6:58 AM Mike Freyberger <
> mike.freyber...@xandr.com>
> > wrote:
> >
> > > +1 (non binding)
> > >
> > > On 12/4/18, 9:43 AM, "Patrick Williams" <
> patrick.willi...@storageos.com
> > >
> > > wrote:
> > >
> > > Pls take me off this VOTE list
> > >
> > > Best,
> > >
> > > Patrick Williams
> > >
> > > Sales Manager, UK & Ireland, Nordics & Israel
> > > StorageOS
> > > +44 (0)7549 676279
> > > patrick.willi...@storageos.com
> > >
> > > 20 Midtown
> > > 20 Proctor Street
> > > Holborn
> > > London WC1V 6NX
> > >
> > > Twitter: @patch37
> > > LinkedIn: linkedin.com/in/patrickwilliams4 <
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flinkedin.com%2Fin%2Fpatrickwilliams4data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435%7C1%7C0%7C636801101252994092sdata=ipDTX%2FGARrFkwZfRuOY0M5m3iJ%2Bnkxovv6u9bBDaTyc%3Dreserved=0
> > >
> > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslack.storageos.com%2Fdata=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435%7C1%7C0%7C636801101252994092sdata=hxuKU6aZdQU%2FpxpqaaThR6IjpEmwIP5%2F3NhYzMYijkw%3Dreserved=0
> > >
> > >
> > >
> > > On 03/12/2018, 17:34, "Guozhang Wang"  wrote:
> > >
> > > Hello Boyang,
> > >
> > > I've browsed through the new wiki and there are still a couple of
> > > minor
> > > things to notice:
> > >
> > > 1. RemoveMemberFromGroupOptions seems not defined anywhere.
> > >
> > > 2. LeaveGroupRequest added a list of group instance id, but still
> > > keep the
> > > member id as a singleton; is that intentional? I think to make
> > the
> > > protocol
> > > consistent both member id and instance ids could be plural.
> > >
> > > 3. About the *kafka-remove-member-from-group.sh *tool, I'm
> > > wondering if we
> > > can defer adding this while just add the corresponding calls of
> > the
> > > LeaveGroupRequest inside Streams until we have used it in
> > > production and
> > > hence h

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

[Jason Gustafson]

Hi Boyang,

Thanks, the KIP looks good. Just one comment.

The new schema for the LeaveGroup request is slightly odd since it is
handling both the single consumer use case and the administrative use case.
I wonder we could make it consistent from a batching perspective.

In other words, instead of this:
LeaveGroupRequest => GroupId MemberId [GroupInstanceId]

Maybe we could do this:
LeaveGroupRequest => GroupId [GroupInstanceId MemberId]

For dynamic members, GroupInstanceId could be empty, which is consistent
with JoinGroup. What do you think?

Also, just for clarification, what is the expected behavior if the current
memberId of a static member is passed to LeaveGroup? Will the static member
be removed? I know the consumer will not do this, but we'll still have to
handle the case on the broker.

Best,
Jason


On Mon, Dec 10, 2018 at 11:54 PM Boyang Chen  wrote:

> Thanks Stanislav！
>
> Get Outlook for iOS<https://aka.ms/o0ukef>
>
> 
> From: Stanislav Kozlovski 
> Sent: Monday, December 10, 2018 11:28 PM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> reduce consumer rebalances
>
> This is great work, Boyang. Thank you very much.
>
> +1 (non-binding)
>
> On Mon, Dec 10, 2018 at 6:09 PM Boyang Chen  wrote:
>
> > Hey there, could I get more votes on this thread?
> >
> > Thanks for the vote from Mayuresh and Mike :)
> >
> > Best,
> > Boyang
> > 
> > From: Mayuresh Gharat 
> > Sent: Thursday, December 6, 2018 10:53 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> > reduce consumer rebalances
> >
> > +1 (non-binding)
> >
> > Thanks,
> >
> > Mayuresh
> >
> > On Tue, Dec 4, 2018 at 6:58 AM Mike Freyberger <
> mike.freyber...@xandr.com>
> > wrote:
> >
> > > +1 (non binding)
> > >
> > > On 12/4/18, 9:43 AM, "Patrick Williams" <
> patrick.willi...@storageos.com
> > >
> > > wrote:
> > >
> > > Pls take me off this VOTE list
> > >
> > > Best,
> > >
> > > Patrick Williams
> > >
> > > Sales Manager, UK & Ireland, Nordics & Israel
> > > StorageOS
> > > +44 (0)7549 676279
> > > patrick.willi...@storageos.com
> > >
> > > 20 Midtown
> > > 20 Proctor Street
> > > Holborn
> > > London WC1V 6NX
> > >
> > > Twitter: @patch37
> > > LinkedIn: linkedin.com/in/patrickwilliams4 <
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flinkedin.com%2Fin%2Fpatrickwilliams4data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435%7C1%7C0%7C636801101252994092sdata=ipDTX%2FGARrFkwZfRuOY0M5m3iJ%2Bnkxovv6u9bBDaTyc%3Dreserved=0
> > >
> > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslack.storageos.com%2Fdata=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435%7C1%7C0%7C636801101252994092sdata=hxuKU6aZdQU%2FpxpqaaThR6IjpEmwIP5%2F3NhYzMYijkw%3Dreserved=0
> > >
> > >
> > >
> > > On 03/12/2018, 17:34, "Guozhang Wang"  wrote:
> > >
> > > Hello Boyang,
> > >
> > > I've browsed through the new wiki and there are still a couple of
> > > minor
> > > things to notice:
> > >
> > > 1. RemoveMemberFromGroupOptions seems not defined anywhere.
> > >
> > > 2. LeaveGroupRequest added a list of group instance id, but still
> > > keep the
> > > member id as a singleton; is that intentional? I think to make
> > the
> > > protocol
> > > consistent both member id and instance ids could be plural.
> > >
> > > 3. About the *kafka-remove-member-from-group.sh *tool, I'm
> > > wondering if we
> > > can defer adding this while just add the corresponding calls of
> > the
> > > LeaveGroupRequest inside Streams until we have used it in
> > > production and
> > > hence have a better understanding on how flexible or extensible
> > if
> > > we want
> > > to add any cmd tools. The rationale is that if we do not
> > > necessarily need
> > > it now, we can always add it later with a more think-through API
> > > design,
> > > but if we add the tool in a rush, we may need to extend or modify
> > > it soon
> > > after we realize its limits in operations.
&

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

[Boyang Chen]

Thanks Stanislav！

Get Outlook for iOS<https://aka.ms/o0ukef>


From: Stanislav Kozlovski 
Sent: Monday, December 10, 2018 11:28 PM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to reduce 
consumer rebalances

This is great work, Boyang. Thank you very much.

+1 (non-binding)

On Mon, Dec 10, 2018 at 6:09 PM Boyang Chen  wrote:

> Hey there, could I get more votes on this thread?
>
> Thanks for the vote from Mayuresh and Mike :)
>
> Best,
> Boyang
> 
> From: Mayuresh Gharat 
> Sent: Thursday, December 6, 2018 10:53 AM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> reduce consumer rebalances
>
> +1 (non-binding)
>
> Thanks,
>
> Mayuresh
>
> On Tue, Dec 4, 2018 at 6:58 AM Mike Freyberger 
> wrote:
>
> > +1 (non binding)
> >
> > On 12/4/18, 9:43 AM, "Patrick Williams"  >
> > wrote:
> >
> > Pls take me off this VOTE list
> >
> > Best,
> >
> > Patrick Williams
> >
> > Sales Manager, UK & Ireland, Nordics & Israel
> > StorageOS
> > +44 (0)7549 676279
> > patrick.willi...@storageos.com
> >
> > 20 Midtown
> > 20 Proctor Street
> > Holborn
> > London WC1V 6NX
> >
> > Twitter: @patch37
> > LinkedIn: linkedin.com/in/patrickwilliams4 <
> >
> https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flinkedin.com%2Fin%2Fpatrickwilliams4data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435%7C1%7C0%7C636801101252994092sdata=ipDTX%2FGARrFkwZfRuOY0M5m3iJ%2Bnkxovv6u9bBDaTyc%3Dreserved=0
> >
> >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslack.storageos.com%2Fdata=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435%7C1%7C0%7C636801101252994092sdata=hxuKU6aZdQU%2FpxpqaaThR6IjpEmwIP5%2F3NhYzMYijkw%3Dreserved=0
> >
> >
> >
> > On 03/12/2018, 17:34, "Guozhang Wang"  wrote:
> >
> > Hello Boyang,
> >
> > I've browsed through the new wiki and there are still a couple of
> > minor
> > things to notice:
> >
> > 1. RemoveMemberFromGroupOptions seems not defined anywhere.
> >
> > 2. LeaveGroupRequest added a list of group instance id, but still
> > keep the
> > member id as a singleton; is that intentional? I think to make
> the
> > protocol
> > consistent both member id and instance ids could be plural.
> >
> > 3. About the *kafka-remove-member-from-group.sh *tool, I'm
> > wondering if we
> > can defer adding this while just add the corresponding calls of
> the
> > LeaveGroupRequest inside Streams until we have used it in
> > production and
> > hence have a better understanding on how flexible or extensible
> if
> > we want
> > to add any cmd tools. The rationale is that if we do not
> > necessarily need
> > it now, we can always add it later with a more think-through API
> > design,
> > but if we add the tool in a rush, we may need to extend or modify
> > it soon
> > after we realize its limits in operations.
> >
> > Otherwise, I'm +1 on the proposal.
> >
> > Guozhang
> >
> >
> > On Mon, Dec 3, 2018 at 9:14 AM Boyang Chen 
> > wrote:
> >
> > > Hey community friends,
> > >
> > > after another month of polishing, KIP-345<
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalancesdata=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435%7C1%7C0%7C636801101252994092sdata=T4i7L1i0nIeHrrjTeLOOgYKsfzfNEMGDhTazvBEZbXw%3Dreserved=0
> > >
> > > design is ready for vote. Feel free to add your comment on the
> > discussion
> > > thread or here.
> > >
> > > Thanks for your time!
> > >
> > > Boyang
> > > 
> > > From: Boyang Chen 
> > > Sent: Friday, November 9, 2018 6:35 AM
> > > To: dev@kafka.apache.org
> > > Subject: [VOTE] KIP-345: Introduce static membership protocol
> to
> > reduce
> > > consumer rebalances
> > >
> > > Hey all,
> > >
> > >
> > > thanks so much for all the inputs on KIP-345 so far. The
> > original proposal
> > > has enhanced a lot with your help. To make sure the
> > implementat

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

[Stanislav Kozlovski]

This is great work, Boyang. Thank you very much.

+1 (non-binding)

On Mon, Dec 10, 2018 at 6:09 PM Boyang Chen  wrote:

> Hey there, could I get more votes on this thread?
>
> Thanks for the vote from Mayuresh and Mike :)
>
> Best,
> Boyang
> 
> From: Mayuresh Gharat 
> Sent: Thursday, December 6, 2018 10:53 AM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> reduce consumer rebalances
>
> +1 (non-binding)
>
> Thanks,
>
> Mayuresh
>
> On Tue, Dec 4, 2018 at 6:58 AM Mike Freyberger 
> wrote:
>
> > +1 (non binding)
> >
> > On 12/4/18, 9:43 AM, "Patrick Williams"  >
> > wrote:
> >
> > Pls take me off this VOTE list
> >
> > Best,
> >
> > Patrick Williams
> >
> > Sales Manager, UK & Ireland, Nordics & Israel
> > StorageOS
> > +44 (0)7549 676279
> > patrick.willi...@storageos.com
> >
> > 20 Midtown
> > 20 Proctor Street
> > Holborn
> > London WC1V 6NX
> >
> > Twitter: @patch37
> > LinkedIn: linkedin.com/in/patrickwilliams4 <
> >
> https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flinkedin.com%2Fin%2Fpatrickwilliams4data=02%7C01%7C%7Ca9e7b092d1024e90c1d708d65b261e1e%7C84df9e7fe9f640afb435%7C1%7C0%7C636796616598061118sdata=QIZ7s9HoutiaKs4bAg68oNsUDZ9ertfwlHd%2FRWKRFOg%3Dreserved=0
> >
> >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslack.storageos.com%2Fdata=02%7C01%7C%7Ca9e7b092d1024e90c1d708d65b261e1e%7C84df9e7fe9f640afb435%7C1%7C0%7C636796616598061118sdata=lthlUwYKvWgxquV%2FJE%2FQF9pFYrMYPV1QK72I1mu8E%2BY%3Dreserved=0
> >
> >
> >
> > On 03/12/2018, 17:34, "Guozhang Wang"  wrote:
> >
> > Hello Boyang,
> >
> > I've browsed through the new wiki and there are still a couple of
> > minor
> > things to notice:
> >
> > 1. RemoveMemberFromGroupOptions seems not defined anywhere.
> >
> > 2. LeaveGroupRequest added a list of group instance id, but still
> > keep the
> > member id as a singleton; is that intentional? I think to make
> the
> > protocol
> > consistent both member id and instance ids could be plural.
> >
> > 3. About the *kafka-remove-member-from-group.sh *tool, I'm
> > wondering if we
> > can defer adding this while just add the corresponding calls of
> the
> > LeaveGroupRequest inside Streams until we have used it in
> > production and
> > hence have a better understanding on how flexible or extensible
> if
> > we want
> > to add any cmd tools. The rationale is that if we do not
> > necessarily need
> > it now, we can always add it later with a more think-through API
> > design,
> > but if we add the tool in a rush, we may need to extend or modify
> > it soon
> > after we realize its limits in operations.
> >
> > Otherwise, I'm +1 on the proposal.
> >
> > Guozhang
> >
> >
> > On Mon, Dec 3, 2018 at 9:14 AM Boyang Chen 
> > wrote:
> >
> > > Hey community friends,
> > >
> > > after another month of polishing, KIP-345<
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalancesdata=02%7C01%7C%7Ca9e7b092d1024e90c1d708d65b261e1e%7C84df9e7fe9f640afb435%7C1%7C0%7C636796616598061118sdata=L0X1z8hE%2FebB0KGbUWttz4lvsy%2FkcB49MRc8KZd8I0Y%3Dreserved=0
> > >
> > > design is ready for vote. Feel free to add your comment on the
> > discussion
> > > thread or here.
> > >
> > > Thanks for your time!
> > >
> > > Boyang
> > > 
> > > From: Boyang Chen 
> > > Sent: Friday, November 9, 2018 6:35 AM
> > > To: dev@kafka.apache.org
> > > Subject: [VOTE] KIP-345: Introduce static membership protocol
> to
> > reduce
> > > consumer rebalances
> > >
> > > Hey all,
> > >
> > >
> > > thanks so much for all the inputs on KIP-345 so far. The
> > original proposal
> > 

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

[Boyang Chen]

Hey there, could I get more votes on this thread?

Thanks for the vote from Mayuresh and Mike :)

Best,
Boyang

From: Mayuresh Gharat 
Sent: Thursday, December 6, 2018 10:53 AM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to reduce 
consumer rebalances

+1 (non-binding)

Thanks,

Mayuresh

On Tue, Dec 4, 2018 at 6:58 AM Mike Freyberger 
wrote:

> +1 (non binding)
>
> On 12/4/18, 9:43 AM, "Patrick Williams" 
> wrote:
>
> Pls take me off this VOTE list
>
> Best,
>
> Patrick Williams
>
> Sales Manager, UK & Ireland, Nordics & Israel
> StorageOS
> +44 (0)7549 676279
> patrick.willi...@storageos.com
>
> 20 Midtown
> 20 Proctor Street
> Holborn
> London WC1V 6NX
>
> Twitter: @patch37
> LinkedIn: linkedin.com/in/patrickwilliams4 <
> https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flinkedin.com%2Fin%2Fpatrickwilliams4data=02%7C01%7C%7Ca9e7b092d1024e90c1d708d65b261e1e%7C84df9e7fe9f640afb435%7C1%7C0%7C636796616598061118sdata=QIZ7s9HoutiaKs4bAg68oNsUDZ9ertfwlHd%2FRWKRFOg%3Dreserved=0>
>
> 
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslack.storageos.com%2Fdata=02%7C01%7C%7Ca9e7b092d1024e90c1d708d65b261e1e%7C84df9e7fe9f640afb435%7C1%7C0%7C636796616598061118sdata=lthlUwYKvWgxquV%2FJE%2FQF9pFYrMYPV1QK72I1mu8E%2BY%3Dreserved=0
>
>
>
> On 03/12/2018, 17:34, "Guozhang Wang"  wrote:
>
> Hello Boyang,
>
> I've browsed through the new wiki and there are still a couple of
> minor
> things to notice:
>
> 1. RemoveMemberFromGroupOptions seems not defined anywhere.
>
> 2. LeaveGroupRequest added a list of group instance id, but still
> keep the
> member id as a singleton; is that intentional? I think to make the
> protocol
> consistent both member id and instance ids could be plural.
>
> 3. About the *kafka-remove-member-from-group.sh *tool, I'm
> wondering if we
> can defer adding this while just add the corresponding calls of the
> LeaveGroupRequest inside Streams until we have used it in
> production and
> hence have a better understanding on how flexible or extensible if
> we want
> to add any cmd tools. The rationale is that if we do not
> necessarily need
> it now, we can always add it later with a more think-through API
> design,
> but if we add the tool in a rush, we may need to extend or modify
> it soon
> after we realize its limits in operations.
>
> Otherwise, I'm +1 on the proposal.
>
> Guozhang
>
>
> On Mon, Dec 3, 2018 at 9:14 AM Boyang Chen 
> wrote:
>
> > Hey community friends,
> >
> > after another month of polishing, KIP-345<
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalancesdata=02%7C01%7C%7Ca9e7b092d1024e90c1d708d65b261e1e%7C84df9e7fe9f640afb435%7C1%7C0%7C636796616598061118sdata=L0X1z8hE%2FebB0KGbUWttz4lvsy%2FkcB49MRc8KZd8I0Y%3Dreserved=0
> >
> > design is ready for vote. Feel free to add your comment on the
> discussion
> > thread or here.
> >
> > Thanks for your time!
> >
> > Boyang
> > 
> > From: Boyang Chen 
> > Sent: Friday, November 9, 2018 6:35 AM
> > To: dev@kafka.apache.org
> > Subject: [VOTE] KIP-345: Introduce static membership protocol to
> reduce
> > consumer rebalances
> >
> > Hey all,
> >
> >
> > thanks so much for all the inputs on KIP-345 so far. The
> original proposal
> > has enhanced a lot with your help. To make sure the
> implementation go
> > smoothly without back and forth, I would like to start a vote on
> the final
> > design agreement now:
> >
> > 
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-data=02%7C01%7C%7Ca9e7b092d1024e90c1d708d65b261e1e%7C84df9e7fe9f640afb435%7C1%7C0%7C636796616598061118sdata=Ld0DpCbOmH0Gmu%2FVfkRS5lWA0vBcgi9WmHDvYz4L3b8%3Dreserved=0<
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bst

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

[Mayuresh Gharat]

+1 (non-binding)

Thanks,

Mayuresh

On Tue, Dec 4, 2018 at 6:58 AM Mike Freyberger 
wrote:

> +1 (non binding)
>
> On 12/4/18, 9:43 AM, "Patrick Williams" 
> wrote:
>
> Pls take me off this VOTE list
>
> Best,
>
> Patrick Williams
>
> Sales Manager, UK & Ireland, Nordics & Israel
> StorageOS
> +44 (0)7549 676279
> patrick.willi...@storageos.com
>
> 20 Midtown
> 20 Proctor Street
> Holborn
> London WC1V 6NX
>
> Twitter: @patch37
> LinkedIn: linkedin.com/in/patrickwilliams4 <
> http://linkedin.com/in/patrickwilliams4>
>
> https://slack.storageos.com/
>
>
>
> On 03/12/2018, 17:34, "Guozhang Wang"  wrote:
>
> Hello Boyang,
>
> I've browsed through the new wiki and there are still a couple of
> minor
> things to notice:
>
> 1. RemoveMemberFromGroupOptions seems not defined anywhere.
>
> 2. LeaveGroupRequest added a list of group instance id, but still
> keep the
> member id as a singleton; is that intentional? I think to make the
> protocol
> consistent both member id and instance ids could be plural.
>
> 3. About the *kafka-remove-member-from-group.sh *tool, I'm
> wondering if we
> can defer adding this while just add the corresponding calls of the
> LeaveGroupRequest inside Streams until we have used it in
> production and
> hence have a better understanding on how flexible or extensible if
> we want
> to add any cmd tools. The rationale is that if we do not
> necessarily need
> it now, we can always add it later with a more think-through API
> design,
> but if we add the tool in a rush, we may need to extend or modify
> it soon
> after we realize its limits in operations.
>
> Otherwise, I'm +1 on the proposal.
>
> Guozhang
>
>
> On Mon, Dec 3, 2018 at 9:14 AM Boyang Chen 
> wrote:
>
> > Hey community friends,
> >
> > after another month of polishing, KIP-345<
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances
> >
> > design is ready for vote. Feel free to add your comment on the
> discussion
> > thread or here.
> >
> > Thanks for your time!
> >
> > Boyang
> > 
> > From: Boyang Chen 
> > Sent: Friday, November 9, 2018 6:35 AM
> > To: dev@kafka.apache.org
> > Subject: [VOTE] KIP-345: Introduce static membership protocol to
> reduce
> > consumer rebalances
> >
> > Hey all,
> >
> >
> > thanks so much for all the inputs on KIP-345 so far. The
> original proposal
> > has enhanced a lot with your help. To make sure the
> implementation go
> > smoothly without back and forth, I would like to start a vote on
> the final
> > design agreement now:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-<
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances
> > >
> >
> >
> 345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances<
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances
> > >
> >
> > KIP-345: Introduce static membership protocol to reduce ...<
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances
> > >
> > cwiki.apache.org
> > For stateful applications, one of the biggest performance
> bottleneck is
> > the state shuffling. In Kafka consumer, there is a concept called
> > "rebalance" which means that for given M partitions and N
> consumers in one
> > consumer group, Kafka will try to balance the load between
> consumers and
> > ideally have ...
> >
> >
> > Let me know if you have any questions.
> >
> >
> > Best,
> >
> > Boyang
> >
> >
>
> --
> -- Guozhang
>
>
>
>
>

-- 
-Regards,
Mayuresh R. Gharat
(862) 250-7125

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

[Mike Freyberger]

+1 (non binding)

On 12/4/18, 9:43 AM, "Patrick Williams"  wrote:

Pls take me off this VOTE list

Best,
 
Patrick Williams
 
Sales Manager, UK & Ireland, Nordics & Israel
StorageOS
+44 (0)7549 676279
patrick.willi...@storageos.com
 
20 Midtown
20 Proctor Street
Holborn
London WC1V 6NX
 
Twitter: @patch37
LinkedIn: linkedin.com/in/patrickwilliams4 

 
https://slack.storageos.com/
 
 

On 03/12/2018, 17:34, "Guozhang Wang"  wrote:

Hello Boyang,

I've browsed through the new wiki and there are still a couple of minor
things to notice:

1. RemoveMemberFromGroupOptions seems not defined anywhere.

2. LeaveGroupRequest added a list of group instance id, but still keep 
the
member id as a singleton; is that intentional? I think to make the 
protocol
consistent both member id and instance ids could be plural.

3. About the *kafka-remove-member-from-group.sh *tool, I'm wondering if 
we
can defer adding this while just add the corresponding calls of the
LeaveGroupRequest inside Streams until we have used it in production and
hence have a better understanding on how flexible or extensible if we 
want
to add any cmd tools. The rationale is that if we do not necessarily 
need
it now, we can always add it later with a more think-through API design,
but if we add the tool in a rush, we may need to extend or modify it 
soon
after we realize its limits in operations.

Otherwise, I'm +1 on the proposal.

Guozhang


On Mon, Dec 3, 2018 at 9:14 AM Boyang Chen  wrote:

> Hey community friends,
>
> after another month of polishing, KIP-345<
> 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances>
> design is ready for vote. Feel free to add your comment on the 
discussion
> thread or here.
>
> Thanks for your time!
>
> Boyang
> 
> From: Boyang Chen 
> Sent: Friday, November 9, 2018 6:35 AM
> To: dev@kafka.apache.org
> Subject: [VOTE] KIP-345: Introduce static membership protocol to 
reduce
> consumer rebalances
>
> Hey all,
>
>
> thanks so much for all the inputs on KIP-345 so far. The original 
proposal
> has enhanced a lot with your help. To make sure the implementation go
> smoothly without back and forth, I would like to start a vote on the 
final
> design agreement now:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-<
> 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances
> >
>
> 
345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances<
> 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances
> >
>
> KIP-345: Introduce static membership protocol to reduce ...<
> 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances
> >
> cwiki.apache.org
> For stateful applications, one of the biggest performance bottleneck 
is
> the state shuffling. In Kafka consumer, there is a concept called
> "rebalance" which means that for given M partitions and N consumers 
in one
> consumer group, Kafka will try to balance the load between consumers 
and
> ideally have ...
>
>
> Let me know if you have any questions.
>
>
> Best,
>
> Boyang
>
>

-- 
-- Guozhang

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

[Patrick Williams]

Pls take me off this VOTE list

Best,
 
Patrick Williams
 
Sales Manager, UK & Ireland, Nordics & Israel
StorageOS
+44 (0)7549 676279
patrick.willi...@storageos.com
 
20 Midtown
20 Proctor Street
Holborn
London WC1V 6NX
 
Twitter: @patch37
LinkedIn: linkedin.com/in/patrickwilliams4 

 
https://slack.storageos.com/
 
 

On 03/12/2018, 17:34, "Guozhang Wang"  wrote:

Hello Boyang,

I've browsed through the new wiki and there are still a couple of minor
things to notice:

1. RemoveMemberFromGroupOptions seems not defined anywhere.

2. LeaveGroupRequest added a list of group instance id, but still keep the
member id as a singleton; is that intentional? I think to make the protocol
consistent both member id and instance ids could be plural.

3. About the *kafka-remove-member-from-group.sh *tool, I'm wondering if we
can defer adding this while just add the corresponding calls of the
LeaveGroupRequest inside Streams until we have used it in production and
hence have a better understanding on how flexible or extensible if we want
to add any cmd tools. The rationale is that if we do not necessarily need
it now, we can always add it later with a more think-through API design,
but if we add the tool in a rush, we may need to extend or modify it soon
after we realize its limits in operations.

Otherwise, I'm +1 on the proposal.

Guozhang


On Mon, Dec 3, 2018 at 9:14 AM Boyang Chen  wrote:

> Hey community friends,
>
> after another month of polishing, KIP-345<
> 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances>
> design is ready for vote. Feel free to add your comment on the discussion
> thread or here.
>
> Thanks for your time!
>
> Boyang
> 
> From: Boyang Chen 
> Sent: Friday, November 9, 2018 6:35 AM
> To: dev@kafka.apache.org
> Subject: [VOTE] KIP-345: Introduce static membership protocol to reduce
> consumer rebalances
>
> Hey all,
>
>
> thanks so much for all the inputs on KIP-345 so far. The original proposal
> has enhanced a lot with your help. To make sure the implementation go
> smoothly without back and forth, I would like to start a vote on the final
> design agreement now:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-<
> 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances
> >
>
> 345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances<
> 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances
> >
>
> KIP-345: Introduce static membership protocol to reduce ...<
> 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances
> >
> cwiki.apache.org
> For stateful applications, one of the biggest performance bottleneck is
> the state shuffling. In Kafka consumer, there is a concept called
> "rebalance" which means that for given M partitions and N consumers in one
> consumer group, Kafka will try to balance the load between consumers and
> ideally have ...
>
>
> Let me know if you have any questions.
>
>
> Best,
>
> Boyang
>
>

-- 
-- Guozhang

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

[Boyang Chen]

Thanks Guozhang for confirming！Let‘s move followup back to the discussion 
thread. Would appreciate a +1 from you when you feel this is ready :)

Get Outlook for iOS<https://aka.ms/o0ukef>

From: Guozhang Wang 
Sent: Tuesday, December 4, 2018 7:21:41 AM
To: dev
Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to reduce 
consumer rebalances

2. That's a good point. I think I'm convinced.


Guozhang

On Mon, Dec 3, 2018 at 5:43 PM Boyang Chen  wrote:

> Thanks Guozhang for the reply!
>
> 1. RemoveMemberFromGroupOptions seems not defined anywhere.
> Added the definition.
> 2. LeaveGroupRequest added a list of group instance id, but still keep the
> member id as a singleton; is that intentional? I think to make the protocol
> consistent both member id and instance ids could be plural.
> Since a dynamic member would send LeaveGroupRequest with its member.id,
> I feel it's ok to keep the existing API instead of expanding singleton to
> a list. Haven't
> been able to define a scenario where we need to pass a list of `member.id
> `.
> What do you think?
>
> 3. About the *kafka-remove-member-from-group.sh *tool, I'm wondering if we
> can defer adding this while just add the corresponding calls of the
> LeaveGroupRequest inside Streams until we have used it in production and
> hence have a better understanding on how flexible or extensible if we want
> to add any cmd tools. The rationale is that if we do not necessarily need
> it now, we can always add it later with a more think-through API design,
> but if we add the tool in a rush, we may need to extend or modify it soon
> after we realize its limits in operations.
> Totally agree. I moved this part to the future work, because tooling
> options could be addressed
> in a separate KIP and a universally favorable solution could be discussed
> independently (for different
> company setup)
>
> Best,
> Boyang
>
> ____________
> From: Guozhang Wang 
> Sent: Tuesday, December 4, 2018 1:27 AM
> To: dev
> Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> reduce consumer rebalances
>
> Hello Boyang,
>
> I've browsed through the new wiki and there are still a couple of minor
> things to notice:
>
> 1. RemoveMemberFromGroupOptions seems not defined anywhere.
>
> 2. LeaveGroupRequest added a list of group instance id, but still keep the
> member id as a singleton; is that intentional? I think to make the protocol
> consistent both member id and instance ids could be plural.
>
> 3. About the *kafka-remove-member-from-group.sh *tool, I'm wondering if we
> can defer adding this while just add the corresponding calls of the
> LeaveGroupRequest inside Streams until we have used it in production and
> hence have a better understanding on how flexible or extensible if we want
> to add any cmd tools. The rationale is that if we do not necessarily need
> it now, we can always add it later with a more think-through API design,
> but if we add the tool in a rush, we may need to extend or modify it soon
> after we realize its limits in operations.
>
> Otherwise, I'm +1 on the proposal.
>
> Guozhang
>
>
> On Mon, Dec 3, 2018 at 9:14 AM Boyang Chen  wrote:
>
> > Hey community friends,
> >
> > after another month of polishing, KIP-345<
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalancesdata=02%7C01%7C%7C22bfb18c4c334920ecd908d659b92ecb%7C84df9e7fe9f640afb435%7C1%7C0%7C636795049214148190sdata=Ywn2m03BnJvfIzaoF2XsCF2%2BcWaO529mUFaXl7Hd%2BFk%3Dreserved=0
> >
> > design is ready for vote. Feel free to add your comment on the discussion
> > thread or here.
> >
> > Thanks for your time!
> >
> > Boyang
> > 
> > From: Boyang Chen 
> > Sent: Friday, November 9, 2018 6:35 AM
> > To: dev@kafka.apache.org
> > Subject: [VOTE] KIP-345: Introduce static membership protocol to reduce
> > consumer rebalances
> >
> > Hey all,
> >
> >
> > thanks so much for all the inputs on KIP-345 so far. The original
> proposal
> > has enhanced a lot with your help. To make sure the implementation go
> > smoothly without back and forth, I would like to start a vote on the
> final
> > design agreement now:
> >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-data=02%7C01%7C%7C22bfb18c4c334920ecd908d659b92ecb%7C84df9e7fe9f640afb435%7C1%7C0%7C636795049214148190sda

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

[Guozhang Wang]

2. That's a good point. I think I'm convinced.


Guozhang

On Mon, Dec 3, 2018 at 5:43 PM Boyang Chen  wrote:

> Thanks Guozhang for the reply!
>
> 1. RemoveMemberFromGroupOptions seems not defined anywhere.
> Added the definition.
> 2. LeaveGroupRequest added a list of group instance id, but still keep the
> member id as a singleton; is that intentional? I think to make the protocol
> consistent both member id and instance ids could be plural.
> Since a dynamic member would send LeaveGroupRequest with its member.id,
> I feel it's ok to keep the existing API instead of expanding singleton to
> a list. Haven't
> been able to define a scenario where we need to pass a list of `member.id
> `.
> What do you think?
>
> 3. About the *kafka-remove-member-from-group.sh *tool, I'm wondering if we
> can defer adding this while just add the corresponding calls of the
> LeaveGroupRequest inside Streams until we have used it in production and
> hence have a better understanding on how flexible or extensible if we want
> to add any cmd tools. The rationale is that if we do not necessarily need
> it now, we can always add it later with a more think-through API design,
> but if we add the tool in a rush, we may need to extend or modify it soon
> after we realize its limits in operations.
> Totally agree. I moved this part to the future work, because tooling
> options could be addressed
> in a separate KIP and a universally favorable solution could be discussed
> independently (for different
> company setup)
>
> Best,
> Boyang
>
> ________
> From: Guozhang Wang 
> Sent: Tuesday, December 4, 2018 1:27 AM
> To: dev
> Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> reduce consumer rebalances
>
> Hello Boyang,
>
> I've browsed through the new wiki and there are still a couple of minor
> things to notice:
>
> 1. RemoveMemberFromGroupOptions seems not defined anywhere.
>
> 2. LeaveGroupRequest added a list of group instance id, but still keep the
> member id as a singleton; is that intentional? I think to make the protocol
> consistent both member id and instance ids could be plural.
>
> 3. About the *kafka-remove-member-from-group.sh *tool, I'm wondering if we
> can defer adding this while just add the corresponding calls of the
> LeaveGroupRequest inside Streams until we have used it in production and
> hence have a better understanding on how flexible or extensible if we want
> to add any cmd tools. The rationale is that if we do not necessarily need
> it now, we can always add it later with a more think-through API design,
> but if we add the tool in a rush, we may need to extend or modify it soon
> after we realize its limits in operations.
>
> Otherwise, I'm +1 on the proposal.
>
> Guozhang
>
>
> On Mon, Dec 3, 2018 at 9:14 AM Boyang Chen  wrote:
>
> > Hey community friends,
> >
> > after another month of polishing, KIP-345<
> >
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalancesdata=02%7C01%7C%7C94b56c977f3647e1141908d659458c8c%7C84df9e7fe9f640afb435%7C1%7C0%7C636794552568572008sdata=LiNnhFJm8Avri26aEBa3q4%2Fr4aRKVIrZzKHzn71U3Xk%3Dreserved=0
> >
> > design is ready for vote. Feel free to add your comment on the discussion
> > thread or here.
> >
> > Thanks for your time!
> >
> > Boyang
> > 
> > From: Boyang Chen 
> > Sent: Friday, November 9, 2018 6:35 AM
> > To: dev@kafka.apache.org
> > Subject: [VOTE] KIP-345: Introduce static membership protocol to reduce
> > consumer rebalances
> >
> > Hey all,
> >
> >
> > thanks so much for all the inputs on KIP-345 so far. The original
> proposal
> > has enhanced a lot with your help. To make sure the implementation go
> > smoothly without back and forth, I would like to start a vote on the
> final
> > design agreement now:
> >
> >
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-data=02%7C01%7C%7C94b56c977f3647e1141908d659458c8c%7C84df9e7fe9f640afb435%7C1%7C0%7C636794552568572008sdata=GwbfkDFkY2m38V2e%2B6bEWU7PKWPoia5Hw6KmdOXrdcs%3Dreserved=0
> <
> >
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalancesdata=02%7C01%7C%7C94b56c977f3647e1141908d659458c8c%7C84df9e7fe9f640afb435%7C1%7C0%7C6367945525685720

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

[Boyang Chen]

Oops, sure thing Mayuresh :)

I have only one open question for Guozhang. Will definitely move the discussion 
back.

Boyang

From: Mayuresh Gharat 
Sent: Tuesday, December 4, 2018 9:52 AM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to reduce 
consumer rebalances

Hi Folks,

Would it be good to move this to the DISCUSS thread and keep this thread
only for voting purposes, else it will be hard to coordinate responses
between 2 threads.

Thanks,

Mayuresh



On Mon, Dec 3, 2018 at 5:43 PM Boyang Chen  wrote:

> Thanks Guozhang for the reply!
>
> 1. RemoveMemberFromGroupOptions seems not defined anywhere.
> Added the definition.
> 2. LeaveGroupRequest added a list of group instance id, but still keep the
> member id as a singleton; is that intentional? I think to make the protocol
> consistent both member id and instance ids could be plural.
> Since a dynamic member would send LeaveGroupRequest with its member.id,
> I feel it's ok to keep the existing API instead of expanding singleton to
> a list. Haven't
> been able to define a scenario where we need to pass a list of `member.id
> `.
> What do you think?
>
> 3. About the *kafka-remove-member-from-group.sh *tool, I'm wondering if we
> can defer adding this while just add the corresponding calls of the
> LeaveGroupRequest inside Streams until we have used it in production and
> hence have a better understanding on how flexible or extensible if we want
> to add any cmd tools. The rationale is that if we do not necessarily need
> it now, we can always add it later with a more think-through API design,
> but if we add the tool in a rush, we may need to extend or modify it soon
> after we realize its limits in operations.
> Totally agree. I moved this part to the future work, because tooling
> options could be addressed
> in a separate KIP and a universally favorable solution could be discussed
> independently (for different
> company setup)
>
> Best,
> Boyang
>
> ________
> From: Guozhang Wang 
> Sent: Tuesday, December 4, 2018 1:27 AM
> To: dev
> Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> reduce consumer rebalances
>
> Hello Boyang,
>
> I've browsed through the new wiki and there are still a couple of minor
> things to notice:
>
> 1. RemoveMemberFromGroupOptions seems not defined anywhere.
>
> 2. LeaveGroupRequest added a list of group instance id, but still keep the
> member id as a singleton; is that intentional? I think to make the protocol
> consistent both member id and instance ids could be plural.
>
> 3. About the *kafka-remove-member-from-group.sh *tool, I'm wondering if we
> can defer adding this while just add the corresponding calls of the
> LeaveGroupRequest inside Streams until we have used it in production and
> hence have a better understanding on how flexible or extensible if we want
> to add any cmd tools. The rationale is that if we do not necessarily need
> it now, we can always add it later with a more think-through API design,
> but if we add the tool in a rush, we may need to extend or modify it soon
> after we realize its limits in operations.
>
> Otherwise, I'm +1 on the proposal.
>
> Guozhang
>
>
> On Mon, Dec 3, 2018 at 9:14 AM Boyang Chen  wrote:
>
> > Hey community friends,
> >
> > after another month of polishing, KIP-345<
> >
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalancesdata=02%7C01%7C%7C879b92d6c3fe4da92c1908d6598d7b3d%7C84df9e7fe9f640afb435%7C1%7C0%7C636794861519727292sdata=s6MOIBWvgDrGPXXBu4vXcsI1Z2dcJIteJQT4zdo%2FSMY%3Dreserved=0
> >
> > design is ready for vote. Feel free to add your comment on the discussion
> > thread or here.
> >
> > Thanks for your time!
> >
> > Boyang
> > 
> > From: Boyang Chen 
> > Sent: Friday, November 9, 2018 6:35 AM
> > To: dev@kafka.apache.org
> > Subject: [VOTE] KIP-345: Introduce static membership protocol to reduce
> > consumer rebalances
> >
> > Hey all,
> >
> >
> > thanks so much for all the inputs on KIP-345 so far. The original
> proposal
> > has enhanced a lot with your help. To make sure the implementation go
> > smoothly without back and forth, I would like to start a vote on the
> final
> > design agreement now:
> >
> >
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-data=02%7C01%7C%7C879b92d6c3fe4da92c1908d6598d7

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

[Mayuresh Gharat]

Hi Folks,

Would it be good to move this to the DISCUSS thread and keep this thread
only for voting purposes, else it will be hard to coordinate responses
between 2 threads.

Thanks,

Mayuresh



On Mon, Dec 3, 2018 at 5:43 PM Boyang Chen  wrote:

> Thanks Guozhang for the reply!
>
> 1. RemoveMemberFromGroupOptions seems not defined anywhere.
> Added the definition.
> 2. LeaveGroupRequest added a list of group instance id, but still keep the
> member id as a singleton; is that intentional? I think to make the protocol
> consistent both member id and instance ids could be plural.
> Since a dynamic member would send LeaveGroupRequest with its member.id,
> I feel it's ok to keep the existing API instead of expanding singleton to
> a list. Haven't
> been able to define a scenario where we need to pass a list of `member.id
> `.
> What do you think?
>
> 3. About the *kafka-remove-member-from-group.sh *tool, I'm wondering if we
> can defer adding this while just add the corresponding calls of the
> LeaveGroupRequest inside Streams until we have used it in production and
> hence have a better understanding on how flexible or extensible if we want
> to add any cmd tools. The rationale is that if we do not necessarily need
> it now, we can always add it later with a more think-through API design,
> but if we add the tool in a rush, we may need to extend or modify it soon
> after we realize its limits in operations.
> Totally agree. I moved this part to the future work, because tooling
> options could be addressed
> in a separate KIP and a universally favorable solution could be discussed
> independently (for different
> company setup)
>
> Best,
> Boyang
>
> ________
> From: Guozhang Wang 
> Sent: Tuesday, December 4, 2018 1:27 AM
> To: dev
> Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> reduce consumer rebalances
>
> Hello Boyang,
>
> I've browsed through the new wiki and there are still a couple of minor
> things to notice:
>
> 1. RemoveMemberFromGroupOptions seems not defined anywhere.
>
> 2. LeaveGroupRequest added a list of group instance id, but still keep the
> member id as a singleton; is that intentional? I think to make the protocol
> consistent both member id and instance ids could be plural.
>
> 3. About the *kafka-remove-member-from-group.sh *tool, I'm wondering if we
> can defer adding this while just add the corresponding calls of the
> LeaveGroupRequest inside Streams until we have used it in production and
> hence have a better understanding on how flexible or extensible if we want
> to add any cmd tools. The rationale is that if we do not necessarily need
> it now, we can always add it later with a more think-through API design,
> but if we add the tool in a rush, we may need to extend or modify it soon
> after we realize its limits in operations.
>
> Otherwise, I'm +1 on the proposal.
>
> Guozhang
>
>
> On Mon, Dec 3, 2018 at 9:14 AM Boyang Chen  wrote:
>
> > Hey community friends,
> >
> > after another month of polishing, KIP-345<
> >
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalancesdata=02%7C01%7C%7C94b56c977f3647e1141908d659458c8c%7C84df9e7fe9f640afb435%7C1%7C0%7C636794552568572008sdata=LiNnhFJm8Avri26aEBa3q4%2Fr4aRKVIrZzKHzn71U3Xk%3Dreserved=0
> >
> > design is ready for vote. Feel free to add your comment on the discussion
> > thread or here.
> >
> > Thanks for your time!
> >
> > Boyang
> > 
> > From: Boyang Chen 
> > Sent: Friday, November 9, 2018 6:35 AM
> > To: dev@kafka.apache.org
> > Subject: [VOTE] KIP-345: Introduce static membership protocol to reduce
> > consumer rebalances
> >
> > Hey all,
> >
> >
> > thanks so much for all the inputs on KIP-345 so far. The original
> proposal
> > has enhanced a lot with your help. To make sure the implementation go
> > smoothly without back and forth, I would like to start a vote on the
> final
> > design agreement now:
> >
> >
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-data=02%7C01%7C%7C94b56c977f3647e1141908d659458c8c%7C84df9e7fe9f640afb435%7C1%7C0%7C636794552568572008sdata=GwbfkDFkY2m38V2e%2B6bEWU7PKWPoia5Hw6KmdOXrdcs%3Dreserved=0
> <
> >
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Br

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

[Boyang Chen]

Thanks Guozhang for the reply!

1. RemoveMemberFromGroupOptions seems not defined anywhere.
Added the definition.
2. LeaveGroupRequest added a list of group instance id, but still keep the
member id as a singleton; is that intentional? I think to make the protocol
consistent both member id and instance ids could be plural.
Since a dynamic member would send LeaveGroupRequest with its member.id,
I feel it's ok to keep the existing API instead of expanding singleton to a 
list. Haven't
been able to define a scenario where we need to pass a list of `member.id`.
What do you think?

3. About the *kafka-remove-member-from-group.sh *tool, I'm wondering if we
can defer adding this while just add the corresponding calls of the
LeaveGroupRequest inside Streams until we have used it in production and
hence have a better understanding on how flexible or extensible if we want
to add any cmd tools. The rationale is that if we do not necessarily need
it now, we can always add it later with a more think-through API design,
but if we add the tool in a rush, we may need to extend or modify it soon
after we realize its limits in operations.
Totally agree. I moved this part to the future work, because tooling options 
could be addressed
in a separate KIP and a universally favorable solution could be discussed 
independently (for different
company setup)

Best,
Boyang


From: Guozhang Wang 
Sent: Tuesday, December 4, 2018 1:27 AM
To: dev
Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to reduce 
consumer rebalances

Hello Boyang,

I've browsed through the new wiki and there are still a couple of minor
things to notice:

1. RemoveMemberFromGroupOptions seems not defined anywhere.

2. LeaveGroupRequest added a list of group instance id, but still keep the
member id as a singleton; is that intentional? I think to make the protocol
consistent both member id and instance ids could be plural.

3. About the *kafka-remove-member-from-group.sh *tool, I'm wondering if we
can defer adding this while just add the corresponding calls of the
LeaveGroupRequest inside Streams until we have used it in production and
hence have a better understanding on how flexible or extensible if we want
to add any cmd tools. The rationale is that if we do not necessarily need
it now, we can always add it later with a more think-through API design,
but if we add the tool in a rush, we may need to extend or modify it soon
after we realize its limits in operations.

Otherwise, I'm +1 on the proposal.

Guozhang


On Mon, Dec 3, 2018 at 9:14 AM Boyang Chen  wrote:

> Hey community friends,
>
> after another month of polishing, KIP-345<
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalancesdata=02%7C01%7C%7C94b56c977f3647e1141908d659458c8c%7C84df9e7fe9f640afb435%7C1%7C0%7C636794552568572008sdata=LiNnhFJm8Avri26aEBa3q4%2Fr4aRKVIrZzKHzn71U3Xk%3Dreserved=0>
> design is ready for vote. Feel free to add your comment on the discussion
> thread or here.
>
> Thanks for your time!
>
> Boyang
> 
> From: Boyang Chen 
> Sent: Friday, November 9, 2018 6:35 AM
> To: dev@kafka.apache.org
> Subject: [VOTE] KIP-345: Introduce static membership protocol to reduce
> consumer rebalances
>
> Hey all,
>
>
> thanks so much for all the inputs on KIP-345 so far. The original proposal
> has enhanced a lot with your help. To make sure the implementation go
> smoothly without back and forth, I would like to start a vote on the final
> design agreement now:
>
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-data=02%7C01%7C%7C94b56c977f3647e1141908d659458c8c%7C84df9e7fe9f640afb435%7C1%7C0%7C636794552568572008sdata=GwbfkDFkY2m38V2e%2B6bEWU7PKWPoia5Hw6KmdOXrdcs%3Dreserved=0<
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalancesdata=02%7C01%7C%7C94b56c977f3647e1141908d659458c8c%7C84df9e7fe9f640afb435%7C1%7C0%7C636794552568572008sdata=LiNnhFJm8Avri26aEBa3q4%2Fr4aRKVIrZzKHzn71U3Xk%3Dreserved=0
> >
>
> 345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances<
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalancesdata=02%7C01%7C%7C94b56c977f3647e1141908d659458c8c%7C84df9e7fe9f640afb435%7C1%7C0%7C636794552568572008sdata=LiNnhFJm8Avri26aEBa3q4%2Fr4aRKVIrZzKHzn71U3Xk%3Dreserved=0
> >
>
> KIP-34

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

[Guozhang Wang]

Hello Boyang,

I've browsed through the new wiki and there are still a couple of minor
things to notice:

1. RemoveMemberFromGroupOptions seems not defined anywhere.

2. LeaveGroupRequest added a list of group instance id, but still keep the
member id as a singleton; is that intentional? I think to make the protocol
consistent both member id and instance ids could be plural.

3. About the *kafka-remove-member-from-group.sh *tool, I'm wondering if we
can defer adding this while just add the corresponding calls of the
LeaveGroupRequest inside Streams until we have used it in production and
hence have a better understanding on how flexible or extensible if we want
to add any cmd tools. The rationale is that if we do not necessarily need
it now, we can always add it later with a more think-through API design,
but if we add the tool in a rush, we may need to extend or modify it soon
after we realize its limits in operations.

Otherwise, I'm +1 on the proposal.

Guozhang


On Mon, Dec 3, 2018 at 9:14 AM Boyang Chen  wrote:

> Hey community friends,
>
> after another month of polishing, KIP-345<
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances>
> design is ready for vote. Feel free to add your comment on the discussion
> thread or here.
>
> Thanks for your time!
>
> Boyang
> 
> From: Boyang Chen 
> Sent: Friday, November 9, 2018 6:35 AM
> To: dev@kafka.apache.org
> Subject: [VOTE] KIP-345: Introduce static membership protocol to reduce
> consumer rebalances
>
> Hey all,
>
>
> thanks so much for all the inputs on KIP-345 so far. The original proposal
> has enhanced a lot with your help. To make sure the implementation go
> smoothly without back and forth, I would like to start a vote on the final
> design agreement now:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-<
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances
> >
>
> 345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances<
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances
> >
>
> KIP-345: Introduce static membership protocol to reduce ...<
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances
> >
> cwiki.apache.org
> For stateful applications, one of the biggest performance bottleneck is
> the state shuffling. In Kafka consumer, there is a concept called
> "rebalance" which means that for given M partitions and N consumers in one
> consumer group, Kafka will try to balance the load between consumers and
> ideally have ...
>
>
> Let me know if you have any questions.
>
>
> Best,
>
> Boyang
>
>

-- 
-- Guozhang

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

[Boyang Chen]

Hey community friends,

after another month of polishing, 
KIP-345
 design is ready for vote. Feel free to add your comment on the discussion
thread or here.

Thanks for your time!

Boyang

From: Boyang Chen 
Sent: Friday, November 9, 2018 6:35 AM
To: dev@kafka.apache.org
Subject: [VOTE] KIP-345: Introduce static membership protocol to reduce 
consumer rebalances

Hey all,


thanks so much for all the inputs on KIP-345 so far. The original proposal has 
enhanced a lot with your help. To make sure the implementation go smoothly 
without back and forth, I would like to start a vote on the final design 
agreement now:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-

345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances

KIP-345: Introduce static membership protocol to reduce 
...
cwiki.apache.org
For stateful applications, one of the biggest performance bottleneck is the 
state shuffling. In Kafka consumer, there is a concept called "rebalance" which 
means that for given M partitions and N consumers in one consumer group, Kafka 
will try to balance the load between consumers and ideally have ...


Let me know if you have any questions.


Best,

Boyang

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

[Boyang Chen]

Adding discussion thread here:

https://lists.apache.org/thread.html/9868b07692960d7ab2924d2c66ad6ce7e42e492c35af9cbb7c9981ae@%3Cdev.kafka.apache.org%3E



From: Boyang Chen 
Sent: Friday, November 9, 2018 6:35 AM
To: dev@kafka.apache.org
Subject: [VOTE] KIP-345: Introduce static membership protocol to reduce 
consumer rebalances


Hey all,


thanks so much for all the inputs on KIP-345 so far. The original proposal has 
enhanced a lot with your help. To make sure the implementation go smoothly 
without back and forth, I would like to start a vote on the final design 
agreement now:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-

345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances

KIP-345: Introduce static membership protocol to reduce 
...
cwiki.apache.org
For stateful applications, one of the biggest performance bottleneck is the 
state shuffling. In Kafka consumer, there is a concept called "rebalance" which 
means that for given M partitions and N consumers in one consumer group, Kafka 
will try to balance the load between consumers and ideally have ...


Let me know if you have any questions.


Best,

Boyang