, June 14, 2019 8:42 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hi Boyang,
Thanks for the update. To what extent will KIP-345 be available in 2.3.0?
Mike
On 6/13/19, 5:36 PM, "Boyang Chen" wrote:
Hey al
, April 26, 2019 11:16 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hey all,
there is a minor change to the stream side logic for static
membership<https://cwiki.apache.org/confluence/display/
any concerns.
Best,
Boyang
From: Boyang Chen
Sent: Friday, April 26, 2019 11:16 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hey all,
there is a minor change to the stream side logic
: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hey all,
there is a minor change to the stream side logic for static
membership<https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalan
nsumer rebalances by
specifying member id
Hi Mike,
Yes that's the plan!
From: Mike Freyberger
Sent: Saturday, March 9, 2019 10:04 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hi
Hi Mike,
Yes that's the plan!
From: Mike Freyberger
Sent: Saturday, March 9, 2019 10:04 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hi Boyang,
Is this work targeted for Kafka 2.3? I
Hi Boyang,
Is this work targeted for Kafka 2.3? I am eager to use this new feature.
Thanks,
Mike Freyberger
On 12/21/18, 1:21 PM, "Mayuresh Gharat" wrote:
Hi Boyang,
Regarding "However, we shall still attempt to remove the member static info
if the given `member.id` points
Yep, that is correct!
Get Outlook for iOS<https://aka.ms/o0ukef>
From: Mayuresh Gharat
Sent: Wednesday, January 2, 2019 8:30 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hi
gt; Sent: Saturday, December 22, 2018 2:21 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> Hi Boyang,
>
> Regarding "However, we shall still attempt to remove the member static info
> if the given
ent: Saturday, December 22, 2018 2:21 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hi Boyang,
Regarding "However, we shall still attempt to remove the member static info
if the given `member.id` points to an
Hi Boyang,
Regarding "However, we shall still attempt to remove the member static info
if the given `member.id` points to an existing `group.instance.id` upon
LeaveGroupRequest, because I could think of the possibility that in long
term we could want to add static membership leave group logic for
with the suggestion to make the leave group request change
> generic. So this new Stream API
> will be added on the rest layer to expose the necessary ids correct?
>
> Looking forward to your confirmation
>
> Best,
> Boyang
>
> --------------
> *From:
this new Stream API
> will be added on the rest layer to expose the necessary ids correct?
>
> Looking forward to your confirmation
>
> Best,
> Boyang
>
> ____________
> From: Guozhang Wang
> Sent: Saturday, December 1, 2018 7:00 AM
> To: dev
&
instanceid prefix, and list[instanceid]
> > for clarity purpose. As you know, two options are equivalent since full
> > name is subset of prefix.
> >
> > Let me know your thoughts!
> >
> > Boyang
> > ____
> > From: Boyang Ch
t; From: Boyang Chen
> Sent: Thursday, November 29, 2018 3:39 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> Thanks Guozhang for the new proposal here!
>
> So I'd like to propose a slightly
: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Thanks Guozhang for the new proposal here!
So I'd like to propose a slightly modified version of LeaveGroupRequest:
instead of letting the static member consumer client themselves to send the
request (which means we still
management tooling.
How does this workaround sound?
Boyang
From: Guozhang Wang
Sent: Thursday, November 29, 2018 2:38 AM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hi Boyang,
I was thinking that with the optional static members i
>
> 3) personally I favor "ids" over "names" :) Since we already have some
> "ids" and hence it sounds more consistent, plus on the producer side we
> have a `transactional.id` whose semantics is a bit similar to this one,
> i.e. for unique distinguishment of a client which may
may comes and goes but
need to be persist over multiple "instance life-times".
Sure we have enough votes for ids I will finalize the name to
`group.instance.id`, does that
sound good?
Best,
Boyang
From: Guozhang Wang
Sent: Wednesday, November 28, 2018 4:51 AM
To: dev
Subject: R
ession timeout expiration. I am wondering how critical
> > this piece is and whether we can leave it for future work. If not, then
> it
> > would be helpful to elaborate on its implementation. How would the
> > coordinator know which members to kick out of the group?
> >
> > This API is ne
b0cbc94390f04b08d653fa2832%7C84df9e7fe9f640afb435%7C1%7C0%7C636788731208857126sdata=EDM7PmpOo2HenYhFHX2rxrszpkI7di401WhKh2Vjw5k%3Dreserved=0
> > >> >
> > >> we will automatically fence all join requests with UNKNOWN_MEMBER_ID.
> > >>
> > >> 4)
Yea, Dong and Stanislav also mentioned this naming. I personally buy in the
namespace idea, and
since we already use `member.name` in a lot of context, I decide to rename the
config to `group.member.name`
which should be sufficient for solving all the concerns we have now. Sounds
good?
Than
e%7C84df9e7fe9f640afb435%7C1%7C0%7C636786393153281080sdata=%2BNasMFlJf9rEJc9iDfndcyxA4%2BGWieS1azSKbtdGRW4%3Dreserved=0
> >> .
> >> Added the script.
> >>
> >> 7) Would it be simpler to replace name "forceStaticRebalance" with
> &
t; > > A and produces to Kafka cluster B.
>> > > Depending on the data and the Kafka cluster configuration, Kafka
>> service
>> > > providers can set a mirroring group saying that it will take, for
>> example
>> > > 300 consumer hosts/members to
ethod through admin
> client API.
>
> 8) It is not very clear how the newly added AdminClient API trigger
> rebalance. For example, does it send request? Can this be explained in the
> KIP?
>
> Sure, I will add more details to the API.
>
>
> Thanks again for the help
he API.
Thanks again for the helpful suggestions!
Best,
Boyang
From: Dong Lin
Sent: Saturday, November 24, 2018 2:54 PM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hey Boyang,
Thanks for the update! Here are
coded
> client ids to the group and replace the dead host, or as you proposed to
> define spare host as
> what I understood as hot backup. I will put both Jason and your
> suggestions into a separate section
> called "Future works". Note that this spare host ide
through rebalance protocol
IMO.
Thank you again for the great feedback!
Boyang
____________
From: Boyang Chen
Sent: Thursday, November 22, 2018 3:39 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member i
round Mayuresh )
Best,
Boyang
From: Mayuresh Gharat
Sent: Wednesday, November 21, 2018 7:50 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hi Boyang,
Thanks for updating the KIP. This is a
nce
> > duplicate member.name issue could be handled by join group request. We
> > shall reject join group with known member name but no member id (which
> > means we already have an active member using this identity).
> > 6. I agree to remove that internal config once we mo
that internal config once we move forward with
> static membership. And I already removed the entire section from the KIP.
>
> Let me know if you have other concerns.
>
> Best,
> Boyang
>
> From: Guozhang Wang
> Sent: Tuesday, November 20, 20
.
>
> Let me know if you have other concerns.
>
> Best,
> Boyang
> ____________
> From: Guozhang Wang
> Sent: Tuesday, November 20, 2018 4:21 PM
> To: dev
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying m
: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hello Boyang,
Thanks a lot for the KIP! It is a great write-up and I appreciate your
patience answering to the feedbacks from the community. I'd like to add my
2cents here:
1. By introducing another two timeout
Hello Boyang,
Thanks a lot for the KIP! It is a great write-up and I appreciate your
patience answering to the feedbacks from the community. I'd like to add my
2cents here:
1. By introducing another two timeout configs, registration_timeout and
expansion_timeout, we are effectively having four
Hey Boyang,
Thanks for the proposal! This is very useful. I have some comments below:
1) The motivation currently explicitly states that the goal is to improve
performance for heavy state application. It seems that the motivation can
be stronger with the following use-case. Currently for
like I mentioned above for replication (auto healing during
> > mid-night if one broker is never back), we could continue discussing the
> > new approaches to have basic guarantee of consumer group liveness.
> >
> >
> > The discussion so far is to make sure that all th
is never back), we could continue discussing the
> > new approaches to have basic guarantee of consumer group liveness.
> >
> >
> > The discussion so far is to make sure that all the design approaches we
> > have taken are pointing to real scenarios. Once we clarify the
. I hope these
> discussions make sense. Thanks again for helping make the design solid!
>
>
> Boyang
>
>
> From: Jason Gustafson
> Sent: Thursday, November 15, 2018 9:54 AM
> To: dev
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple
sions make
sense. Thanks again for helping make the design solid!
Boyang
From: Jason Gustafson
Sent: Thursday, November 15, 2018 9:54 AM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
>
> I feel this would ma
yang
>
> ____________
> From: Jason Gustafson
> Sent: Wednesday, November 14, 2018 9:31 AM
> To: dev
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> Hey Boyang,
>
> Thanks for the updates. From a h
n correspondingly in the
> follow-up KIPs.
>
>
> Boyang
>
>
> From: Mayuresh Gharat
> Sent: Sunday, November 11, 2018 1:06 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
>
> Boyang
>
>
> From: Mayuresh Gharat
> Sent: Sunday, November 11, 2018 1:06 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> Hi Boyang,
>
> Thanks
ets topic. I will do more analysis and make a trade-off
> comparison. Nice catch!
>
>
> I hope the explanations make sense to you. I will keep polishing on the
> edge cases and details.
>
>
> Best,
>
> Boyang
>
> ____________
> From:
ore analysis and make a trade-off
> comparison. Nice catch!
>
>
> I hope the explanations make sense to you. I will keep polishing on the
> edge cases and details.
>
>
> Best,
>
> Boyang
>
> ____________
> From: Mayuresh Gharat
> Sent: Sat
g
From: Mayuresh Gharat
Sent: Saturday, November 10, 2018 10:25 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hi Boyang,
Thanks for the KIP and sorry for being late to the party. This KIP
is great client proposal. This is great complementation to KIP 345. In a
> high level, we are not having any collision on the path and both proposals
> are making sense here. Just need better sync to avoid duplicate effort :)
>
>
> Best,
>
> Boyang
>
>
> ___
cess of KIP-345.
From: Matthias J. Sax
Sent: Wednesday, November 7, 2018 7:02 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hey,
there was quite a pause on this KIP discussion and
AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hey,
there was quite a pause on this KIP discussion and in the mean time, a
new design for incremental cooporative rebalance was suggested:
https://cwiki.apache.org/confluen
uce admin API in this KIP or a separate one.
>
>
> Thanks again for the proposed ideas!
>
>
> Boyang
>
> ____________
> From: Mike Freyberger
> Sent: Monday, November 5, 2018 6:13 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP
, but agree we could discuss whether we want to
introduce admin API in this KIP or a separate one.
Thanks again for the proposed ideas!
Boyang
From: Mike Freyberger
Sent: Monday, November 5, 2018 6:13 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-3
gt; >> send a
> >>>>>> leave group request (static membership mode). In static membership,
> >>> we
> >>>>> will
> >>>>>> also define `change-group-timeout` to hold on rebalance provided by
IP with our newly discussed
items and details. Really excited to see the design has become more solid.
Best,
Boyang
From: Jason Gustafson
Sent: Saturday, August 25, 2018 6:04 AM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specif
e design has become more solid.
Best,
Boyang
From: Jason Gustafson
Sent: Saturday, August 25, 2018 6:04 AM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hey Mike,
Yeah, that's a good point. A l
mber`. If not, this means there
> >>> are
> >>>>>> duplicate consumer members with same member name and the request
> >> will
> >>>> be
> >>>>>> rejected to guarantee consumption uniqueness.
> >>>>>>
&g
>>>>>> So we will wait for all the members to rejoin the group and do
>>> exactly
>>>>> one
>>>>>> rebalance since all members are expected to rejoin within timeout.
>> If
>>>>>> consumer crashes, t
etween dynamic to static membership.
> > > > >
> > > > > 1. Dynamic to static: the first joiner shall revise the
> membership
> > > to
> > > > > static and wait for all the current members to restart, since their
> > > > >
gt; > > > smooth: just erase the cached mapping, and wait for session timeout
> to
> > > > trigger rebalance should be sufficient. (Fallback to current
> behavior)
> > > > 3. Halfway switch: a corner case is like some clients keep dynamic
> > > &
cause dynamic/static states are
> > > bouncing each other. This could guarantee that we will not make the
> > > consumer group work in a wrong state by having half static and half
> > dynamic.
> > >
> > > To guarantee correctness, we will also pus
e KIP.
> >
> >
> > Are there any concern for this high level proposal? Just want to
> reiterate
> > on the core idea of the KIP: "If the broker recognize this consumer as an
> > existing member, it shouldn't trigger rebalance".
> >
> > Thanks a lot
st want to reiterate
> on the core idea of the KIP: "If the broker recognize this consumer as an
> existing member, it shouldn't trigger rebalance".
>
> Thanks a lot for everyone's input! I feel this proposal is much more
> robust than previous one!
>
>
> Best,
>
ldn't trigger rebalance".
Thanks a lot for everyone's input! I feel this proposal is much more robust
than previous one!
Best,
Boyang
From: Matthias J. Sax
Sent: Friday, August 10, 2018 2:24 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduc
Hi,
thanks for the detailed discussion. I learned a lot about internals again :)
I like the idea or a user config `member.name` and to keep `member.id`
internal. Also agree with Guozhang, that reusing `client.id` might not
be a good idea.
To clarify the algorithm, each time we generate a new
@James
What you described is true: the transition from dynamic to static
memberships are not thought through yet. But I do not think it is an
impossible problem: note that we indeed moved the offset commit from ZK to
kafka coordinator in 0.8.2 :) The migration plan is to first to
double-commits
Guozhang, in a previous message, you proposed said this:
> On Jul 30, 2018, at 3:56 PM, Guozhang Wang wrote:
>
> 1. We bump up the JoinGroupRequest with additional fields:
>
> 1.a) a flag indicating "static" or "dynamic" membership protocols.
> 1.b) with "static" membership, we also add the
From: Guozhang Wang
Sent: Monday, August 6, 2018 8:46 AM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Re: reuse of "client.id": good question. Although client.id could be unique
to guarantee fencing, by using it we are binding
o a minor comment is that our current codebase has two different
> > classes: the GroupCoordinator and ConsumerCoordinator. When talking about
> > term "coordinator", it would be great to distinguish between the two, or
> > explicitly nominate broker coordinator as d
!
From: Guozhang Wang
Sent: Saturday, August 4, 2018 5:25 AM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
@Boyang,
* Yes registry id should be generated via a config, similar to what you
proposed in the KIP currently for member
t;, it would be great to distinguish between the two, or
> explicitly nominate broker coordinator as default "coordinator".
>
>
> Best,
>
> Boyang
>
> ________
> From: Guozhang Wang
> Sent: Thursday, August 2, 2018 1:59 AM
> To: dev
uld be great to distinguish between the two, or
> explicitly nominate broker coordinator as default "coordinator".
>
>
> Best,
>
> Boyang
>
> ____________
> From: Guozhang Wang
> Sent: Thursday, August 2, 2018 1:59 AM
> To: dev
> Subj
: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hey Guozhang,
thanks for this detailed proposal! Quickly I want to clarify, where this
registry id is generated? My understanding is that the registry id is a unique
id provided by user correct? This way even if multiple
___
From: Guozhang Wang
Sent: Thursday, August 2, 2018 1:59 AM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hello Boyang,
Thanks for the great summary.
Personally I think 1) is still doable as we are not necessarily have to
rel
plex (no source of truth model). Hope we are on
> the same page now.
>
>
> Best,
>
> Boyang
>
> ____________
> From: John Roesler
> Sent: Wednesday, August 1, 2018 5:28 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-345: Reduce multi
ly complex (no source of truth model). Hope we are on the same
page now.
Best,
Boyang
From: John Roesler
Sent: Wednesday, August 1, 2018 5:28 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying mem
> > > Sent: Saturday, July 28, 2018 6:50 AM
> > > > > To: dev
> > > > > Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer
> > > rebalances
> > > > by
> >
> > > > instance, where each Stream thread will be using one
> dedicated
> > main
> > > > consumer. So in a Stream use case, we could internally
> generate
> > > member id
> > > > with USER_DEFINED_ID + STREAM_THRE
> > > understand the problem, and use the same set of initialization
> logic
> > cross
> > > multiple consumers on the same instance.
> > >
> > >
> > > I hope I have explained my understanding of the pros and co
; > advanced config, user may use member id config even before
> they fully
> > > understand the problem, and use the same set of initialization
> logic
> > cross
> > > multiple consumers on the same instance.
> > >
> > &
oup be functional, but the
> rebalancing be
> > > non-optimal and require more round-trips/data-transfer? (similar
> to the
> > > current algorithm)
> > >
> > > I'm trying to assess the potential for u
motivation to leader-follower rebalance model, I feel that having
> broker to
> > create membership info and let client maintain would be less
> appealing and
> > fragile. Having client generate membership data could build up
> > source-of-truth model and streamline the curr
server side. To simplify the first version, we do not plan
> to
> > enforce validation. A good comparison would be the EOS producer which is
> in
> > charge of generating unique transaction id sequence. IMO for broker
> logic,
> > the tolerance of cli
gt; > port) on the server side. To simplify the first version, we do not plan
> to
> > enforce validation. A good comparison would be the EOS producer which is
> in
> > charge of generating unique transaction id sequence. IMO for broker
> logic,
> > the tolerance
t: Saturday, July 28, 2018 6:50 AM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hey Boyang,
Thanks for the KIP. I think my main question is in the same vein as James'.
The problem is that the coordinator needs to be able to ide
broker logic,
> the tolerance of client side error is not unlimited.
> >
> >
> > Thank you!
> >
> >
> > ________________
> > From: James Cheng
> > Sent: Saturday, July 28, 2018 1:26 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] K
c, the
> tolerance of client side error is not unlimited.
>
>
> Thank you!
>
>
>
> From: James Cheng
> Sent: Saturday, July 28, 2018 1:26 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer re
-345: Reduce multiple consumer rebalances by
specifying member id
> On Jul 26, 2018, at 11:09 PM, Guozhang Wang wrote:
>
> Hi Boyang,
>
> Thanks for the proposed KIP. I made a pass over the wiki and here are some
> comments / questions:
>
> 1. In order to preserve broke
> On Jul 26, 2018, at 11:09 PM, Guozhang Wang wrote:
>
> Hi Boyang,
>
> Thanks for the proposed KIP. I made a pass over the wiki and here are some
> comments / questions:
>
> 1. In order to preserve broker compatibility, we need to make sure the
> broker version discovery logic can be
Hi Boyang,
Thanks for the proposed KIP. I made a pass over the wiki and here are some
comments / questions:
1. In order to preserve broker compatibility, we need to make sure the
broker version discovery logic can be integrated with this new logic. I.e.
if a newer versioned consumer is talking
Hey friends,
I would like to open a discussion thread on KIP-345:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Reduce+multiple+consumer+rebalances+by+specifying+member+id
This KIP is trying to resolve multiple rebalances by maintaining the consumer
member id across rebalance
88 matches
Mail list logo
