>>>>>> 2. In "Consumer Coordinator Algorithm", 1c suggests to revoke
> all
> >>>>>>> partition upon heartbeat/commit fail. What's the gain here? Do we
> >> want
> >>>> to
> >>>>>>> keep a
evoke all
>>>>>>> partition upon heartbeat/commit fail. What's the gain here? Do we
>> want
>>>> to
>>>>>>> keep all partitions running at this moment, to be optimistic for the
>>>> case
>>>>>>> when no partiti
nd the rest of the paragraph did not
> >>>>> talk
> >>>>>>> about Streams any more. If you feel it's a bit distracted I can
> >> remove
> >>>>>>> those examples.
> >>>>>>>
> >>>>>>> 10). While working
t;> proactive changes. The idea is to keep logic simpler and not doing
>> any
>>>>>>> "split brain" stuff.
>>>>>>>
>>>>>>> 20). 2.b. No we do not need, since the owned-partitions will be part
>>>>> of
>>>&g
ut two "looking into the future" into a separate category from
> >>> migration session. It seems inconsistent for readers to see this
> before we
> >>> finished discussion for everything.
> >>>
> >>
> >> Ack.
> >>
> >>
gt; finished discussion for everything.
>>>
>>
>> Ack.
>>
>>
>>> 5. Have we discussed the concern on the serialization? Could the new
>>> metadata we are adding grow larger than the message size cap?
>>>
>>
>&
ers to see this before we
>> finished discussion for everything.
>>
>
> Ack.
>
>
>> 5. Have we discussed the concern on the serialization? Could the new
>> metadata we are adding grow larger than the message size cap?
>>
>
> We're completing https
5. Have we discussed the concern on the serialization? Could the new
> metadata we are adding grow larger than the message size cap?
>
We're completing https://issues.apache.org/jira/browse/KAFKA-7149 which
should largely reduce the message size (will update the wiki accordingly as
well).
&
re adding grow larger than the message size cap?
Boyang
________________________
From: Guozhang Wang
Sent: Monday, April 15, 2019 9:20 AM
To: dev
Subject: Re: [DISCUSS] KIP-429 : Smooth Auto-Scaling for Kafka Streams
Hello Jason,
I agree with you that for range / round-robin it makes less
ability and resource shuffling. If we eventually trigger
> > > rebalance after rolling bounce, certain consumer
> > >
> > > setup is still faced with global shuffles, for example member.id
> ranking
> > > based round robin strategy, as rejoining dynamic
> > &g
ryone's version -- again it is
> assuming
> >> that higher versioned protocol is always backward compatible -- and
> hence
> >> can successfully do the assignment at that round.
> >>
> >> 60). My bad, this section was not updated while the design was evolved,
> >> I've updated it.
> >&g
ask availability and resource shuffling. If we eventually trigger
>>> rebalance after rolling bounce, certain consumer
>>>
>>> setup is still faced with global shuffles, for example member.id ranking
>>> based round robin strategy, as rejoining dynamic
>>>
>>
I think the primary goal of incremental
> >
> > rebalancing is still improving the cluster availability during rebalance,
> > because it didn't revoke any partition during this
> >
> > process. Also, the perk is minimum configuration requirement :)
> >
> &
, the perk is minimum configuration requirement :)
>
>
> Best,
>
> Boyang
>
>
> From: Matthias J. Sax
> Sent: Tuesday, April 9, 2019 7:47 AM
> To: dev
> Subject: Re: [DISCUSS] KIP-429 : Smooth Auto-Scaling for Kafka Streams
>
> T
evoke any partition during this
> >
> > process. Also, the perk is minimum configuration requirement :)
> >
> >
> > Best,
> >
> > Boyang
> >
> >
> > From: Matthias J. Sax
> > Sent: Tuesday, April 9, 2
Best,
>
> Boyang
>
>
> From: Matthias J. Sax
> Sent: Tuesday, April 9, 2019 7:47 AM
> To: dev
> Subject: Re: [DISCUSS] KIP-429 : Smooth Auto-Scaling for Kafka Streams
>
> Thank for the KIP, Boyang and Guozhang!
>
>
> I made an in
this
process. Also, the perk is minimum configuration requirement :)
Best,
Boyang
From: Matthias J. Sax
Sent: Tuesday, April 9, 2019 7:47 AM
To: dev
Subject: Re: [DISCUSS] KIP-429 : Smooth Auto-Scaling for Kafka Streams
Thank for the KIP, Boyang and Guozhang
fferent stages. At very beginning, our goal is to trigger
>>> learner task assignment only on
>>> `new` hosts, where we shall leverage leader's knowledge of previous round
>>> of rebalance to figure out. After
>>> stage one, our goal is to have a smooth sc
a smooth scaling up experience, but the
>> task balance problem is kind of orthogonal.
>> The load balance problem is a much broader topic than auto scaling, which
>> I figure worth discussing within
>> this KIP's context since it's a naturally next-step, but wouldn't be the
&g
Sounds good Guozhang! I will kick another discussion thread for KIP-444.
Boyang
From: Guozhang Wang
Sent: Thursday, March 21, 2019 9:17 AM
To: dev
Subject: Re: [DISCUSS] KIP-429 : Smooth Auto-Scaling for Kafka Streams
Hello Boyang,
I've made another thorough
ce`, but not `an algorithm defining load
> balance`. It would be great if you could share some insights of the stream
> task balance, which eventually helps us to break out of the KIP-429's scope
> and even define a separate KIP to focus on task weight & assignment logic
> improv
t: Saturday, March 2, 2019 6:00 AM
To: dev
Subject: Re: [DISCUSS] KIP-429 : Smooth Auto-Scaling for Kafka Streams
Hello Boyang,
I've just made a quick pass on the KIP and here are some thoughts.
Meta:
1. I'm still not sure if it's worthwhile to add a new type of "learner
task" in
Hello Boyang,
I've just made a quick pass on the KIP and here are some thoughts.
Meta:
1. I'm still not sure if it's worthwhile to add a new type of "learner
task" in addition to "standby task": if the only difference is that for the
latter, we would consider workload balance while for the
Hey community friends,
I'm gladly inviting you to have a look at the proposal to add incremental
rebalancing to Kafka Streams, A.K.A auto-scaling support.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-429%3A+Smooth+Auto-Scaling+for+Kafka+Streams
Special thanks to Guozhang for giving
24 matches
Mail list logo
