] KIP-54 Sticky Partition Assignment Strategy
I'm still very interested in seeing this KIP progress ...
On Tue, 2 Aug 2016 at 20:09, Vahid S Hashemian
wrote:
> I would like to revive this thread and ask for additional feedback on
this
> KIP.
>
> There have already been some fee
I'm still very interested in seeing this KIP progress ...
On Tue, 2 Aug 2016 at 20:09, Vahid S Hashemian
wrote:
> I would like to revive this thread and ask for additional feedback on this
> KIP.
>
> There have already been some feedback, mostly in favor, plus some concern
> about the value gain
I would like to revive this thread and ask for additional feedback on this
KIP.
There have already been some feedback, mostly in favor, plus some concern
about the value gain considering the complexity and the semantics; i.e.
how the eventually revoked assignments need to be processed in the
o
uot;
Date: 06/23/2016 03:28 PM
Subject: Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy
Just adding some related reference here:
Henry Cai is contributing some advanced feature in Kafka Streams regarding
static assignment: https://github.com/apache/kafka/pull/1543
The main mot
:Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy
Hey Vahid,
Comments below:
I'm not very clear on the first part of this paragraph. You could clarify
> it for me, but in general balancing out the partitions across consumers
in
> a group as much as possible would no
both topics), this is what we get:
> * C0: [t1p0]
> * C1: [t0p1, t1p1]
> * C2: [t0p0]
>
> I think both range and round robin assignors would produce this:
> * C0: [t0p0, t1p1]
> * C1: [t0p1]
> * C2: [t1p0]
>
> Regards,
> --Vahid
>
>
>
>
> From:
t0p1, t1p1]
* C2: [t0p0]
I think both range and round robin assignors would produce this:
* C0: [t0p0, t1p1]
* C1: [t0p1]
* C2: [t1p0]
Regards,
--Vahid
From: Jason Gustafson
To: dev@kafka.apache.org
Date: 06/23/2016 10:06 AM
Subject: Re: [DISCUSS] KIP-54 Sticky Partition
Hi Jason,
I appreciate your feedback.
Please see my comments below, and advise if you have further suggestions.
Thanks.
Regards,
--Vahid
From: Jason Gustafson
To: dev@kafka.apache.org
Date: 06/22/2016 04:41 PM
Subject:Re: [DISCUSS] KIP-54 Sticky Partition Assignment
Just adding some related reference here:
Henry Cai is contributing some advanced feature in Kafka Streams regarding
static assignment: https://github.com/apache/kafka/pull/1543
The main motivation is that when you do rolling bounce for upgrading your
Kafka Streams code, for example, you would pre
Hey Jason,
Good to know on the round robin assignment. I'll look into that.
The issue I have with the current rebalance listener is that it's not
intuitive and unnecessarily exposes the inner workings of rebalance logic.
When the onPartitionsRevoked method is called it's not really saying the
par
:Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy
>From what I understood, it seems that stickiness is preserved only for the
remaining live consumers.
Say a consumer owns some partitions and then dies. Those partitions will
get redistributed to the rest of the group.
Now if
Hey Andy,
Thanks for jumping in. A couple comments:
In addition, I think it is important that during a rebalance consumers do
> not first have all partitions revoked, only to have a very similar, (or the
> same!), set reassigned. This is less than initiative and complicates client
> code unnecess
Hi all,
I think sticky assignment is immensely important / useful in many
situations. Apps that use Kafka are many and varied. Any app that stores
any state, either in the form of data from incoming messages, cached
results from previous out-of-process calls or expensive operations, (and
let's fac
>From what I understood, it seems that stickiness is preserved only for the
remaining live consumers.
Say a consumer owns some partitions and then dies. Those partitions will
get redistributed to the rest of the group.
Now if the consumer comes back up, based on the algorithm described with
the c
Hey Vahid,
Thanks for the updates. I think the lack of comments on this KIP suggests
that the motivation might need a little work. Here are the two main
benefits of this assignor as I see them:
1. It can give a more balanced assignment when subscriptions do not match
in a group (this is the same
Hi Jason,
I updated the KIP and added some details about the user data, the
assignment algorithm, and the alternative strategies to consider.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-54+-+Sticky+Partition+Assignment+Strategy
Please let me know if I missed to add something. Thank you
2016 08:14 PM
Subject: Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy
Hi Vahid,
The only thing I added was the specification of the UserData field. The
rest comes from here:
https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol
.
See the section on the
> (so we check the second assign() call output preserves the assignments
> > > coming from the first assign() call output); or
> > > - start off by a preset assignment, call assign() after some
> subscription
> > > change, and verify the previous assignment are prese
again.
Regards,
--Vahid
From: Jason Gustafson
To: dev@kafka.apache.org
Date: 06/06/2016 01:26 PM
Subject: Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy
Hi Vahid,
Can you add some detail to the KIP on the structure of the user data? I'm
guessing it would be some
do these. Did I overlook them, or
> > stickiness is being tested in some other fashion?
> >
> > Also, if there is a high-level write-up about how this assignor works
> > could you please point me to it? Thanks.
> >
> > Regards.
> > --Vahid
> >
> >
> >
>
Thanks.
>
> Regards.
> --Vahid
>
>
>
>
> From: Guozhang Wang
> To: "dev@kafka.apache.org"
> Date: 05/02/2016 10:34 AM
> Subject:Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy
>
>
>
> Just FYI, the StreamsPartitionAssignor in Kaf
is a high-level write-up about how this assignor works
could you please point me to it? Thanks.
Regards.
--Vahid
From: Guozhang Wang
To: "dev@kafka.apache.org"
Date: 05/02/2016 10:34 AM
Subject: Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy
Jus
2/2016 10:34 AM
Subject: Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy
Just FYI, the StreamsPartitionAssignor in Kafka Streams are already doing
some sort of sticky partitioning mechanism. This is done through the
userData field though; i.e. all group members send their current "
e KIP please let me know or directly update the KIP.
Regards,
Vahid Hashemian
From: Ewen Cheslack-Postava
To: dev@kafka.apache.org
Date: 04/29/2016 09:48 PM
Subject:Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy
I think I'm unclear how we leverage the
onPartiti
t; you have any suggestion on this please share. Thanks.
> >
> > Regards,
> > Vahid Hashemian
> >
> >
> >
> >
> > From: Jason Gustafson
> > To: dev@kafka.apache.org
> > Date: 04/14/2016 11:37 AM
> > Subject:Re: [DISCUS
re. Thanks.
>
> Regards,
> Vahid Hashemian
>
>
>
>
> From: Jason Gustafson
> To: dev@kafka.apache.org
> Date: 04/14/2016 11:37 AM
> Subject:Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy
>
>
>
> Hi Vahid,
>
> Thanks fo
Hi
why is the calculation of the partition assignments to group member being
executed by the client (leader of the group),
rather than server (eg by the group Coordinator) ?
This question came up working with Vahid Hashemian on
https://issues.apache.org/jira/browse/KAFKA-2273
We have impleme
Hey Edo,
The background is here:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Client-side+Assignment+Proposal.
Yes, it's more work for implementations, but it also opens the door for
other use cases (e.g. Kafka Connect is using the same protocol). It also
makes it easier for clients to
Hi
why is the calculation of the partition assignments to group member being
executed by the client (leader of the group),
rather than server (eg by the group Coordinator) ?
This question came up working with Vahid Hashemian on
https://issues.apache.org/jira/browse/KAFKA-2273
We have impleme
Hashemian
From: Jason Gustafson
To: dev@kafka.apache.org
Date: 04/14/2016 11:37 AM
Subject:Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy
Hi Vahid,
Thanks for the proposal. I think one of the advantages of having sticky
assignment would be reduce the need to
Hi Vahid,
Thanks for the proposal. I think one of the advantages of having sticky
assignment would be reduce the need to cleanup local partition state
between rebalances. Do you have any thoughts on how the user would take
advantage of this assignor in the consumer to do this? Maybe one approach
i
Hi all,
I have started a new KIP under
https://cwiki.apache.org/confluence/display/KAFKA/KIP-54+-+Sticky+Partition+Assignment+Strategy
The corresponding JIRA is at
https://issues.apache.org/jira/browse/KAFKA-2273
The corresponding PR is at https://github.com/apache/kafka/pull/1020
Your feedback
32 matches
Mail list logo
