Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy

ubject: Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy I'm still very interested in seeing this KIP progress ... On Tue, 2 Aug 2016 at 20:09, Vahid S Hashemian <vahidhashem...@us.ibm.com> wrote: > I would like to revive this thread and ask for additional feedback

Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy

I'm still very interested in seeing this KIP progress ... On Tue, 2 Aug 2016 at 20:09, Vahid S Hashemian wrote: > I would like to revive this thread and ask for additional feedback on this > KIP. > > There have already been some feedback, mostly in favor, plus some

Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy

I would like to revive this thread and ask for additional feedback on this KIP. There have already been some feedback, mostly in favor, plus some concern about the value gain considering the complexity and the semantics; i.e. how the eventually revoked assignments need to be processed in the

Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy

"dev@kafka.apache.org" <dev@kafka.apache.org> Date: 06/23/2016 03:28 PM Subject: Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy Just adding some related reference here: Henry Cai is contributing some advanced feature in Kafka Streams regarding static assignment: htt

Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy

016 12:53 PM Subject: Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy Hey Vahid, Comments below: I'm not very clear on the first part of this paragraph. You could clarify > it for me, but in general balancing out the partitions across consumers in > a group as much as po

Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy

1, t1p1] > * C2: [t0p0] > > I think both range and round robin assignors would produce this: > * C0: [t0p0, t1p1] > * C1: [t0p1] > * C2: [t1p0] > > Regards, > --Vahid > > > > > From: Jason Gustafson <ja...@confluent.io> > To: dev@kafka.

Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy

] * C2: [t0p0] I think both range and round robin assignors would produce this: * C0: [t0p0, t1p1] * C1: [t0p1] * C2: [t1p0] Regards, --Vahid From: Jason Gustafson <ja...@confluent.io> To: dev@kafka.apache.org Date: 06/23/2016 10:06 AM Subject: Re: [DISCUSS] KIP-54

Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy

Hi Jason, I appreciate your feedback. Please see my comments below, and advise if you have further suggestions. Thanks. Regards, --Vahid From: Jason Gustafson <ja...@confluent.io> To: dev@kafka.apache.org Date: 06/22/2016 04:41 PM Subject: Re: [DISCUSS] KIP-54

Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy

Just adding some related reference here: Henry Cai is contributing some advanced feature in Kafka Streams regarding static assignment: https://github.com/apache/kafka/pull/1543 The main motivation is that when you do rolling bounce for upgrading your Kafka Streams code, for example, you would

Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy

Hey Jason, Good to know on the round robin assignment. I'll look into that. The issue I have with the current rebalance listener is that it's not intuitive and unnecessarily exposes the inner workings of rebalance logic. When the onPartitionsRevoked method is called it's not really saying the

Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy

06/23/2016 01:03 AM Subject: Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy >From what I understood, it seems that stickiness is preserved only for the remaining live consumers. Say a consumer owns some partitions and then dies. Those partitions will get redistributed to

Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy

Hey Andy, Thanks for jumping in. A couple comments: In addition, I think it is important that during a rebalance consumers do > not first have all partitions revoked, only to have a very similar, (or the > same!), set reassigned. This is less than initiative and complicates client > code

Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy

Hi all, I think sticky assignment is immensely important / useful in many situations. Apps that use Kafka are many and varied. Any app that stores any state, either in the form of data from incoming messages, cached results from previous out-of-process calls or expensive operations, (and let's

Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy

>From what I understood, it seems that stickiness is preserved only for the remaining live consumers. Say a consumer owns some partitions and then dies. Those partitions will get redistributed to the rest of the group. Now if the consumer comes back up, based on the algorithm described with the

Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy

Hey Vahid, Thanks for the updates. I think the lack of comments on this KIP suggests that the motivation might need a little work. Here are the two main benefits of this assignor as I see them: 1. It can give a more balanced assignment when subscriptions do not match in a group (this is the same

Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy

Hi Jason, I updated the KIP and added some details about the user data, the assignment algorithm, and the alternative strategies to consider. https://cwiki.apache.org/confluence/display/KAFKA/KIP-54+-+Sticky+Partition+Assignment+Strategy Please let me know if I missed to add something. Thank

Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy

g Date: 06/06/2016 08:14 PM Subject: Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy Hi Vahid, The only thing I added was the specification of the UserData field. The rest comes from here: https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol

Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy

culating assignments. > > Thanks again. > > Regards, > --Vahid > > > > > From: Jason Gustafson <ja...@confluent.io> > To: dev@kafka.apache.org > Date: 06/06/2016 01:26 PM > Subject:Re: [DISCUSS] KIP-54 Sticky Partition Assignment St

Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy

s, --Vahid From: Jason Gustafson <ja...@confluent.io> To: dev@kafka.apache.org Date: 06/06/2016 01:26 PM Subject: Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy Hi Vahid, Can you add some detail to the KIP on the structure of the user data? I'm guessing it would b

Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy

s seem to do these. Did I overlook them, or > > stickiness is being tested in some other fashion? > > > > Also, if there is a high-level write-up about how this assignor works > > could you please point me to it? Thanks. > > > > Regards. > > --Vahid > > > &

Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy

ks. > > Regards. > --Vahid > > > > > From: Guozhang Wang <wangg...@gmail.com> > To: "dev@kafka.apache.org" <dev@kafka.apache.org> > Date: 05/02/2016 10:34 AM > Subject:Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy > > >

Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy

ances but have not been able to pinpoint it. > If > > you have any suggestion on this please share. Thanks. > > > > Regards, > > Vahid Hashemian > > > > > > > > > > From: Jason Gustafson <ja...@confluent.io> > > To: dev@kafka.a

Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy

e.org" <dev@kafka.apache.org> Date: 05/02/2016 10:34 AM Subject: Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy Just FYI, the StreamsPartitionAssignor in Kafka Streams are already doing some sort of sticky partitioning mechanism. This is done through the userData field

Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy

please let me know or directly update the KIP. Regards, Vahid Hashemian From: Ewen Cheslack-Postava <e...@confluent.io> To: dev@kafka.apache.org Date: 04/29/2016 09:48 PM Subject: Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy I think I'm unclear how we le

Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy

t; > you have any suggestion on this please share. Thanks. > > > > Regards, > > Vahid Hashemian > > > > > > > > > > From: Jason Gustafson <ja...@confluent.io> > > To: dev@kafka.apache.org > > Date: 04/14/2016 11:37 AM > &

Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy

egards, > Vahid Hashemian > > > > > From: Jason Gustafson <ja...@confluent.io> > To: dev@kafka.apache.org > Date: 04/14/2016 11:37 AM > Subject:Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy > > > > Hi Vahid, > > Thanks for t

Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy

Hi why is the calculation of the partition assignments to group member being executed by the client (leader of the group), rather than server (eg by the group Coordinator) ? This question came up working with Vahid Hashemian on https://issues.apache.org/jira/browse/KAFKA-2273 We have

Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy

Hey Edo, The background is here: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Client-side+Assignment+Proposal. Yes, it's more work for implementations, but it also opens the door for other use cases (e.g. Kafka Connect is using the same protocol). It also makes it easier for clients to

Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy

Hi why is the calculation of the partition assignments to group member being executed by the client (leader of the group), rather than server (eg by the group Coordinator) ? This question came up working with Vahid Hashemian on https://issues.apache.org/jira/browse/KAFKA-2273 We have

Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy

Hashemian From: Jason Gustafson <ja...@confluent.io> To: dev@kafka.apache.org Date: 04/14/2016 11:37 AM Subject: Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy Hi Vahid, Thanks for the proposal. I think one of the advantages of having sticky assignment

Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy

Hi Vahid, Thanks for the proposal. I think one of the advantages of having sticky assignment would be reduce the need to cleanup local partition state between rebalances. Do you have any thoughts on how the user would take advantage of this assignor in the consumer to do this? Maybe one approach

[DISCUSS] KIP-54 Sticky Partition Assignment Strategy

Hi all, I have started a new KIP under https://cwiki.apache.org/confluence/display/KAFKA/KIP-54+-+Sticky+Partition+Assignment+Strategy The corresponding JIRA is at https://issues.apache.org/jira/browse/KAFKA-2273 The corresponding PR is at https://github.com/apache/kafka/pull/1020 Your