I think I'm unclear how we leverage the onPartitionsRevoked/onPartitionsAssigned here in any way that's different from our normal usage -- certainly you can use them to generate a diff, but you still need to commit when partitions are revoked and that has a non-trivial cost. Are we just saying that you might be able to save some overhead, e.g. closing/reopening some other resources by doing a flush but not a close() or something? You still need to flush any output and commit offsets before returning from onPartitionsRevoked, right? Otherwise you couldn't guarantee clean handoff of partitions.
In terms of the rebalancing, the basic requirements in the KIP seem sound. Passing previous assignment data via UserData also seems reasonable since it avoids redistributing all assignment data to all members and doesn't rely on the next generation leader being a member of the current generation. Hopefully this shouldn't be surprising since I think I discussed this w/ Jason before he updated the relevant wiki pages :) -Ewen On Mon, Apr 18, 2016 at 9:34 AM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > HI Jason, > > Thanks for your feedback. > > I believe your suggestion on how to take advantage of this assignor is > valid. We can leverage onPartitionsRevoked() and onPartitionsAssigned() > callbacks and do a comparison of assigned partitions before and after the > re-balance and do the cleanup only if there is a change (e.g., if some > previously assigned partition is not in the assignment). > > On your second question, a number of tests that I ran shows that the old > assignments are preserved in the current implementation; except for when > the consumer group leader is killed; in which case, a fresh assignment is > performed. This is something that needs to be fixed. I tried to use your > pointers to find out where the best place is to preserve the old > assignment in such circumstances but have not been able to pinpoint it. If > you have any suggestion on this please share. Thanks. > > Regards, > Vahid Hashemian > > > > > From: Jason Gustafson <ja...@confluent.io> > To: dev@kafka.apache.org > Date: 04/14/2016 11:37 AM > Subject: Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy > > > > Hi Vahid, > > Thanks for the proposal. I think one of the advantages of having sticky > assignment would be reduce the need to cleanup local partition state > between rebalances. Do you have any thoughts on how the user would take > advantage of this assignor in the consumer to do this? Maybe one approach > is to delay cleanup until you detect a change from the previous assignment > in the onPartitionsAssigned() callback? > > Also, can you provide some detail on how the sticky assignor works at the > group protocol level? For example, do you pass old assignments through the > "UserData" field in the consumer's JoinGroup? > > Thanks, > Jason > > On Thu, Apr 14, 2016 at 11:05 AM, Vahid S Hashemian < > vahidhashem...@us.ibm.com> wrote: > > > Hi all, > > > > I have started a new KIP under > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-54+-+Sticky+Partition+Assignment+Strategy > > > The corresponding JIRA is at > > https://issues.apache.org/jira/browse/KAFKA-2273 > > The corresponding PR is at https://github.com/apache/kafka/pull/1020 > > > > Your feedback is much appreciated. > > > > Regards, > > Vahid Hashemian > > > > > > > > > -- Thanks, Ewen
