Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

+1 to make it into V3.1. And the partial and risk-free proposal to make default to ["range", "cooperative-sticky"] in 3.0 is a brilliant idea. Sounds good to me. I can work on it for V3.0 if there's no objections. Thank you. Luke On Wed, Jun 23, 2021 at 12:38 PM Ismael Juma wrote: > +1 to

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

+1 to making the switch in 3.1 with more time versus rushing it in 3.0. Ismael On Tue, Jun 22, 2021 at 8:58 PM Sophie Blee-Goldman wrote: > I believe we've figured out the root cause of KAFKA-12896 > , and should have a fix > prepared shortly.

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

I believe we've figured out the root cause of KAFKA-12896 , and should have a fix prepared shortly. See the linked issues for more details. Regarding KIP-726 itself, given that the latest proposal is fully compatible and does not require any

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

Hi Ryan, Thanks for your good comments. I've listed your comments in "Rejected Alternatives" in KIP. 1. Some cooperative-sticky related defects might not free before V3.0 → We've marked important defects as blocker for V3.0, ex: KAFKA-12896. Please raise any important defect if you found any.

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

Thanks for the quick replies, Luke and Sophie. I've not voted, but I agree with accepting the KIP since it's a superior feature. I was just reacting mostly to this comment since it didn't mention open issues: > > > Thanks Luke. We may as well get this KIP in to 3.0 so that we can fully > > >

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

> rles...@bloomberg.net> wrote: > > > > > Hey guys, > > > > > > Should open bugs concerning cooperative-sticky also be considered > > blockers > > > to making it the default? For example, KAFKA-12896 is perhaps still > being >

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

gt; > Should open bugs concerning cooperative-sticky also be considered > > blockers > > > to making it the default? For example, KAFKA-12896 is perhaps still > being > > > investigated: > > > > > > https://issues.apache.org/jira/browse/KAFKA-12896 &

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

gt; > > > Thanks, > > > > Ryan > > > > From: dev@kafka.apache.org At: 06/07/21 19:37:45 UTC-4:00To: > > dev@kafka.apache.org > > Subject: Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the > > default assignor > > > > Thank

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

ticky also be considered blockers > to making it the default? For example, KAFKA-12896 is perhaps still being > investigated: > > https://issues.apache.org/jira/browse/KAFKA-12896 > > Thanks, > > Ryan > > From: dev@kafka.apache.org At: 06/07/21 19:37:45 UTC-4:00To

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

:00To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor Thanks Luke. We may as well get this KIP in to 3.0 so that we can fully enable cooperative rebalancing by default in 3.0 if we have KAFKA-12477 done in time, and if we don't

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

Thanks Luke. We may as well get this KIP in to 3.0 so that we can fully enable cooperative rebalancing by default in 3.0 if we have KAFKA-12477 done in time, and if we don't then there's no harm as it's not going to change the behavior. On Wed, Jun 2, 2021 at 7:28 PM Luke Chen wrote: > Hi

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

Hi Sophie, Thanks for the reminder. Yes, I was thinking this KIP doesn't have to be put into a major release since it will be fully backward compatible, so no need to push it. Currently, if we want to work on this KIP, we need KAFKA-12477 and KAFKA-12487. But you're right, we can at least try our

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

Hey Luke, It's been a while since the last update on this, which is mostly my fault for picking up other things in the meantime. I'm planning to get back to work on KAFKA-12477 next week but there are minimal changes to the current implementation given the proposal I put forth earlier in this KIP

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

1) From user's perspective, it is always possible that a commit within onPartitionsRevoked throw in practice (e.g. if the member missed the previous rebalance where its assigned partitions are already re-assigned) -- and the onPartitionsLost was introduced for that exact reason, i.e. it is

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

1) Once the short-circuit is triggered, the member will downgrade to the EAGER protocol, but won't necessarily try to rejoin the group right away. In the "happy path", the user has implemented #onPartitionsLost correctly and will not attempt to commit partitions that are lost. And since these

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

Hello Sophie, Thanks for the detailed explanation, a few clarifying questions: 1) when the short-circuit is triggered, what would happen next? Would the consumers switch back to EAGER, and try to re-join the group, and then upon succeeding the next rebalance reset the flag to allow committing?

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

Whoops, small correction--meant to say ConsumerRebalanceListener::onPartitionsLost, not Consumer::onPartitionsLost On Mon, Apr 12, 2021 at 8:17 AM Chris Egerton wrote: > Hi Sophie, > > This sounds fantastic. I've made a note on KAFKA-12487 about being sure to > implement

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

Hi Sophie, This sounds fantastic. I've made a note on KAFKA-12487 about being sure to implement Consumer::onPartitionsLost to avoid unnecessary task failures on consumer protocol downgrade, but besides that, I don't think things could get any smoother for Connect users or developers. The

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

1) Yes, all of the above will be part of KAFKA-12477 (not KIP-726) 2) No, KAFKA-12638 would be nice to have but I don't think it's appropriate to remove the default implementation of #onPartitionsLost in 3.0 since we never gave any indication yet that we intend to remove it 3) Yes, this would be

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

Hi Sophie, That sounds great to take care of each case I can think of. Questions: 1. Do you mean the short-Circuit will also be implemented in KAFKA-12477? 2. I don't think KAFKA-12638 is the blocker of this KIP-726, Am I right? 3. So, does that mean we still have possibility to have multiple

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

Alright, here's the detailed proposal for KAFKA-12477. This assumes we will change the default assignor to ["cooperative-sticky", "range"] in KIP-726. It also acknowledges that users may attempt any kind of upgrade without reading the docs, and so we need to put in safeguards against data

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

Hi Sophie, Making the default to "cooperative-sticky, range" is a smart idea, to ensure we can at least fall back to rangeAssignor if consumers are not following our recommended upgrade path. I updated the KIP accordingly. Hi Chris, No problem, I updated the KIP to include the change in Connect.

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

Hi all, @Sophie - I like the sound of the dual-protocol default. The smooth upgrade path it permits sounds fantastic! @Luke - Do you think we can also include Connect in this KIP? Right now we don't set any custom partition assignment strategies for the consumer groups we bring up for sink

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

Ok I'm still fleshing out all the details of KAFKA-12477 but I think we can simplify some things a bit, and avoid any kind of "fail-fast" which will require user intervention. In fact I think we can avoid requiring the user to make any changes at all for KIP-726, so we don't have to worry about

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

Hi Sophie, My question is more related to KAFKA-12477, but since your latest replies are on this thread I figured we can follow-up on the same venue. Just so I understand your latest comments above about the approach: * I think, we would need to persist this decision so that the group would

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

Hi Sophie & Ismael, Thank you for your feedback. No problem, let's pause this KIP and wait for this improvement: KAFKA-12477 . Stay tuned :) Thank you. Luke On Tue, Mar 30, 2021 at 3:14 AM Ismael Juma wrote: > Hi Sophie, > > I didn't analyze

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

Hi Sophie, I didn't analyze the KIP in detail, but the two suggestions you mentioned sound like great improvements. A bit more context: breaking changes for a widely used product like Kafka are costly and hence why we try as hard as we can to avoid them. When it comes to the brokers, they are

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

Ismael, It seems like given 3.0 is a breaking release, we have to rely on users being aware of this and responsible enough to read the upgrade guide. Otherwise we could never ever make any breaking changes beyond just removing deprecated APIs or other compilation-breaking errors that would be

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

Hi Ismael, Thanks for your good question. Answer them below: *1. Are we saying that every consumer upgraded would have to follow the complex path described in the KIP? * --> We suggest that every consumer did these 2 steps of rolling upgrade. And after KAFKA-12477

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

Thanks for the KIP. Are we saying that every consumer upgraded would have to follow the complex path described in the KIP? Also, what happens if they don't read the instructions and upgrade as they have in the past? Ismael On Fri, Mar 26, 2021, 1:53 AM Luke Chen wrote: > Hi everyone, > > >

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

Hi Sophie, Thanks for your good suggestion. I've updated in the KIP-726. Thank you. Luke On Sat, Mar 27, 2021 at 3:24 AM Sophie Blee-Goldman wrote: > Thanks for the KIP! I'm 100% on board with this (obviously :P) and the KIP > itself looks good to me > overall. Just one clarification I think

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

Thanks for the KIP! I'm 100% on board with this (obviously :P) and the KIP itself looks good to me overall. Just one clarification I think you should make: In the *Public Interfaces* section you say "It won't affect the current consumers" -- this is only true if those current consumers have