Hi David, Is the idea here to skip calling performAssignment(...) on the AbstractCoordinator.onJoinLeader(...) method, or adding a new boolean parameter to the performAssignment(...) method? The reason I ask is because I raised KIP-795 a few weeks back, which aims to add a public API for AbstractCoordinator, which might change (or not) with this KIP.
I see you also mentioned there's some discussions regarding a new consumer protocol. Is this being discussed somewhere else? I'm curious to know how would it work with other systems (like Kafka Connect or Schema Registry) that rely on the rebalance protocol to handle resource assignments. Apologies in advance if these questions are off-topic for the discussion at hand. Regards, Hector From: dev@kafka.apache.org At: 01/24/22 09:08:58 UTC-5:00To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-814: Static membership protocol should let the leader skip assignment Hey Jason, Thanks for your comments. Regarding your first point. Yes, you have it right. Let me complement the KIP to be clearer. Regarding your second point. That is right. New partitions would not be detected while the leader is down. It is definitely not ideal but that seems acceptable to me, at least as a first step. Adding partitions to a topic is an infrequent event so the likelihood of having it while the leader is down is rather low but that could happen. The only way to not suffer from this would be to monitor the metadata changes on the broker side. This implies that we would parse both the subscriptions and the assignments in order to have the full list of topics. I am not sure that it is worth doing it at the moment given that we are thinking about a new consumer protocol. What do you think? I suppose that we would need both in the long term as the current protocol is a bit weird at the moment so we need to fix it anyway. We could use this KIP to fix the protocol and do a subsequent KIP in the future for the server side monitoring if we need it. Best, David On Fri, Jan 21, 2022 at 7:51 PM Jason Gustafson <ja...@confluent.io.invalid> wrote: > > Hey David, > > Thanks for the proposal. This was a tricky bug and I think your approach is > probably the best way forward. > > It would be helpful to add a little more detail to the proposal. When the > coordinator detects that the static leader is returning, it will set > `skipAssignment` to true in the `JoinGroup` response. I believe the intent > is to return all member subscriptions in this response so that the leader > can monitor all topics subscribed in the group (which might be different > from the consumer's own subscription). The leader will then send an empty > `SyncGroup` request to collect its own assignment. Do I have that right? > > I think there might still be an edge case in this proposal (assuming I've > understood it correctly). In between the time that the leader shuts down > and is restarted, it is possible that new partitions are added to one of > the subscribed topics. The returning leader would not know about it > because it has no way to collect the full assignment. Do you think this is > a problem? > > Thanks, > Jason > > On Wed, Jan 19, 2022 at 7:27 AM David Jacot <da...@apache.org> wrote: > > > Hi folks, > > > > I'd like to start a discussion for KIP-814: Static membership protocol > > should let the > > leader skip assignment. This is a small extension to the static > > membership protocol > > to address KAFKA-13435. > > > > The KIP is here: https://cwiki.apache.org/confluence/x/C5-kCw. > > > > Please let me know what you think. > > > > Best, > > David > >
