Hi Hector, Thanks for the KIP.
At a high level, I think the question to be answered by the community is "Should Kafka really be providing this kind of cluster management API?". While Kafka clients need this to provide their functionality it's a different thing to expose that as a public API of the project, which is otherwise about providing a distributed event log/data streaming platform/<insert your favourite definition of Kafka here>. Having a public API brings a significant commitment for API compatibility, which could impair the ability of the project to change the API in order to make improvements to the Kafka clients. The current AbstractCoordinator not being a supported API means we don't currently have to reason about compatibility here. So I think it would help the motivation section of the KIP to describe in a bit more detail the use case(s) you have for implementing your own coordinators. For example, are these applications using Kafka otherwise, or just to leverage this API? And what alternatives to implementing your own coordinators did you consider, and why did you reject them? >From a technical point of view, there are a number of issues I think would need addressing in order to do something like this: 1. There probably ought to be a way to ensure that protocolTypes() don't collide, or at least reduce the chances of a collision. While probably unlikely in practice the consequences of different protocols having the same name could be pretty confusing to debug. 2. JoinGroupRequestData and JoinGroupResponseData are not public classes (none of the *RequestData or *ResponseData classes are, intentionally), so there would have to be an abstraction for them. 3. It's all well and good having an interface that anyone can implement, but there is no supported Kafka API which takes an instance as a parameter (i.e. where do you plug your implementation in without having to use a bunch of other non-public Kafka APIs?) I assume in your current usage you're having to make use of other non-supported client APIs to make use of your coordinator. The KIP isn't really a complete piece of work without a way to use a custom implementation, in my opinion. It would be confusing if it looked like we were encouraging people to use those other non-supported APIs by making this coordinator public. Kind regards, Tom On Mon, Nov 8, 2021 at 2:01 PM Hector Geraldino (BLOOMBERG/ 919 3RD A) < hgerald...@bloomberg.net> wrote: > Hi Kafka devs, > > I would like to start the discussion of KIP-795: Add public APIs for > AbstractCoordinator > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-795%3A+Add+public+APIs+for+AbstractCoordinator > > Looking forward for some feedback from the community. > > Regards, > Hector
