Say you have 50 connectors all with different ACLs and service account. That's 50 connect clusters to maintain. So 50*3 internal connect topics to maintain (they can't share the same connect topics because they're different clusters). At default config we're talking 1500 partitions which is a lot for a Kafka cluster.
The parallel is with consumer groups. Are all consumer groups backing their offset in their own topic or in a central topic ? Connect should be able to achieve the same centrality of Configs. Finally , Configs should go along with the jar, and not be stored in Kafka, especially for connectors that have secrets. There's no reason Kafka needs to have a database secret on its own disk On Thu., 17 May 2018, 5:55 pm Rahul Singh, <rahul.xavier.si...@gmail.com> wrote: > First sentence fat fingered. > > “Just curious as to why there’s an issue with the backing topics for Kafka > Connect.” > > -- > Rahul Singh > rahul.si...@anant.us > > Anant Corporation > > On May 17, 2018, 6:17 AM -0400, Stephane Maarek < > steph...@simplemachines.com.au>, wrote: > > Hi Salius > > > > I think you're on the money, but you're not pushing things too far. > > This is something I've hoped for a long time. > > Let's talk Kafka Connect v2 > > > > Kafka Connect Cluster, as you said, are not convenient to work with (the > > KIP details drawbacks well). I'm all about containerisation just like > > stream apps support (and boasts!). > > > > Now, here's the problem with Kafka Connect. There are three backing > topics. > > Here's the analysis of how they can evolve: > > - Config topic: this one is irrelevant if each connect cluster comes > with a > > config bundled with the corresponding JAR, as you mentioned in your KIP > > - Status topic: this is something I wish was gone too. The consumers > have a > > coordinator, and I believe the connect workers should have a coordinator > > too, for task rebalancing. > > - Source Offset topic: only relevant for sources. I wish there was a > > __connect_offsets global topic just like for consumers and an > > "ConnectOffsetCoordinator" to talk to to retrieve latest committed > offset. > > > > If we look above, with a few back-end fundamental transformations, we can > > probably make Connect "cluster-less". > > > > What the community would get out of it is huge: > > - Connect workers for a specific connector are independent and isolated, > > measurable (in CPU and Mem) and auto-scalable > > - CI/CD is super easy to integrate, as it's just another container / jar. > > - You can roll restart a specific connector and upgrade a JAR without > > interrupting your other connectors and while keeping the current > connector > > from running. > > - The topics backing connect are removed except the global one, which > > allows you to scale easily in terms of number of connectors > > - Running a connector in dev or prod (for people offering connectors) is > as > > easy as doing a simple "docker run". > > - Each consumer / producer settings can be configured at the container > > level. > > - Each connect process is immutable in configuration. > > - Each connect process has its own security identity (right now, you > need a > > connect cluster per service role, which is a lot of overhead in terms of > > backing topic) > > > > Now, I don't have the Kafka expertise to know exactly which changes to > make > > in the code, but I believe the final idea is achievable. > > The change would be breaking for how Kafka Connect is run, but I think > > there's a chance to make the change non breaking to how Connect is > > programmed. I believe the same public API framework can be used. > > > > Finally, the REST API can be used for monitoring, or the JMX metrics as > > usual. > > > > I may be completely wrong, but I would see such a change drive the > > utilisation, management of Connect by a lot while lowering the barrier to > > adoption. > > > > This change may be big to implement but probably worthwhile. I'd be happy > > to provide more "user feedback" on a PR, but probably won't be able to > > implement a PR myself. > > > > More than happy to discuss this > > > > Best, > > Stephane > > > > > > Kind regards, > > Stephane > > > > [image: Simple Machines] > > > > Stephane Maarek | Developer > > > > +61 416 575 980 > > steph...@simplemachines.com.au > > simplemachines.com.au > > Level 2, 145 William Street, Sydney NSW 2010 > > > > On 17 May 2018 at 14:42, Saulius Valatka <saulius...@gmail.com> wrote: > > > > > Hi, > > > > > > the only real usecase for the REST interface I can see is providing > > > health/liveness checks for mesos/kubernetes. It's also true that the > API > > > can be left as is and e.g. not exposed publicly on the platform level, > but > > > this would still leave opportunities to accidentally mess something up > > > internally, so it's mostly a safety concern. > > > > > > Regarding the option renaming: I agree that it's not necessary as it's > not > > > clashing with anything, my reasoning is that assuming some other offset > > > storage appears in the future, having all config properties at the root > > > level of offset.storage.* _MIGHT_ introduce clashes in the future, so > this > > > is just a suggestion for introducing a convention of > > > offset.storage.<store>.<properties>, which the existing > > > property offset.storage.file.filename already adheres to. But in > general, > > > yes -- this can be left as is. > > > > > > > > > > > > 2018-05-17 1:20 GMT+03:00 Jakub Scholz <ja...@scholz.cz>: > > > > > > > Hi, > > > > > > > > What do you plan to use the read-only REST interface for? Is there > > > > something what you cannot get through metrics interface? Otherwise it > > > might > > > > be easier to just disable the REST interface (either in the code, or > just > > > > on the platform level - e.g. in Kubernetes). > > > > > > > > Also, I do not know what is the usual approach in Kafka ... but do we > > > > really have to rename the offset.storage.* options? The current > names do > > > > not seem to have any collision with what you are adding and they > would > > > get > > > > "out of sync" with the other options used in connect > (status.storage.* > > > and > > > > config.storage.*). So it seems a bit unnecessary change to me. > > > > > > > > Jakub > > > > > > > > > > > > > > > > On Wed, May 16, 2018 at 10:10 PM Saulius Valatka < > saulius...@gmail.com > > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > I'd like to start a discussion on the following KIP: > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > > 304%3A+Connect+runtime+mode+improvements+for+container+platforms > > > > > > > > > > Basically the idea is to make it easier to run separate instances > of > > > > Kafka > > > > > Connect hosting isolated connectors on container platforms such as > > > Mesos > > > > or > > > > > Kubernetes. > > > > > > > > > > In particular it would be interesting to hear opinions about the > > > proposed > > > > > read-only REST API mode, more specifically I'm concerned about the > > > > > possibility to implement it in distributed mode as it appears the > > > > framework > > > > > is using it internally ( > > > > > > > > > > https://github.com/apache/kafka/blob/trunk/connect/ > > > > runtime/src/main/java/org/apache/kafka/connect/runtime/ > > > > distributed/DistributedHerder.java#L1019 > > > > > ), > > > > > however this particular API method appears to be undocumented(?). > > > > > > > > > > Looking forward for your feedback. > > > > > > > > > > > > >
